spark-submit in local mode - configuration

apache-spark, hadoop

Solution

Try to increase --driver-memory, --executor-memory, default value is 1g for both.

command should be like this:

spark-submit --master local --driver-memory 2g --executor-memory  2g  --class classpath jarfile

Problem

I am doing a `spark-submit` using `--master local` on my laptop (spark 1.6.1) to load data into hive tables. Laptop has 8 GB RAM and 4 cores. I have not set any properties manually - just using defaults. When I load 50k records, the jobs finishes successfully. But when I try and load 200k records, I get a "GC Overhead Limit Exceeded" error. In --master local mode, are there properties for job memory or heap memory that could be set manually?

Original source