How to set spark.akka.frameSize in spark-shell?
apache-spark
Solution
This is documented in Spark's configuration guide under Dynamically Loading Spark Properties:
The Spark shell and `spark-submit` tool support two ways to load configurations dynamically. The first are command line options, such as `--master`, as shown above. `spark-submit` can accept any Spark property using the `--conf` flag, but uses special flags for properties that play a part in launching the Spark application.
For example:
./bin/spark-submit --name "My app" --master local[4] --conf spark.akka.frameSize=100 --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar
Problem
For a particular spark shell session, I am attempting ``` spark-shell -Dspark.akka.frameSize=10000 --executor-memory 4g ``` Within the shell, I get this: ``` System.getProperty("spark.executor.memory") res0: String = 4g System.getProperty("spark.akka.frameSize") res1: String = null ``` It could be this string is incorrect, but I get a frameSize error when attempting to do a take() on my dataset. ``` org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 6:0 was 12518780 bytes which exceeds spark.akka.frameSize (10485760 bytes). Consider using broadcast variables for large values. ``` This shows a frameSize of the default 10M. Perhaps I have the wrong syntax. Please help. Thanks!