SparkConf settings not used when running Spark app in cluster mode on YARN
apache-spark, executor, hadoop-yarn, memory-management
Solution
OK, I think I found out the problem! In short form: There's a difference between running Spark settings in Standalone and in YARN-managed mode!
So when you run Spark applications in the Standalone mode, you can focus on the Configuration documentation of Spark, see http://spark.apache.org/docs/1.6.2/configuration.html
You can use the following settings for Driver & Executor CPU/RAM (just as explained in the documentation):
- `spark.executor.cores`
- `spark.executor.memory`
- `spark.driver.cores`
- `spark.driver.memory`
BUT: When running Spark inside a YARN-managed Hadoop environment, you have to be careful with the following settings and consider the following points:
orientate on the "Spark on YARN" documentation rather then on the Configuration documentation linked above: http://spark.apache.org/docs/1.6.2/running-on-yarn.html (the properties explained here have a higher priority then the ones explained in the Configuration docu (this seems to describe only the Standalone cluster vs. client mode, not the YARN cluster vs. client mode!!))
you can't use `SparkConf` to set properties in yarn-cluster mode! Instead use the corresponding `spark-submit` parameters:
- `--executor-cores 5`
- `--executor-memory 5g`
- `--driver-cores 3`
- `--driver-memory 3g`
In yarn-client mode you can't use the `spark.driver.cores` and `spark.driver.memory` properties! You have to use the corresponding AM properties in a `SparkConf` instance:
- `spark.yarn.am.cores`
- `spark.yarn.am.memory`
- You can't set these AM properties via `spark-submit` parameters!
- To set executor resources in yarn-client mode you can use
- `spark.executor.cores` and `spark.executor.memory` in `SparkConf`
- `--executor-cores` and `executor-memory` parameters in `spark-submit`
- if you set both, the `SparkConf` settings overwrite the `spark-submit` parameter values!
This is the textual form of my notes:
Hope I can help anybody else with this findings...
Problem
I wrote a Spark application, which sets sets some configuration stuff via `SparkConf` instance, like this: ``` SparkConf conf = new SparkConf().setAppName("Test App Name"); conf.set("spark.driver.cores", "1"); conf.set("spark.driver.memory", "1800m"); conf.set("spark.yarn.am.cores", "1"); conf.set("spark.yarn.am.memory", "1800m"); conf.set("spark.executor.instances", "30"); conf.set("spark.executor.cores", "3"); conf.set("spark.executor.memory", "2048m"); JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<String> inputRDD = sc.textFile(...); ... ``` When I run this application with the command (`master=yarn` & `deploy-mode=client`) ``` spark-submit --class spark.MyApp --master yarn --deploy-mode client /home/myuser/application.jar ``` everything seems to work fine, the Spark History UI shows correct executor information: But when running it with (`master=yarn` & `deploy-mode=cluster`) my Spark UI shows wrong executor information (~512 MB instead of ~1400 MB): Also my App name equals `Test App Name` when running in client mode, but is `spark.MyApp` when running in cluster mode. It seems that however some default settings are taken when running in Cluster mode. What am I doing wrong here? How can I make these settings for the Cluster mode? I'm using Spark 1.6.2 on a HDP 2.5 cluster, managed by YARN.