Spark: executor memory exceeds physical limit
apache-spark, apache-spark-sql
Solution
The 9GB is composed of the 8GB executor memory which you add as a parameter, `spark.yarn.executor.memoryOverhead` which is set to `.1`, so the total memory of the container is `spark.yarn.executor.memoryOverhead + (spark.yarn.executor.memoryOverhead * spark.yarn.executor.memoryOverhead)` which is `8GB + (.1 * 8GB) ≈ 9GB`.
You could run the entire process using a single executor, but this would take ages. To understand this you need to know the notion of partitions and tasks. The number of partition is defined by your input and the actions. For example, if you read a 150gb csv from hdfs and your hdfs blocksize is 128mb, you will end up with `150 * 1024 / 128 = 1200` partitions, which maps directly to 1200 tasks in the Spark UI.
Every single tasks will be picked up by an executor. You don't need to hold all the 150gb in memory ever. For example, when you have a single executor, you obviously won't benefit from the parallel capabilities of Spark, but it will just start at the first task, process the data, and save it back to the dfs, and start working on the next task.
What you should check:
- How big are the input partitions? Is the input file splittable at all? If a single executor has to load a massive amount of memory, it will run out of memory for sure.
- What kind of actions are you performing? For example, if you do a join with very low cardinality, you end up with a massive partitions because all the rows with a specific value, end up in the same partitions.
- Very expensive or inefficient actions performed? Any cartesian product etc.
Hope this helps. Happy sparking!
Problem
My input dataset is about 150G. I am setting ``` --conf spark.cores.max=100 --conf spark.executor.instances=20 --conf spark.executor.memory=8G --conf spark.executor.cores=5 --conf spark.driver.memory=4G ``` but since data is not evenly distributed across executors, I kept getting ``` Container killed by YARN for exceeding memory limits. 9.0 GB of 9 GB physical memory used ``` here are my questions: ``` 1. Did I not set up enough memory in the first place? I think 20 * 8G > 150G, but it's hard to make perfect distribution, so some executors will suffer 2. I think about repartition the input dataFrame, so how can I determine how many partition to set? the higher the better, or? 3. The error says "9 GB physical memory used", but i only set 8G to executor memory, where does the extra 1G come from? ``` Thank you!