pyspark Do python processes on an executor node share broadcast variables in ram?
apache-spark, pyspark, python, shared-memory
Solution
I believe that PySpark doesn't use any form of shared memory to share broadcast variables between the workers.
On Unix-like systems broadcast variables are loaded in the main function of the worker which is called only after forking from the daemon so there are not accessible from the parent process space.
If you want to reduce footprint of the large variables without using external service I would recommend using file backed objects with memory-map. This way you can efficiently use for example NumPy arrays.
In contrast native (JVM) Spark applications indeed share broadcast variables between multiple executor threads on a single executor JVM.
Problem
I have a node that has 24 cores and 124Gb ram in my spark cluster. When I set the spark.executor.memory field to 4g, and then broadcast a variable that takes 3.5gb to store in ram, will the cores collectively hold 24 copies of that variable? Or one copy? I am using pyspark - v1.6.2