I can't seem to get --py-files on Spark to work

Question

I'm having a problem with using Python on Spark. My application has some dependencies, such as numpy, pandas, astropy, etc. I cannot use virtualenv to create an environment with all dependencies, since the nodes on the cluster do not have any common mountpoint or filesystem, besides HDFS. Therefore I am stuck with using `spark-submit --py-files`. I package the contents of site-packages in a ZIP file and submit the job like with `--py-files=dependencies.zip` option (as suggested in Easiest way to install Python dependencies on Spark executor nodes?). However, the nodes on cluster still do not seem to see the modules inside and they throw `ImportError` such as this when importing numpy. ``` File "/path/anonymized/module.py", line 6, in import numpy File "/tmp/pip-build-4fjFLQ/numpy/numpy/__init__.py", line 180, in File "/tmp/pip-build-4fjFLQ/numpy/numpy/add_newdocs.py", line 13, in File "/tmp/pip-build-4fjFLQ/numpy/numpy/lib/__init__.py", line 8, in #…

Accepted Answer

To get this dependency distribution approach to work with compiled extensions we need to do two things:

- Run the pip install on the same OS as your target cluster (preferably on the master node of the cluster). This ensures compatible binaries are included in your zip.

- Unzip your archive on the destination node. This is necessary since Python will not import compiled extensions from zip files. (https://docs.python.org/3.8/library/zipimport.html)

Using the following script to create your dependencies zip will ensure that you are isolated from any packages already installed on your system. This assumes virtualenv is installed and `requirements.txt` is present in your current directory, and outputs a `dependencies.zip` with all your dependencies at the root level.

```
env_name=temp_env

# create the virtual env
virtualenv --python=$(which python3) --clear /tmp/${env_name}

# activate the virtual env
source /tmp/${env_name}/bin/activate

# download and install dependencies
pip install -r requirements.txt

# package the dependencies in dependencies.zip. the cd magic works around the fact that you can't specify a base dir to zip
(cd /tmp/${env_name}/lib/python*/site-packages/ && zip -r - *) > dependencies.zip
```

The dependencies can now be deployed, unzipped, and included in the PYTHONPATH as so

```
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --conf 'spark.yarn.dist.archives=dependencies.zip#deps' \
  --conf 'spark.yarn.appMasterEnv.PYTHONPATH=deps' \
  --conf 'spark.executorEnv.PYTHONPATH=deps' \
.
.
.
```

spark.yarn.dist.archives=dependencies.zip#deps distributes your zip file and unzips it to a directory called `deps`

spark.yarn.appMasterEnv.PYTHONPATH=deps spark.executorEnv.PYTHONPATH=deps includes the `deps` directory in the PYTHONPATH for the master and all workers

--deploy-mode cluster runs the master executor on the cluster so it picks up the dependencies

I can't seem to get --py-files on Spark to work

Solution

Problem

Related problems