Spark: Run Spark shell from a different directory than where Spark is installed on slaves and master

apache-spark, cluster-computing

Solution

You need to edit three files, `spark-submit`, `spark-class` and `pyspark` (all in the `bin` folder).

Find the line

export SPARK_HOME = [...]

Then change it to

SPARK_HOME = [...]

Finally make sure you set `SPARK_HOME` to the directory where spark is installed on the cluster.

This works for me.

Here you can find a detailed explanation.

http://apache-spark-user-list.1001560.n3.nabble.com/executor-failed-cannot-find-compute-classpath-sh-td859.html

Problem

I have a small cluster (4 machines) set up with 3 slaves and a master node, all installed to `/home/spark/spark`. (I.e, `$SPARK_HOME` is `/home/spark/spark`) When I use the spark shell: `/home/spark/spark/bin/pyspark --master spark://192.168.0.11:7077` everything works fine. However I'd like for my colleagues to be able to connect to the cluster from a local instance of spark on their machine installed in whatever directory they wish. Currently if somebody has spark installed in say `/home/user12/spark` and run `/home/user12/spark/bin/pyspark --master spark://192.168.0.11:7077` the spark shell will connect to the master without problems but fails with an error when I try to run code: ``` class java.io.IOException: Cannot run program "/home/user12/bin/compute-classpath.sh" (in directory "."): error=2, No such file or directory) ``` The problem here is that Spark is looking for the spark installation in /home/user12/spark/, where as I'd like to just tell spark to look in /home/spark/spark/ instead. How do I do this?

Original source