Why does spark-shell --master yarn-client fail (yet pyspark --master yarn seems to work)?

Question

I'm trying to run the spark shell on my Hadoop cluster via Yarn. I use - Hadoop 2.4.1 - Spark 1.0.0 My Hadoop cluster already works. In order to use Spark, I built Spark as described here : ``` mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.1 -DskipTests clean package ``` The compilation works fine, and I can run `spark-shell` without troubles. However, running it on yarn : ``` spark-shell --master yarn-client ``` gets me the following error : ``` 14/07/07 11:30:32 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: appMasterRpcPort: -1 appStartTime: 1404725422955 yarnAppState: ACCEPTED 14/07/07 11:30:33 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: appMasterRpcPort: -1 appStartTime: 1404725422955 yarnAppState: FAILED org.apache.spark.SparkException: Yarn application already ended,might be killed or not able to launch application master . at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala…

Accepted Answer

Please check does your Hadoop cluster is running correctly. On the master node next YARN process must be running:

```
$ jps
24970 ResourceManager
```

On slave nodes/executors:

```
$ jps
14389 NodeManager
```

Also make sure that you created a reference (or copied those files) to Hadoop configuration in Spark config directory :

```
$ ll /spark/conf/ | grep site
lrwxrwxrwx  1 hadoop hadoop   33 Jun  8 18:13 core-site.xml -> /hadoop/etc/hadoop/core-site.xml
lrwxrwxrwx  1 hadoop hadoop   33 Jun  8 18:13 hdfs-site.xml -> /hadoop/etc/hadoop/hdfs-site.xml
```

You also can check ResourceManager Web UI on port 8088 - http://master:8088/cluster/nodes. There must be a list of available nodes and resources.

You must take a look at your log files using next command (application ID you can find in Web UI):

```
$ yarn logs -applicationId <yourApplicationId>
```

Or you can look directly to entire log files on Master/ResourceManager host:

```
$ ll /hadoop/logs/ | grep resourcemanager
-rw-rw-r--  1 hadoop hadoop  368414 Jun 12 18:12 yarn-hadoop-resourcemanager-master.log
-rw-rw-r--  1 hadoop hadoop    2632 Jun 12 17:52 yarn-hadoop-resourcemanager-master.out
```

And on Slave/NodeManager hosts:

```
$ ll /hadoop/logs/ | grep nodemanager
-rw-rw-r--  1 hadoop hadoop  284134 Jun 12 18:12 yarn-hadoop-nodemanager-slave.log
-rw-rw-r--  1 hadoop hadoop     702 Jun  9 14:47 yarn-hadoop-nodemanager-slave.out
```

Also check if all environment variables are correct:

```
HADOOP_CONF_LIB_NATIVE_DIR=/hadoop/lib/native
HADOOP_MAPRED_HOME=/hadoop
HADOOP_COMMON_HOME=/hadoop
HADOOP_HDFS_HOME=/hadoop
YARN_HOME=/hadoop
HADOOP_INSTALL=/hadoop
HADOOP_CONF_DIR=/hadoop/etc/hadoop
YARN_CONF_DIR=/hadoop/etc/hadoop
SPARK_HOME=/spark
```

Why does spark-shell --master yarn-client fail (yet pyspark --master yarn seems to work)?

Solution

Problem