Spark-submit not working when application jar is in hdfs

apache-spark, hadoop, hdfs

Solution

The only way it worked for me, when I was using

--master yarn-cluster

Problem

I'm trying to run a spark application using bin/spark-submit. When I reference my application jar inside my local filesystem, it works. However, when I copied my application jar to a directory in hdfs, i get the following exception: Warning: Skip remote jar hdfs://localhost:9000/user/hdfs/jars/simple-project-1.0-SNAPSHOT.jar. java.lang.ClassNotFoundException: com.example.SimpleApp Here's the command: `$ ./bin/spark-submit --class com.example.SimpleApp --master local hdfs://localhost:9000/user/hdfs/jars/simple-project-1.0-SNAPSHOT.jar` I'm using hadoop version 2.6.0, spark version 1.2.1

Original source