Remotely execute a Spark job on an HDInsight cluster

apache-spark, azure, azure-hdinsight, remote-access

Solution

Spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.

https://github.com/spark-jobserver/spark-jobserver

My solution is using both Scheduler and Spark-jobserver to launch the Spark-job periodically.

Problem

I am trying to automatically launch a Spark job on an HDInsight cluster from Microsoft Azure. I am aware that several methods exist to automate Hadoop job submission (provided by Azure itself), but so far I have not been able to found a way to remotely run a Spark job withouth setting a RDP with the master instance. Is there any way to achieve this?

Original source