Spark submit to yarn as a another user
apache-spark, hadoop, hadoop-yarn, kerberos
Solution
For a non-kerberized cluster: `export HADOOP_USER_NAME=zorro` before submitting the Spark job will do the trick. Make sure to `unset HADOOP_USER_NAME` afterwards, if you want to revert to your default credentials in the rest of the shell script (or in your interactive shell session).
For a kerberized cluster, the clean way to impersonate another account without trashing your other jobs/sessions (that probably depend on your default ticket) would be something in this line...
export KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u)_temp_$$
kinit -kt ~/.protectedDir/zorro.keytab zorro@MY.REALM
spark-submit ...........
kdestroy
Problem
Is it possible to submit a spark job to a yarn cluster and choose, either with the command line or inside the jar, which user will "own" the job? The spark-submit will be launch from a script containing the user. PS: is it still possible if the cluster has a kerberos configuration (and the script a keytab) ?