How to keep the Spark web UI alive?

apache-spark

Solution

To add a newbie friendly step-by-step solution of how to work with the history server:

In the spark distribution folder try to start the history server by:

`./sbin/start-history-server.sh`

By default the history server will try to monitor `/tmp/spark-events` for logs and unfortunately it will crash if the path doesn't exist. So if you get an error you may have to `mkdir /tmp/spark-events` first. You can check the history server logs in `./logs` to see details in case of trouble.

In order for a context to persists its event log, you have to enable event logging. This can be done either programmatically or by editing `./conf/spark-defaults.conf` (copy the template if it doesn't exist yet), and uncommenting/adding the line:

`spark.eventLog.enabled true`

Running a `spark-submit` should result in event log folders like `/tmp/spark-events/local-1465133722470`.

Access the history server UI, typically from http://localhost:18080

Problem

Original source