How to see the details of the "finished application" in the web interface after restarting Spark

I want to test my Spark application through the web interface after it completes. So I set spark.eventLog.enabled

to true and then installed spark.eventLog.dir

to local directory. Then when I click on the filled in app name, it shows me the app information. But when I reset the OS and start Spark again, it doesn't show me the finished apps. I spark.eventLog.dir

go through the directory and see that the finished app information was there, but I don't know how to use that folder again to see it in the web interface?

+3


source to share


1 answer


Restarting Spark clears the list of completed applications, Spark stores them only in memory.

If you want to check old applications, you must start the history server with /path/to/spark/sbin/start-history-server.sh /path/to/event-log

.



More information here: https://spark.apache.org/docs/latest/monitoring.html

+1


source







All Articles