How to see the details of the "finished application" in the web interface after restarting Spark
I want to test my Spark application through the web interface after it completes. So I set spark.eventLog.enabled
to true and then installed spark.eventLog.dir
to local directory. Then when I click on the filled in app name, it shows me the app information. But when I reset the OS and start Spark again, it doesn't show me the finished apps. I spark.eventLog.dir
go through the directory and see that the finished app information was there, but I don't know how to use that folder again to see it in the web interface?
source to share
Restarting Spark clears the list of completed applications, Spark stores them only in memory.
If you want to check old applications, you must start the history server with /path/to/spark/sbin/start-history-server.sh /path/to/event-log
.
More information here: https://spark.apache.org/docs/latest/monitoring.html
source to share