Apache Spark - Why is the executor removed? What does "Idle" mean?
With Dynamic Resource Allocation :
Spark app uninstalls an executor when it is inactive for more than spark.dynamicAllocation.executorIdleTimeout seconds.
And when I set the idle idle property of the executor as it should spark.dynamicAllocation.executorIdleTimeout= 300
, it throws the following warning:
spark.ExecutorAllocationManager: Removing executor 0 because it has been idle for 300 seconds (new desired total will be 2)
What does it mean to be "inactive"? Does this mean the worker is not using the CPU? Is a blocking database call being made as Idle?
source to share
What does it mean to be "single"?
To answer this question, I would like to go back to the official documentation. So, as mentioned in the documentation you cited, the dynamic allocation mechanism gives Spark the ability to dynamically adjust the resources your application is using based on the workload.
This means that your application can:
-
Request them again later when necessary.
-
Return resources to the cluster if they are no longer in use: the state of the executor is considered unoccupied , which means the term indicating that he is sitting there doing nothing (and reserving resources), i.e. In your case, it issued the following warning:
spark.ExecutorAllocationManager: Removing executor 0 because it has been idle for 300 seconds (new desired total will be 2)
Does this mean that the employee is not using the processor?
The contractor reserves the processor and memory. In case other apps need to use them, this is well reserved. Your resource manager cannot allocate resources for your other applications. Thus, freeing them can be fruitful when multiple applications share the same cluster.
Is a blocking database call considered idle?
When accessing the database, resources are usually requested, so the executor is not idle, performing any task (even this one).
To learn more about it ExecutorAllocationManager
, I advise you to look at its code here .
source to share