Number of performers in local spark mode
So I am running a spark job in local mode. I am using the following command to start the job
spark-submit --master local[*] --driver-memory 256g --class main.scala.mainClass target/scala-2.10/spark_proj-assembly-1.0.jar 0 large.csv 100 outputFolder2 10
I am running this on a machine with 32 cores and 256GB of RAM. When creating conf I am using the following code
val conf = new SparkConf().setMaster("local[*]").setAppName("My App")
Now I'm in local mode right now, Spark runs everything inside one JVM, but does that mean it only runs one driver and uses it as an executor. My timeline shows one executor driver added. And when I go to the Executors page there is only one executor with 32 cores assigned to it
Is this the default behavior? I expected spark to run one executor per core, instead of one executor getting the entire core. If someone can explain the behavior that would be great
source to share
Is this the default behavior?
In local mode, your driver + executors, as you said, are created inside one JVM process. What you see is not a performer, it is an idea of how many cores your job has at its disposal. Typically, when working in local mode, you should only see the driver in the artist view.
If you look at the code LocalSchedulerBackend
, you will see the following comment:
/**
* Used when running a local version of Spark where the executor, backend, and master all run in
* the same JVM. It sits behind a [[TaskSchedulerImpl]] and handles launching tasks on a single
* Executor (created by the [[LocalSchedulerBackend]]) running locally.
We have one, in the same JVM instance, that handles all the tasks.
source to share