Clever Geek Handbook

The role of performers on the Spark master machine

In a single Spark cluster, does the node master perform tasks? I was not sure if the Executors processes would be running on the Master node and they would work alongside the Worker nodes.

Thank!

+3


source to share


1 answer


Executors run only on nodes where this node has at least one working daemon, i.e. no executor will run in a node that will not serve as a worker.

However, where to start Master and Workers are based on your decision, there are no restrictions that Master and Worker cannot find together on the same node.

To start a working daemon on the same machine with your master, you can either edit the file conf/slaves

to add the master-ip to it, use start -all.sh during startup, or start anytime you want on the master node, start-slave.sh

and put the main url to Spark--master spark://master-host:7077



Update (based on suggestion from Daniel Darabos):

When you link to a tab Application Detail UI

Executors

, you can also find a line <driver>

for Executor ID

yours, then the driver it stands for is the process in which your job is scheduled and monitored, at startup main

, which you sent to the spark cluster, slicing your transformations and actions into RDD into stages, scheduling stages like TaskSets and organizing Executors

to complete tasks.

This one <driver>

will run in the node you call spark-submit

in client mode

, or one of the worker nodes incluster mode

+2


source






More articles:


All Articles