Increase Java heap size in Spark on Yarn
How to increase core Java Heap Space with spark on Yarn extra java options?
This code is what I am still working on:
./bin/spark-shell --master yarn-client --num-executors 10 --executor-memory 4g
This is what doesn't work:
./bin/spark-shell --master yarn-client --num-executors 10 --executor-memory 4g --extrajavaoptions '-Xmx2g'
I want to add spark.yarn.am.extraJavaOptions. The default is none. I know it must be a string of additional JVM parameters to navigate to the YARN Application Wizard in client mode. I want to enter something like -Xmx2g. Can anyone tell me what I am doing wrong?
source to share
First of all, the way to use the command is as follows:
--conf spark.executor.extrajavaoptions="Option" [Cluster]
--conf spark.yarn.am.extraJavaOptions="Option" [YARN]
notice, that
According to Spark configuration documentation
spark.executor.extraJavaOptions
A string of additional JVM parameters to pass to executors. For example GC settings or other logging. Note that this option prevents you from setting Spark properties or heap size parameters. Spark properties must be set using a SparkConf object or spark-defaults.conf file used with the spark-submit script parameter. Heap size parameters can be set using spark.executor.memory.
You should not set heap size parameters with this parameter. Instead, you can tweak them in the spark script defaults .
An example layout can be found here: Git repo
source to share