In Spark 1.4.x. How can I set a driver system property using spark-submit parameters?
In Spark & lt; The 1.3.x driver system property can be set using the -conf option shared between spark property parameters and system properties:
spark-submit --conf xxx.xxx=vvvvv
In Spark 1.4.0 this feature is removed, instead the driver writes the following warning:
Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
How do I set the system property of a driver in 1.4.0? Is there a reason it gets removed without a deprecation warning?
Thanks a lot for your advice.
source to share
This behavior was introduced in the 336f7f5373e5f6960ecd9967d3703c8507e329ec
JIRA discussion at https://issues.apache.org/jira/browse/SPARK-7037 . As per the previous version of JIRA, spark-submit simply silently ignores conf options that don't start with spark.
. You can set spark.driver.extraJavaOptions
to have the options you want (depending on what you are trying to accomplish).
source to share