How to connect from JMX remotely to Spark worker on Dataproc
I can connect to the driver just fine by adding the following:
spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9178 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false
But do ...
spark.executor.extraJavaOptions=-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9178 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false
... only gives a bunch of errors in the driver ...
Container id: container_1501548048292_0024_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
... and finally the task will work.
The workers have no errors, it just exits with:
[org.apache.spark.util.ShutdownHookManager] - Shutdown hook called
Spark v2.2.0
and the cluster is a simple 1m-2w configuration and my jobs run smoothly with no executor options.
+3
source to share
1 answer
As Rick Mortis pointed out, the issue is with conflicting ports for the jmx executor.
Installation:
-Dcom.sun.management.jmxremote.port=0
gives a random port and removes errors from Spark. To find out which port it is using, follow these steps:
netstat -alp | grep LISTEN.*<executor-pid>/java
which lists the currently open ports for this process.
0
source to share