How do I set up Spark's native web interface for HTTPS?
I would like to customize Spark Standalone Web UI so that it can be accessed over HTTPS.
The spark is fired on a cluster external to the computer that I use to access the browser.
Here's what I have done so far and it doesn't work:
- Used by OpenSSL to create a self-signed certificate and key and then keytool to create a keystore
- Packaged certificate and key as .p12 file
- In Chrome settings under Certificate Management, imported p12 file
-
Added options to spark-defaults.conf
spark.ui.https.enabled true spark.ui.ssl.server.keystore.location /path/to/spark.keystore spark.ui.ssl.server.keystore.keypassword password spark.ui.ssl.server.keystore.password password
-
Added to spark-env.sh
export SPARK_MASTER_OPTS="-Dspark.ui.https.enabled=true \ -Dspark.ui.ssl.server.keystore.location=/path/to/spark.keystore \ -Dspark.ui.ssl.server.keystore.keypassword=password \ -Dspark.ui.ssl.server.keystore.password=password" export SPARK_WORKER_OPTS="-Dspark.ui.https.enabled=true \ -Dspark.ui.ssl.server.keystore.location=/path/to/spark.keystore \ -Dspark.ui.ssl.server.keystore.keypassword=password \ -Dspark.ui.ssl.server.keystore.password=password"
I tried to connect to the server before, after and in between these steps, and I keep getting the error "This site cannot provide a secure connection." What am I missing here?
source to share
As per this line I think it is not possible to configure the native Spark Standalone web interface using HTTPS.
masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort
My recommendation is to log the problem in Spark JIRA and find a Spark developer to fix it.
source to share
Below settings worked for me, try putting this in "spark-defaults.conf" and restart the Spark service. Also check the logs for which the Spark UI port is listening since "spark.ssl.ui.port" is set to "0". In my case, it worked on port 8480.
spark.ssl.enabled true
spark.ssl.ui.port 0
spark.ssl.keyStore <path_to_keystore>
spark.ssl.keyStorePassword <keystore_password>
spark.ssl.keyPassword <key_password>
spark.ssl.trustStore <path_to_truststore>
spark.ssl.trustStorePassword <truststore_password>
spark.ssl.enabledAlgorithms ECDHE-RSA-AES256-SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
spark.ssl.protocol TLSv1.2
spark.ssl.trustStoreType JKS
See screenshot for reference.
source to share