Apache Spark for anonymous UID (no username)
I am running Apache node spark slave on OpenShift platform. OpenShift internally runs the docker image as an anonymous user (User with no name, but only UID). I am getting the following exception
17/07/17 16:46:53 INFO SignalUtils: Registered signal handler for INT
12 17/07/17 16:46:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13 Exception in thread "main" java.io.IOException: failure to login
14 at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
15 at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
16 at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
17 at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
18 at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
19 at scala.Option.getOrElse(Option.scala:121)
20 at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
21 at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
22 at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:714)
23 at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:696)
24 at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
25 Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
26 at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
27 at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
28 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
29 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
30 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
31 at java.lang.reflect.Method.invoke(Method.java:497)
32 at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
33 at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
34 at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
35 at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
36 at java.security.AccessController.doPrivileged(Native Method)
37 at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
38 at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
39 at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
40 at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
41 at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
42 at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
43 at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
44 at scala.Option.getOrElse(Option.scala:121)
45 at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
46 at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
47 at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:714)
48 at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:696)
49 at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
50
51 at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
52 at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
53 at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
54 at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
55 at java.security.AccessController.doPrivileged(Native Method)
56 at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
57 at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
58 at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
59 ... 10 more
code> I tried to set the following properties on spark-default.conf, still no use.
spark.eventLog.enabled false
spark.ui.enabled false
spark.acls.enable false
spark.admin.acls *
spark.modify.acls *
spark.modify.acls.groups *
spark.ui.view.acls.groups *
spark.ui.enabled false
Could you please help me on this issue.
thank
Naveen
source to share
Here's an alternative approach that doesn't require nss_wrapper
.
By default, OpenShift containers run with an anonymous user ID and group ID 0
(also called "root"). First, set up your images to be /etc/passwd
owned by group-id 0
and have write access to the group, for example this Dockerfile snippet:
RUN chgrp root /etc/passwd && chmod ug+rw /etc/passwd
Then you can add the following logic when starting the container, for example the following script can be used like ENTRYPOINT
:
#!/bin/bash
myuid=$(id -u)
mygid=$(id -g)
uidentry=$(getent passwd $myuid)
if [ -z "$uidentry" ] ; then
# assumes /etc/passwd has root-group (gid 0) ownership
echo "$myuid:x:$myuid:$mygid:anonymous uid:/tmp:/bin/false" >> /etc/passwd
fi
exec "$@"
This script entry point will automatically provide a passwd file entry for the anonymous uid, so the required tools don't crash.
There is a nice blog post on this and related topics regarding anonymous uids in OpenShift: https://blog.openshift.com/jupyter-on-openshift-part-6-running-as-an-assigned-user-id/
source to share
(I'm keeping this answer because it's useful to know about nss_wrapper
, however this other answer works without the need for an installation or user nss_wrapper)
Spark wants to find its UID in passwd. This integral kink can be resolved with nss_wrapper; a good example of using this solution at the entry point for an image can be found here:
https://github.com/radanalyticsio/openshift-spark/blob/master/scripts/spark/added/entrypoint
# spark likes to be able to lookup a username for the running UID, if
# no name is present fake it.
cat /etc/passwd > /tmp/passwd
echo "$(id -u):x:$(id -u):$(id -g):dynamic uid:$SPARK_HOME:/bin/false" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
# NSS_WRAPPER_GROUP must be set for NSS_WRAPPER_PASSWD to be used
export NSS_WRAPPER_GROUP=/etc/group
export LD_PRELOAD=libnss_wrapper.so
exec "$@"
If you are interested in Spark previews that can be used in Openshift, I recommend starting here:
https://github.com/radanalyticsio/openshift-spark
These images were generated as part of a tool for the community project Radanalytics.io, which has created many tools to easily create spark clusters in Openshift. You can find out more about this project here:
source to share