Spark 1.6.Token can only be issued using Kerberos or Internet Authentication
I am calling kinit keytab right before sending the script to the shell driver. The thing is, it works on its own, but when I call the shell driver, script through Oozie, I got this error:
Stdoutput py4j.protocol.Py4JJavaError: An error occurred while calling
o49.saveAsTextFile.
Stdoutput : org.apache.hadoop.ipc.RemoteException(java.io.IOException):
Delegation Token can be issued only with kerberos or web authentication
The problem is probably here
file.coalesce(1,True).saveAsTextFile(FQDNofHadoop+output)
EDIT: My script has: kinit -k -t / home / me / me.keytab me@DOMAIN.HAD
EDIT: Working solution:
I used
spark-submit --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \
and executing the pyspark script in oozie worked without error (even writing to the hive table, etc.). Log4j logger didn't work (with standalone scripts it does), but at least print () (stdout to yarn logs) yes ...
thank
+3
source to share