Spark 1.6.Token can only be issued using Kerberos or Internet Authentication

I am calling kinit keytab right before sending the script to the shell driver. The thing is, it works on its own, but when I call the shell driver, script through Oozie, I got this error:

Stdoutput py4j.protocol.Py4JJavaError: An error occurred while calling 
o49.saveAsTextFile.
Stdoutput : org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
Delegation Token can be issued only with kerberos or web authentication

      

The problem is probably here

file.coalesce(1,True).saveAsTextFile(FQDNofHadoop+output) 

      

EDIT: My script has: kinit -k -t / home / me / me.keytab me@DOMAIN.HAD

EDIT: Working solution:

I used

spark-submit --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \ 

      

and executing the pyspark script in oozie worked without error (even writing to the hive table, etc.). Log4j logger didn't work (with standalone scripts it does), but at least print () (stdout to yarn logs) yes ...

thank

+3


source to share


1 answer


I used --principal ' me@DOMAIN.HAD ' \ - keytab '/home/me/me.keytab' and executing the pyspark script worked without error (even writing to the beehive table and so on). Logger didn't work, but at least print () yes ...



+1


source







All Articles