How do I renew Kerberos ticket in client yarn mode?
I used Spark 1.6.0
to access data on the Kerberos supported HDFS API DataFrame.read.parquet($path)
.
My application unfolds like yarn in client mode. By default, a Kerberos ticket expires every 24 hours. Everything works fine for the first 24 hours, but can't read files after 24 hours (or more, like 27 hours).
I tried several ways to login and renew the ticket, doesn't work.
- Install
spark.yarn.keytab
andspark.yarn.principal
inspark-defaults.conf
- Install
--keytab
and--principal
on the command linespark-submit
- Start a timer in code to call
UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab()
every 2 hours.
Error details:
WARN [org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)] - Couldn't setup connection for adam/cluster1@DEV.COM to cdh01/192.168.1.51:8032
DEBUG [org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1632)] - PrivilegedActionException as:adam/cluster1@DEV.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for adam/cluster1@DEV.COMto cdh01/192.168.1.51:8032
ERROR [org.apache.spark.Logging$class.logError(Logging.scala:95)] - Failed to contact YARN for application application_1490607689611_0002.
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for adam/cluster1@DEV.COM to cdh01/192.168.1.51:8032; Host Details : local host is: "cdh05/192.168.1.41"; destination host is: "cdh01":8032;
source to share
The problem has been resolved. This was caused by the wrong version of the Hadoop lib. In the Spark 1.6 assembly box, this refers to the old version. from the Hadoop lib, so I downloaded it again without the built-in Hadoop library and referenced the third party Hadop 2.8 lib. Then it just works.
source to share