How do I renew Kerberos ticket in client yarn mode?

I used Spark 1.6.0

to access data on the Kerberos supported HDFS API DataFrame.read.parquet($path)

.

My application unfolds like yarn in client mode. By default, a Kerberos ticket expires every 24 hours. Everything works fine for the first 24 hours, but can't read files after 24 hours (or more, like 27 hours).

I tried several ways to login and renew the ticket, doesn't work.

  • Install spark.yarn.keytab

    and spark.yarn.principal

    inspark-defaults.conf

  • Install --keytab

    and --principal

    on the command linespark-submit

  • Start a timer in code to call UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab()

    every 2 hours.

Error details:

WARN  [org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)] - Couldn't setup connection for adam/cluster1@DEV.COM to cdh01/192.168.1.51:8032
DEBUG [org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1632)] - PrivilegedActionException as:adam/cluster1@DEV.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for adam/cluster1@DEV.COMto cdh01/192.168.1.51:8032
ERROR [org.apache.spark.Logging$class.logError(Logging.scala:95)] - Failed to contact YARN for application application_1490607689611_0002.
  java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for adam/cluster1@DEV.COM to cdh01/192.168.1.51:8032; Host Details : local host is: "cdh05/192.168.1.41"; destination host is: "cdh01":8032; 

      

+3


source to share


1 answer


The problem has been resolved. This was caused by the wrong version of the Hadoop lib. In the Spark 1.6 assembly box, this refers to the old version. from the Hadoop lib, so I downloaded it again without the built-in Hadoop library and referenced the third party Hadop 2.8 lib. Then it just works.



+1


source







All Articles