Hue Beeswax / HCat no longer works (default user kerberos) after upgrading to HDP2.2
I almost completed the migration of my secure HDP2.1 to HDP2.2 hasoop cluster. Everything seems to work (including the command line hive), but the tint. If file browser, work browser, pig interface and oozie interface are working, this does not apply to beeswax and webhcat interface. (NB: they worked before migration, with the same hue.ini file).
The error I am getting:
Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/LOCALDOMAIN@HADOOP.DEV not found in Kerberos database)
It seems that thrift is trying to authenticate the user by default krbtgt/LOCALDOMAIN
instead of the configured ones.
I tried to log what is going on in the python file, but couldn't see where it gets this user by default: kerberos main short name is the hive, permissions activation is enabled. The hue and hive proxies are configured in the hdfs conf files.
Full stack trace:
[11 / May / 2015 06:10:40 +0000] access INFO 172.20.43.39 alinz - "GET / beeswax / HTTP / 1.0" [11 / May / 2015 06:10:40 +0000] hive_server2_lib INFO use_sasl = True, mechanism = GSSAPI, kerberos_principal_short_name = hive, impersonation_enabled = True [11 / May / 2015 06:10:40 +0000] thrift_util INFO Thrift exception; retrying: Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) [11 / May / 2015 06:10:40 +0000] thrift_util INFO Thrift exception; retrying: Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) [11 / May / 2015 06:10:40 +0000] thrift_util WARNING Out of retries for thrift call: OpenSession [11 / May / 2015 06:10:40 +0000] thrift_util INFO Thrift saw a transport exception: Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) [11 / May / 2015 06:10:40 +0000] middleware INFO Processing exception: Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) (code THRIFTTRANSPORT): TTransportException ('Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) ',): Traceback (most recent call last): File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response response = callback (request, * callback_args, ** callback_kwargs) File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 69, in index return execute_query (request) File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 526, in execute_query databases = _get_db_choices (request) File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 1849, in _get_db_choices dbs = _get_databases (request) File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 1844, in _get_databases dbs = db.get_databases () File "/usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 110, in get_databases return self.client.get_databases () File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 746, in get_databases return [table [col] for table in self._client.get_databases ()] File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 445, in get_databases res = self.call (self._client.GetSchemas, req) File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 408, in call session = self.open_session (self.user) File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 382, in open_session res = self._client.OpenSession (req) File "/usr/lib/hue/desktop/core/src/desktop/lib/thrift_util.py", line 329, in wrapper raise StructuredThriftTransportException (e, error_code = 502) StructuredThriftTransportException: Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) (code THRIFTTRANSPORT): TTransportException ('Could not start SASL: Error in sasl_client_start (-1) SASL (-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt / LOCALDOMAIN@HADOOP.DEV not found in Kerberos database) ',)
Any idea what could be wrong?
krb5.conf:
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = HADOOP.DEV ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false [logging] default = FILE: /var/log/krb5kdc.log admin_server = FILE: /var/log/kadmind.log kdc = FILE: /var/log/krb5kdc.log [realms] HADOOP.DEV = { admin_server = bt1svlmy kdc = bt1svlmy }
and sudo klist -e /tmp/hue_krb5_ccache
gives:
Ticket cache: FILE: / tmp / hue_krb5_ccache Default principal: hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV Valid starting Expires Service principal 05/11/15 15:10:34 05/12/15 15:10:34 krbtgt / HADOOP.DEV@HADOOP.DEV renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 05/11/15 15:49:52 05/12/15 15:10:34 HTTP / bt1svlmy.bpa.bouyguestelecom.fr @ renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 05/11/15 15:49:52 05/12/15 15:10:34 HTTP / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
I have a ticket krbtgt/HADOOP.DEV@HADOOP.DEV
, but no krbtgt/LOCALDOMAIN@HADOOP.DEV
; maybe this is the cause of the problem?
Kerberos log file:
May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for hive / localhost.localdomain@HADOOP.DEV , Server not found in Kerberos database May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for krbtgt / LOCALDOMAIN@HADOOP.DEV , Server not found in Kerberos database May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for hive / localhost.localdomain@HADOOP.DEV , Server not found in Kerberos database May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for krbtgt / LOCALDOMAIN@HADOOP.DEV , Server not found in Kerberos database May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for hive / localhost.localdomain@HADOOP.DEV , Server not found in Kerberos database May 11 16:12:35 bt1svlmy krb5kdc [12636] (info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0, hue / bt1svlmy.bpa.bouyguestelecom.fr@HADOOP.DEV for krbtgt / LOCALDOMAIN@HADOOP.DEV , Server not found in Kerberos database
It seems to me that I missed the default hostname in conf, but could not find a documentation entry for it.
source to share
Ok, found it (needed to debug full python stack to understand). This is not advertised, but some parameter names hue.ini
have changed:
-
beeswax_server_host
→hive_server_host
-
beeswax_server_port
→hive_server_port
By default, the value hive_server_host
is localhost
, which is not valid in a secure cluster.
source to share