Shulou(Shulou.com)06/01 Report--

This article shares information about Java application authentication timeout exceptions in Kerberos environments. Xiaobian thinks it is quite practical, so share it with everyone for reference. Let's follow Xiaobian and have a look.

test environment

CM and CDH version 5.15.1

2. The operating system version is RedHat 7.2

3. Cluster Kerberos enabled

problem description

When using JDK 8, applications in Kerberos environments report the following error during execution:

Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

problem analysis

Long-running jobs in the Kerberos environment experience authentication failures due to expired tickets. Ticket expiration is controlled by two parameters, ticket_lifetime and renew_lifetime. The specific analysis is as follows:

Let's look at the ticket_lifetime and renew_lifetime parameters in krb5.conf:

ticket_lifetime = 24h

renew_lifetime = 7d

Note: This is actually related to ticket_lifetime and renew_lifetime in kdc.conf. This article does not discuss it, assuming that your configuration is consistent.

Each Kerberos ticket, including TGT, has a ticket_lifetime (default: 1 day); a Ticket can be renewed, but only up to renew_lifetime (default: 7 days), and cannot be renewed beyond 7 days.

If your application needs to run for a long time or continuously, there is a problem, namely:

After Kerberos authentication login at application startup, do I still need to renew ticket periodically or use keytab to login again after ticket expiration? For example, do you need to add a new UGI.doAS(...) Before calling UGI.checkTGTAndReloginFromKeytab or periodically calling UGI.checkTGTAndReloginFromKeytab using a Timer?

To answer this question, first understand how Kerberos authentication works in Hadoop:

The primary use case for Hadoop Kerberos authentication is the Hadoop RPC framework (kertos-authentication using SASL). Most Hadoop daemon processes call UGI(org.apache.hadoop.security.UserGroupInformation) at startup, UGI.loginUserFromKeytab to authenticate keranthus and get a ticket, and use that ticket authentication on every subsequent RPC call. For example, DataNode must authenticate its RPC calls to NameNode, and NodeManager must authenticate its RPC calls to ResourceManager. So why do these daemons continue to run for so long after startup without keranthus ticket errors (even beyond renew_lifetime)? This is because Hadoop implements an automatic relogin mechanism at the RPC Client layer. The Client.handleSaslConnectionFailure(org.apache.hadoop.ipc.Client) method has the following code:

// try re-login

if (UserGroupInformation.isLoginKeytabBased()) {

UserGroupInformation.getLoginUser().reloginFromKeytab();

} else if (UserGroupInformation.isLoginTicketBased()) {

UserGroupInformation.getLoginUser().reloginFromTicketCache();

}

The above code shows that when using RPC connection, if authentication fails due to ticket failure, it will automatically relogin.

Based on the above knowledge, the following conclusions can be drawn:

1. If the usage pattern of your application is to perform typical Hadoop RPC Java calls (such as calling the HDFS FileSystem API) after logging in from a keytab, then you don't need to add renew ticket or relogin code to the application layer because the RPC Client layer already does that for you.

2. If the usage pattern of your application is not to use Hadoop RPC, but to call HDFS REST API or YARN REST API (using SPNEGO for keranthus authentication), you need to add relogin code to your application. The specific implementation mode is:

Java mode: The main Java program uses keytab to call UGI.loginUserFromKeytab to log in, and then calls the UGI.checkTGTAndReloginFromKeytab method before each UGI.doAS, or starts another thread to call this method periodically.

Shell mode: The main program logs in using kinit and then starts another child process periodically calling kinit -R to renew ticket or kinit -kt to relogin.

Note that when using JDK 8, there is a bug in UGI's relogin, HADOOP-10786. The bug is due to minor changes in JDK 8's Krb5LoginModule that cause UGI's relogin code to assume that previous logins are not keytab-based logins. So, the UGI.reloginFromKeytab and UGI.checkTGTAndReloginFromKeytab methods do virtually nothing and do not perform relogin. So, at this point the application will still report the following error:

Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

solutions

For bugs that do not reloginFromKeytab properly using JDK 8 applications, workaround is:

1. Use JDK 7 instead of JDK 8, or

2. Make sure the hadoop-common includes the HADOOP-10786 patch.

It is recommended to use packages with CDH5.13.0 or above that already include this patch. If you still experience these problems in the IDE development environment, check the version of the hadoop-common package that the IDE relies on.

Thank you for reading! About "Kerberos environment Java application authentication timeout exception is how to return a responsibility" This article is shared here, I hope the above content can be of some help to everyone, so that we can learn more knowledge, if you think the article is good, you can share it to let more people see it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.