Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to configure the Kafka client to use LDAP for authentication". Many people will encounter this dilemma in the operation of the actual case, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The example shown here highlights authentication-related attributes in bold to distinguish them from other required security attributes, as shown in the following example. Assume that TLS is enabled for the Apache Kafka cluster and that TLS should be enabled for each secure cluster.

Security.protocol=SASL_SSLssl.truststore.location=/opt/cloudera/security/jks/truststore.jks

We use Kafka-console-consumer in all the examples below. All concepts and configurations also apply to other applications.

LDAP verification

LDAP represents a lightweight directory access protocol and is an industry-standard application protocol for authentication. It is one of the authentication mechanisms supported by Kafka on CDP.

LDAP authentication is also done through the SASL framework, similar to Kerberos. SASL supports a variety of authentication mechanisms, such as GSSAPI, which we introduced in the previous article, and PLAIN, which will be used for LDAP authentication.

The following Kafka client properties must be set to configure Kafka clients to authenticate through LDAP:

# Uses SASL/PLAIN over a TLS encrypted connectionsecurity.protocol=SASL_SSLsasl.mechanism=PLAIN# LDAP credentialssasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username= "alice" password= "supersecret1"; # TLS truststoressl.truststore.location=/opt/cloudera/security/jks/truststore.jks

The above configuration uses SASL / PLAIN for authentication and TLS (SSL) for data encryption. The choice of LDAP authentication is configured on the server-side handler of SASL / PLAIN, which we will cover later in this section.

LDAP and Kerberos

LDAP and Kerberos are different authentication protocols, each with its own advantages and disadvantages. However, the use of these protocols in Kafka clusters is not mutually exclusive. Enabling both Kerberos and LDAP authentication for the cluster is a valid configuration.

Identity directory services, such as Active Directory,RedHat IPA and FreeIPA, support Kerberos and LDAP authentication, which are enabled for the Kafka cluster, providing clients with different options for handling authentication.

LDAP eliminates some of the complexities associated with configuring Kerberos clients, such as requiring Kerberos libraries to be installed on the client and network connectivity to Kerberos KDC in a more rigorous environment.

Ensure that the cluster uses TLS / SSL encryption

Unlike the Kerberos protocol, when using LDAP for authentication, user credentials (username and password) are sent to the Kafka cluster over the network. Therefore, when enabling LDAP authentication for Kafka, it is important to enable and implement TLS encryption for all communication between Kafka clients. This ensures that the credentials are always encrypted over the network and are not compromised.

All Kafka agents must be configured to use the SASL_SSL security protocol for their SASL endpoints.

Enable LDAP authentication on Kafka Broker

When you install the Kafka service, LDAP authentication is not enabled for the Kafka agent by default, but it is easy to configure on the Cloudera data platform (CDP):

In Cloudera Manager, set the following properties in the Kafka service configuration to match your environment: by selecting LDAP as the SASL / PLAIN authentication option above, Cloudera Manager automatically configures Kafka Brokers to use the following SASL / PLAIN callback handler, which implements LDAP authentication:

Org.apache.kafka.common.security.ldap.internals.LdapPlainServerCallbackHandler

Kafka must connect to the LDAP server through a TLS connection (LDAPS). To ensure that the Kafka agent can trust the LDAP server certificate, add the LDAP server's CA certificate to the truststore used by the Kafka service. You can find the location of the truststore in the following properties of Cloudera Manager:

Run the following command (as root) to add the LDAP CA certificate to the truststore:

Keytool\-importcert\-keystore / opt/cloudera/security/jks/truststore.jks\-storetype JKS\-alias ldap-ca\-file / path/to/ldap-ca-cert.pem

Click Kafka > actions > restart to restart the Kafka service and make the changes take effect.

Limitation

The LDAP callback handler for the Kafka server uses the Apache Shiro library to map the user ID (short login) to the user entity in the LDAP realm. It does this by providing a "user DN template" that, given the user's short name, can be used to derive the user's proprietary name in LDAP:

For example, as described in the library documentation, "if the directory uses the LDAP uid attribute to represent the user name, the user DN for the jsmith user might look like this:"

Uid=jsmith,ou=users,dc=mycompany,dc=com

In this case, you can set this property with the following template values:

Uid= {0}, ou=users,dc=mycompany,dc=com "

This limits the use of LDAP callback handlers to LDAP directories configured in such a way that the user name is part of the distinguished name. For RedHat IPA and FreeIPA implementations, this is usually the case. However, in Active Directory, by default, the format of the distinguished name is:

CN=Smith, John, ou=users, dc=mycompany, dc=com

They contain the user's full name instead of the user's ID, which makes it impossible to get it from a short user name through a simple schema. Fortunately, @ is also a valid LDAP user name for Active Directory, in addition to the distinguished name. If you are using Active Directory, you can set the LDAP user DN template to the following template (using the mycompany.com example above):

{0} @ mycompany.com

If your LDAP directory does not accept usernames that can be constructed as described above, consider using Kerberos authentication instead of LDAP.

Example

The following is an example of reading from a topic using Kafka console consumers using LDAP authentication:

# Complete configuration file for LDAP auth$ cat ldap-client.propertiessecurity.protocol=SASL_SSLsasl.mechanism=PLAINsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username= "alice" password= "supersecret1"; ssl.truststore.location=/opt/cloudera/security/jks/truststore.jks# Connect to Kafka using LDAP auth$ kafka-console-consumer\-- bootstrap-server host-1.example.com:9093\-- topic test\-- consumer.config. / ldap-client.properties

Note: the configuration file above contains sensitive credentials. Make sure that the file permission is set so that only the file owner can read it.

What if I don't have a Kerberos or LDAP server?

By far, Kerberos and LDAP authentication are industry standards and are the most common authentication mechanisms we use with Kafka throughout our customer base. But they are not the only ones.

Kerberos and LDAP require Kafka clusters to integrate with back-end directory services, which are sometimes not available in some environments. In this case, you can still set authentication for the Kafka cluster using other methods, such as mutual TLS authentication or SASL / PLAIN with the back end of the password file.

That's all for "how to configure the Kafka client to use LDAP for authentication". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.