Shulou(Shulou.com)06/02 Report--

This article is about how to achieve cross-network access in Kafka. The editor thinks it is very practical, so I hope you can get something after reading this article. Let's take a look at it with the editor.

The mode of communication is as follows:

Test conditions:

CDH and CM are 6.3.3

Kafka version 2.2.1

Kerberos is enabled for both the cluster and Kafka

The intranet IP format is 192.168.1.*

The public network IP format is 192.168.0.*

Kafka cluster three machines cdh [1-3] .hadoop.com, public network client cdh02.hadoop.com

Configuration step

2.1 configure hosts

1. The mapping between the host name and private network ip configured by the nodes in the cluster is as follows:

two。 The mapping between the host name and the public network ip configured on the node outside the cluster is as follows:

3. Verify the internal and external network environment here

As you can see from the figure above, the client node of the external network can only access 192.168.0.clients, but not 192.168.1.clients, and port 9092 of the telnet internal network is also impassable.

2.2 configure parameters related to Kafka

1. Go to the Kafka configuration page of CM and search for kafka.properties

two。 Configure Broker according to three different role groups

The configuration is complete at this point.

Note: advertised_listeners is an exposed service port. The real connection is based on the listening configured in listeners,listeners. If it is not published to advertised_listeners, it cannot be accessed directly, and the configuration in advertised_listeners must have been configured in listeners.

Verify Kafka usage

1. Create a topic with the name faysonTopic in the nodes in the cluster and produce some messages.

two。 Nodes in the cluster consume topic in an unsecure way, using port 9092

3. The nodes outside the cluster use the external network environment to consume topic in a secure way, using port 9797

Export KAFKA_OPTS= "- Djava.security.auth.login.config=./jaas-keytab.conf"

Kafka-console-consumer-topic faysonTopic-from-beginning-bootstrap-server cdh2.hadoop.com:9797,cdh3.hadoop.com:9797,cdh4.hadoop.com:9797-consumer.config client.properties

The screenshot of the configuration file used is as follows:

1. For listeners=PLAINTEXT://ip:9092,SASL_PLAINTEXT://:9797 configuration, port 9092 used for secure access is only bound to private network ip, and port 9797 is bound to all ip, so that the public network environment can only access Kafka through secure protocols, as follows:

two。 For advertised.listeners= SASL_PLAINTEXT://hostname:9797 configuration, mainly through domain name mapping, through the client configuration of the public network IP and domain name hosts mapping, to ensure that the clients of the public network can access the Kafka cluster.

The above is how to achieve cross-network access in Kafka. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.