Shulou(Shulou.com)06/01 Report--

In this article, the editor introduces in detail "how to configure the monitoring address of Kafka". The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to configure the monitoring address of Kafka" can help you solve your doubts.

Currently, the main parameters related to monitoring are as follows:

Listeners

Advertised.listeners

Listener.security.protocol.map

Inter.broker.listener.name

Security.inter.broker.protocol

Advertised.host.name (legacy, obsolete, do not use)

Advertised.port (legacy, obsolete, do not use)

Host.name (legacy, obsolete, do not use)

The most important of these are listeners and advertised.listeners: listen to the address configured by listeners during cluster startup and write the address configured by advertised.listeners to Zookeeper as part of the cluster metadata. We can divide the process of connecting the client (producer / consumer) to the Kafka cluster into two steps:

Connect to a Broker through the connection information (ip/host) configured by listeners (broker periodically acquires and caches the metadata information in the zk) to obtain the address information of the advertised.listeners configuration in the metadata.

The advertised.listeners connection information obtained in step 1 and Kafka cluster communication (read / write).

Therefore, in virtualized environments with private and public network isolation (such as Docker and public cloud), external clients often have the ability to connect to Kafka (step 1), but send / consume data Times connection timeout (step 2) because listeners is configured with a public network address, while advertised.listeners is configured with a private network address. So how to configure these parameters?

Let's first look at the configuration format of the connection information: {listener name}: / / {HOST/IP}: {PORT}. HOST/IP and PORT are very clear, mainly the "listener name" field. To understand this, you need to understand the listener.security.protocol.map configuration item: its purpose is to configure the mapping of the listener name to the protocol (so it is a map of key-value), key is the "listener name", value is the "protocol name", and the default value is the same as the "listener name" and the "protocol name". For example, PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL, the colon is preceded by key, that is, the protocol name, and followed by value, that is, the protocol name. The listener name can be chosen at will, while the protocol name is a fixed enumerable range. So if we customize the listener name, we need to explicitly set its corresponding protocol name.

Both inter.broker.listener.name and security.inter.broker.protocol are used to configure communication between Broker, the former configuration name (that is, key in listener.security.protocol.map) and the latter configuration protocol (that is, value in listener.security.protocol.map), and the default value is PLAINTEXT. Only one of these two configuration items can be configured at a time.

Why does a connection have to be so complicated? Mainly for a variety of different scenario requirements. Here is a more complex application scenario to illustrate. For example, we deploy a Kafka cluster on a public cloud. The environment has a public network address external_hostname and a private network address internal_hostname;, and the public network address cannot be obtained internally (this is mostly the case with public clouds). Then you want to realize that the internal client accesses the cluster with the internal address without encryption, while the external client accesses the cluster with the external address and needs encryption. To achieve this requirement, you can configure the cluster as follows:

Listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSLlisteners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092advertised.listeners=INTERNAL:// {internal_hostname}: 19092 Magazine EXTERNALVERVERLARVERGER / {external_hostname}: 9092inter.broker.listener.namecodes INTERNALlistener.security.map.mappings INTERNALRVPLINTEXT.EXTERNALSwitzerland SSL

In fact, we can also customize the connection between the cluster Controller node and other Broker nodes through the optional control.plane.listener.name parameter, and the configuration information will be as follows:

Listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL,CONTROL:SSLlisteners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092advertised.listeners=INTERNAL:// {internal_hostname}: 19092,EXTERNAL:// {external_hostname}: 9092,CONTROL:// {control_ip}: 9094inter.broker.listener.name=INTERNALcontrol.plane.listener.name=CONTROL

Finally, the default values of these configuration items and some considerations are given:

If listeners is not explicitly configured, it will listen on all network cards, which is equivalent to 0.0.0.0 configured. The listeners name and port in this configuration item must be unique and cannot be repeated.

If advertised.listeners is not configured, the value configured by listeners is used by default. If listeners is also not explicitly configured, the IP address obtained by java.net.InetAddress.getCanonicalHostName () is used. If listeners is configured with 0.0.0.0, advertised.listeners must be explicitly configured because the configuration item must be a specific address and 0.0.0.0 is not allowed (because the client cannot connect to the Broker based on this address). In addition, duplicate ports are allowed in advertised.listeners.

For listeners and advertised.listeners, when there are multiple addresses, each address must be configured in the format {listener name}: / / {HOST/IP}: {PORT}, separated by English commas.

If the hostname of all nodes in the cluster can be resolved correctly between the client and server nodes, hostname is preferred over IP. Because java.net.InetAddress.getCanonicalHostName () is used in the code, sometimes there is a lack of access when using IP.

After reading this, the article "how to configure the listening address of Kafka" has been introduced. If you want to master the knowledge of this article, you still need to practice and use it yourself. If you want to know more about related articles, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.