Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the example analysis of Kafka data transfer. Many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Demand origin

Because the data of some overseas nodes are sent to the Shanghai cluster of Kafka, there will be a high delay, because the data may be transferred many times during public network access, resulting in high network latency. Therefore, a regional transit is added, and the network condition of this area is better than that of all nodes (recommended by the manufacturer).

So I want to transfer the data through the region and then send it to Shanghai. From this point of view, the overall delay is about 120ms +. Therefore, the producer in Kafka directly configures host as the transit node. After testing, it is found that the data can be reached in Shanghai.

Imagined data flow

An exception occurs and the delay is still high

After several days of observation, it is found that the anomaly at the peak time is still very high, and the network situation between the nodes in the area at that time is still good. The delay from other areas to the transit area is about 90ms, and the delay from the transit area to the Shanghai node is about 30ms +. Modified the size and number of ping packets. It is found that the packet loss rate, delay and so on are all normal. Kafka data is still delayed.

Network reasons are basically excluded. Then it's something else.

Kafka stress test

Later, a server was purchased in a node in a certain area, and kafka-producer-perf-test was used for stress testing, and a large number of timeout appeared directly. The network situation is very bad.

But the result of ping is good.

View the program log

Look at the log sent by the process and find a problem on the log. Because I used nginx for forwarding, the port on which my data was sent to nginx was modified to 9000, that is, producer was configured with nginx's host:9000, but I found that what appeared in my log was still the port sent to kafka: 9092.

The process has not been restarted? This should be everyone's first reaction, so the restart is going on.

Restart the process and stare at the log.

Discovery request sent to port 9000 of nginx.. It should be right this time.

Then come out and set up a link kafka1:9092,kafka2:9092,kafka3:9092.

Look confused, restart is still the same. Looking at the network of the nginx server, the bandwidth is too low in recent days.

Find the reason

Based on the above problems, we can basically judge. The data sent by producer definitely did not pass through nginx. Query the relevant documents and find.

Kafka cannot transfer data through a nginx proxy, while nginx only connects for the first time. The producer node obtains the metadata information of the kafka through nginx. Then access it through the IP in metadata.

In other words, the actual way through nginx is like this. The first time it is sent is to get the information of metadata through nginx, and metadata contains the ip addresses of broker of kafka. Producer then sends the data directly to the kafka cluster.

After reading the above, do you have any further understanding of the example analysis of Kafka data transfer? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.