Shulou(Shulou.com)05/31 Report--

How to analyze the process of data transfer to the Kafka cluster after transfer, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Due to the current large number of overseas data nodes, but also more scattered, the original overseas data was directly written to the Kafka through the public network. Due to the current large number of overseas data nodes, but also more scattered, the original overseas data was directly written to the Kafka cluster through the public network. However, due to the problem of public network transmission in overseas networks, there is a very big delay. As a result, some businesses cannot be used directly. Such as abnormal alarm and so on.

Overseas data delay problem

We have a lot of nodes overseas, but they are very scattered. Maybe there are only a few machines in a country, and in order to realize the business of log analysis, overseas data needs to be transmitted back to the domestic data center. The current selection is transmitted through Kafka. Write the data to Kafka directly through the public network, and the domestic data center nodes consume the data of Kafka.

The delay of domestic public network transmission is also OK, because our own requirements for data delay are not too high. It is possible that some alarm services need to consider delay.

With the increase of alarm business, there are some requirements for the delay of overseas data, but in practical use, it is found that the delay of overseas data is very big.

Here is the state of ping at that time. You can see that the packet loss and delay are very high, and at the same time, we have carried out traceroute tracking, and found that the writing of some data has gone around the earth in half a circle and then returned to China (I really want to make a packet and travel around the world in a second)

The delay is different in different regions, the most serious areas are very high packet loss, and the delay is also very high.

Solution

Consulted the relevant manufacturers and gave three solutions. After the results of direct connect and overseas calculation are obtained, only the results are transmitted and transfer nodes are set up. At the end of the analysis, the scheme of the transfer node is selected.

We chose an area where the network condition is good for network testing, that is, to transfer data from other nodes to the servers in that area to check the network status. Finally, the test results show that the worst area of the network still has the delay of 200ms, but packet loss basically does not exist. The delay from the transit node to the Kafka node is about 30ms.

Ok, now set up a server on this node to transmit log data.

Set up nginx for data transfer

Because of the TCP protocol used by Kakfa, the construction of nginx is relatively simple, and the relevant data received can be directly transmitted through to the Kafka node. The configuration is as follows

Stream {

Upstream proxy_port {

Hash $remote_addr consistent

Server kafka_1:9092

Server kafka_1:9092

Server kafka_1:9092

}

Server {

Listen 9000

Proxy_connect_timeout 20s

Proxy_timeout 5m

Proxy_pass proxy_port

}

}

After the configuration is completed, the producer node is modified to write to port 9000 of nginx, and it is observed that the data has been successfully written through the transit node.

At the same time, the problem of data delay has been greatly improved, of course, there are still some nodes with higher delay of writing to transit nodes, but the situation of packet loss has been basically solved. Of course, there may still be problems with public network transmission. But the requirements have been basically met.

The transmission of cross-border data needs to be examined for legality.

This is the answer to the question on how to analyze the process of data transfer to the Kafka cluster after transit. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.