Shulou(Shulou.com)06/03 Report--

This article mainly explains "what is the role of Kafka buffering mechanism". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the role of Kafka buffering mechanism"?

1. Client-side buffering mechanism of Kafka

There must be a memory buffering mechanism when the client sends a message to the kafka server. That is, the message is first written to a memory buffer, and then the Batch is not sent in a network communication until multiple messages form a Batch.

2. Frequent GC problems caused by memory buffering

The original intention of this memory buffering mechanism is to compose multiple messages into a Batch, and a network request is a Batch or multiple Batch. In this way, a lot of data can be sent to each network request, avoiding a message and a network request. As a result, the throughput is improved, that is, the amount of data sent per unit time.

But the problem is, you can think about it. The data in a Batch will be taken out and encapsulated in the underlying network packet and sent through the network to the Kafka server. All the data in this Batch has been sent, what should we do with the data in Batch now? The data in the Batch can still be in the JVM memory of the client at this time. At this time, from the code implementation level, we must try to avoid any variables to reference the data corresponding to these Batch, and then try to trigger JVM to automatically collect the memory garbage.

In this way, by constantly letting JVM collect garbage, you can constantly clean up the Batch that has been successfully sent, and then you can constantly free up new memory space for later new data to use. This is a good idea, but there must be problems when running online, and the biggest problem is the JVM GC problem. When JVM GC collects memory garbage, he will have a "Stop the World" process, that is, when the garbage collection thread is running, it will cause other worker threads to pause briefly, which makes it easier for him to collect memory garbage quietly.

This is also easy to figure out, after all, if your worker thread is still writing data into memory while you are collecting memory garbage, how do you ask JVM to collect garbage? This is like on the road, if there is a lot of rubbish on the ground, what is the best way to sweep all the rubbish now? Everybody out of the way, clear the road, and then the cleaner is to clean up the rubbish. But if the cleaners are cleaning garbage and a group of people keep eating melon seeds and throwing melon shells, eating watermelons and throwing watermelon peels, how do you think the cleaners feel? Of course, it is very indignant, if you do it in this way, the rubbish on the ground will never be clean!

Now JVM GC is more and more advanced, from the CMS garbage collector to the G1 garbage collector, one of the core goals is to constantly reduce the time of garbage collection, causing other worker threads to stop. So now the newer the garbage collector, the shorter the worker thread stops, but no matter how short it is, it still exists! And how to avoid the frequent GC of JVM in your design as much as possible is a very test of the level.

3. Buffer pool mechanism implemented by Kafka designers.

Within the Kafka client, an excellent mechanism has been implemented for this problem, which is the buffer pool mechanism. Each Batch layer corresponds to a piece of memory, which is dedicated to storing messages written into it. Then when a Batch is sent to the kafka server, the Batch's data is no longer needed, which means that the Batch's memory space is no longer used. At this time, do not give the underlying memory space of Batch to JVM for garbage collection, but put this memory space into a buffer pool. There are many blocks of memory in this buffer pool. Next time you have a new Batch, can't you get a piece of memory space directly from this buffer pool and ok it? Then if a Batch is sent out, and then give the memory space back to others, wouldn't it be all right? And so on, over and over again.

Thank you for your reading, the above is the content of "what is the role of Kafka buffering mechanism". After the study of this article, I believe you have a deeper understanding of the role of Kafka buffering mechanism, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.