Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about how to analyze the performance optimization of Kafka. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Kafka has made great efforts to improve efficiency. One of the main usage scenarios of Kafka is to deal with website activity logs, and the throughput is very large, with many writes per page. In terms of reading, it is assumed that each message is consumed only once, and the amount of reading is also very large, and Kafka also tries to make the read operation lighter.

We discussed disk performance issues earlier, and there are about two aspects that affect disk performance in the case of linear reads and writes: too many trivial Imax O operations and too many byte copies. The Icano problem occurs between the client and the server, as well as in the persistent operation within the server.

Message set (message set)

To avoid these problems, Kafka established the concept of "message set", which organizes messages together as a unit of processing. Processing messages in terms of message sets can improve a lot of performance compared to processing messages in individual units. Producer sends the message set to the server instead of sending it one by one; the server appends the message set to the log file at one time, thus reducing the trivial Ibank O operations. Consumer can also request a message set at one time.

Another performance optimization is in terms of byte copying. This is not a problem in the case of low load, but its impact is still great in the case of high load. To avoid this problem, Kafka uses the standard binary message format, which can be shared between producer,broker and producer without any changes.

Zero copy

The message logs maintained by Broker are just directory files, and the message set is written to the log file in a fixed queue format, which producer and consumer share, which allows Kafka to optimize at a very important point: message delivery over the network. The modern unix operating system provides a high-performance system function to send data from the page cache to socket. In linux, this function is sendfile.

To better understand the benefits of sendfile, let's first take a look at the data flow that typically sends data from a file to socket:

The operating system copies data from the file to the page cache in the kernel

Applications from the page cache from the memory cache that copies data to itself

The application writes data to the socket cache in the kernel

The operating system copies the data from the socket cache to the Nic interface cache and sends it to the network from here.

This is obviously inefficient, with 4 copies and 2 system calls. Sendfile avoids duplicate copies and greatly optimizes performance by sending data directly from the page cache to the Nic interface cache.

In a multi-consumers scenario, the data is copied to the page cache only once rather than repeated each time a message is consumed. This allows messages to be sent at a rate close to network bandwidth. In this way, you can hardly see any read operations at the disk level, because the data is sent directly to the network from the page cache.

This article introduces in detail the application of sendfile and zero-copy technology in Java.

Data compression

In many cases, the bottleneck of performance is not CPU or hard disk but network bandwidth, especially for applications that need to transfer large amounts of data between data centers. Of course, users can compress their own messages without Kafka support, but this will result in a lower compression ratio, because compressing a large number of files together works best than compressing messages individually.

Kafka uses end-to-end compression: because of the concept of "message set", messages from the client can be compressed and sent to the server together, and the log file can be written in the compressed format and sent to consumer in a compressed format. Messages sent from producer to consumer are compressed and decompressed only when consumer is used, so it is called "end-to-end compression".

Kafka supports GZIP and Snappy compression protocols.

The above is the editor for you to share how to parse Kafka performance optimization, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.