Shulou(Shulou.com)06/03 Report--

This article mainly explains "what is the so-called zero-copy technology in Kafka". The explanation in the article is simple and clear and easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the so-called zero-copy technology in Kafka".

In addition to message sequential append, page cache and other technologies, Kafka also uses zero-copy technology to further improve performance. The so-called zero copy means that the data is copied directly from the disk file to the network card device without going through the application. Zero copy greatly improves the performance of the application and reduces the context switching between kernel and user mode. For Linux operating systems, zero-copy technology depends on the underlying implementation of the sendfile () method. Corresponding to the Java language, the underlying implementation of the FileChannal.transferTo () method is the sendfile () method.

It is abstract to understand "zero copy" conceptually. Here is a brief introduction to it. Consider a common situation where you need to show static content (like pictures, files) to the user. This situation means that the static content needs to be copied from disk and put into a memory buf, and then the buf is transferred to the user through a Socket, so that the user can get the static content. This seems perfectly normal, but in fact it is a very inefficient process, and we abstract the above situation into the following process:

Read (file, tmp_buf, len); write (socket, tmp_buf, len)

First call read () to read the static content (assumed here as file A) to tmp_buf, and then call write () to write tmp_buf to Socket, as shown in the following figure. Phellodendron mandshurica (Thunb.)

In this process, file A goes through four copies:

When read () is called, the contents of file An are copied to Read Buffer in kernel mode.

CPU controls the replication of kernel mode data to user mode.

When write () is called, the contents in user mode are copied to Socket Buffer in kernel mode.

Copy the data of Socket Buffer in kernel mode to the network card device for transmission.

As can be seen from the above process, the data has "walked around" from kernel mode to user mode for no reason, wasting two replication processes: the first is from kernel mode to user mode; the second is to copy from user mode back to kernel mode, that is, steps 2 and 3 of the above four processes. And in the above process, the context of the kernel and user mode is also switched 4 times.

If zero-copy technology is used, the application can directly request the kernel to transfer the data from the disk to Socket, as shown in the following figure.

Zero copy technology copies the contents of files to Read Buffer in kernel mode through DMA (Direct Memory Access) technology. However, no data is copied to the Socket Buffer; instead, only file descriptors containing information about the location and length of the data are added to the Socket Buffer. The DMA engine passes data directly from kernel mode to the network card device (protocol engine). Here, the data is transferred from the disk after only 2 replications, and the context switch becomes 2 times. Zero copy is for kernel mode, and zero copy of data is implemented in kernel mode.

To master the core implementation principle of Kafka, not only the log format, log index, log cleaning and other aspects of Kafka, but also the knowledge points related to the underlying physical storage are included.

Thank you for your reading, the above is the content of "what is the so-called zero-copy technology in Kafka". After the study of this article, I believe you have a deeper understanding of what the so-called zero-copy technology in Kafka is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.