Shulou(Shulou.com)06/01 Report--

This article mainly introduces "the principle of the storage mechanism of kafka files". In the daily operation, I believe that many people have doubts about the principle of the storage mechanism of kafka files. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "the principle of storage mechanism of kafka files". Next, please follow the editor to study!

Each topic is divided into multiple partition, and each partition is a folder in the file system.

Each partition is divided into multiple segment, each segment, including 2 files, .index file and .log file, which are the index information and content of the message, respectively. And the name of the file looks like the following format

00000000000000000000.log

00000000000000000000.index

0000000000000999.log

0000000000000999.index

The file name is the sequence number of the message in this partition. Take the above file as an example, the first segment stores messages of 0999, while the second segment starts from 1000.

What we need to pay attention to is the division rules of segment. According to my research, it combines two factors. One is the size of the set segment. When it is full, there will be another segment. The second factor is the time factor. For example, if a topic does not have a message written for a long time, a new segment is likely to be built when a new message comes in. The purpose of this is to facilitate deletion of the entire segment when the message expires.

The index file contains two parts, namely the number of the message in this segment and the physical offset. It should be noted that instead of storing the index information of each message in the index file, a sparse index strategy is adopted, that is, an index is stored every few times.

When looking for a message, we first find the segment according to the name of the segment and the message number entered, and then find the specific location of the message according to the index file.

Writing process:

Messages are transferred from the java heap to page cache (that is, physical memory)

The disk is flushed by an asynchronous thread, and messages are brushed into the disk from page cache.

Reading process:

Messages are transferred directly from page cache to socket and sent out.

When no corresponding data is found from page cache, disk IO is generated, from disk Load message to page cache, and then sent directly from socket

Advantages of storage design:

Kafka divides a large parition file in topic into multiple small file segments. Through multiple small file segments, it is easy to regularly clear or delete consumed files and reduce disk footprint.

Message can be located quickly through index information.

By mapping all index metadata to memory, the IO disk operation of segment file can be avoided

Through the sparse storage of index files, the space occupied by index file metadata can be greatly reduced.

At this point, the study of "the principle of the storage mechanism of kafka files" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.