Shulou(Shulou.com)05/31 Report--

This article shows you what are the two major tools of kafka to solve the search efficiency, the content is concise and easy to understand, can definitely brighten your eyes, through the detailed introduction of this article, I hope you can get something.

Segmentation of data files

One of the ways Kafka solves query efficiency is by segmenting data files, such as 100 Message, whose offset ranges from 0 to 99. Suppose you divide the data file into five segments, the first segment is 0-19, the second segment is 20-39, and so on, each segment is placed in a separate data file named after the smallest offset in that paragraph. In this way, when looking for the Message of the specified offset, you can use a binary lookup to locate which segment the Message is in.

Index data files

Data file segmentation makes it possible to find the Message of the corresponding offset in a smaller data file, but this still requires sequential scans to find the Message of the corresponding offset. In order to further improve the efficiency of lookup, Kafka establishes an index file for each segmented data file, and the file name is the same as the name of the data file, except that the file extension is .index.

The index file contains several index entries, each representing the index of an Message in the data file. The index contains two parts (both 4-byte numbers), which are relative offset and position, respectively.

Relative offset: because after the data file is segmented, the starting offset of each data file is not 0, and the relative offset represents the size of the Message relative to the smallest offset in the data file to which it belongs. For example, if the offset of a segmented data file starts at 20, then the relative offset of a Message with an offset of 25 in the index file is 25-20 = 5. Storage relative to offset can reduce the space occupied by index files.

Position, indicating the absolute position of the Message in the data file. Just open the file and move the file pointer to the position to read the corresponding Message.

Instead of indexing every Message in the data file, the index file uses sparse storage to build an index every other byte of data. This prevents the index file from taking up too much space, so that the index file can be kept in memory. But the disadvantage is that the Message that is not indexed can not locate its location in the data file at once, so it needs to do a sequential scan, but the scope of this sequential scan is very small.

In Kafka, the implementation class of the index file is OffsetIndex, and its class diagram is as follows:

The main methods are:

Append method, add a pair of offset and position to the index file, where the offset will be converted to the relative offset.

Lookup, which uses binary search to find the largest offset that is less than or equal to a given offset

What are the two major tools used by kafka to solve the search efficiency? have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.