Shulou(Shulou.com)06/03 Report--

Summary:

Kafka, a distributed message queue, uses the file system and the operating system's page cache (page cache) to store and cache messages respectively, abandons the heap cache mechanism of Java, and changes random writes to sequential writes, combined with the characteristics of Zero-Copy to greatly improve the performance of IO. " The linear write speed of a SATA RAID-5 array disk can reach hundreds of KB/s s, while the random write speed can only be more than 100ms, and the linear write speed is thousands of times faster than the random write speed, so we can see that the speed of writing messages to the disk depends on our usage.

Kafka can also provide a more flexible way to handle messages by configuring users to decide when persistent messages are saved. This article will mainly introduce the message structure of data storage in Kafka, how to store it, and how to find messages through offset.

-

-

1. Introduction of several important concepts in Kafka.

(1) Broker: message middleware processing node. A Kafka node is a broker, and one or more Broker can form a Kafka cluster.

(2) Topic: a topic is an abstract classification of a group of messages, such as page view logs, click logs, etc., which can be abstractly classified in the form of topic. Physically, messages of different Topic are stored separately. Logically, messages of a Topic are stored on one or more broker, but users only need to specify the Topic of the message so that the producer or consumer of the data does not have to care about where the data is stored.

(3) Partition: each topic is divided into one or more Partition. Each partition corresponds to a folder on the local disk. The partition naming rule is the topic name followed by the "-" connector, followed by the partition number, which starts from 0 to the total number of partitions minus-1.

(4) LogSegment: each partition is divided into multiple log segments (LogSegment), and the log segment is the smallest unit of Kafka log object segmentation; LogSegment is a logical concept, which corresponds to a specific log file (".log" data file) and two index files (".index" and ".timeindex", representing offset index file and message timestamp index file, respectively).

(5) Offset: each partition consists of a series of ordered, immutable messages that are sequentially appended to the partition. Each message has a contiguous sequence number called offset- offset, which is used to uniquely identify the message within the partition (does not indicate the physical location of the message on disk).

(6) Message: a message is the smallest and most basic unit stored in Kafka, that is, a commit log, which consists of a fixed-length header and a variable-length message body.

2. Log structure and data storage of Kafka

Messages in Kafka are organized on the basis of Topic, and each topic is independent of each other. Here, the topic is only a logical abstraction, while in the storage of actual data files, messages in Kafka are physically composed of one or more Partition, each corresponding to a folder on the local disk, and each folder contains log index files (".index" and ".timeindex") and log data files (".log"). The number of partitions can be specified when the theme is created or modified after the Topic is created. (the number of Partition of ps:Topic can only be increased but not decreased, which is beyond the scope of the reduction of this article. You can think about it first.)

Because the design model of Partition is used in Kafka, the high throughput of message processing is achieved by breaking the messages of topic (Topic) into multiple partitions and storing them on different Kafka Broker nodes. Both producers and consumers can operate in parallel with multiple threads, and each thread processes a partition of data.

At the same time, in order to achieve high availability of the cluster, Kafka can set up one or more replicas (Replica) in each Partition, and the replicas of partitions are distributed on different Broker nodes. At the same time, a copy is selected from the copy as a copy of the Leader,Leader to read and write to the client. Other replicas as Follower synchronize data from Leader replicas.

2.1.Analysis of log file storage of partition / replica in Kafka

Create 3 copies of topic

. / kafka-topics.sh-- create-- zookeeper 10.154.0.73 zookeeper 2181-- replication-factor 3-- partitions 3-- topic kafka-topic-01./kafka-topics.sh-- describe-- zookeeper 10.154.0.73 VR 2181-- topic kafka-topic-01

Each partition physically corresponds to a folder. The naming rule of the partition is the topic name followed by the "-" connector, followed by the partition number. The partition number starts at 0, and the maximum number is the total number of partitions minus 1. Each partition has one or more replicas, and the replicas of the partition are distributed on different agents in the cluster to improve availability. From a storage point of view, each copy of the partition can be logically abstracted as a Log object, that is, the partition copy corresponds to the log object. The following figure shows the physical distribution of the primary / backup replicas of the partition in a cluster of three Kafka Broker nodes:

2.2.Storage structure of log indexes and data files in Kafka

In Kafka, each Log object can be divided into multiple LogSegment files, and each LogSegment file includes a log data file and two index files (offset index file and message timestamp index file). Where the log data files in each LogSegment are equal in size (the size of the log data file can be set by "log.segment.bytes" in Kafka Broker's config/server.properties configuration file, default to 1G size (1073741824 bytes), and a new set of log data and index files will be created if messages are written sequentially beyond this set threshold).

Kafka encapsulates the log file into a FileMessageSet object and encapsulates the offset index file and the message timestamp index file into OffsetIndex and TimerIndex objects respectively. Both Log and LogSegment are logical concepts. Log is the abstraction of the file stored on the Broker, while LogSegment is the abstraction of each log segment under the replica storage, and the log and index files correspond to the physical storage on disk. The following figure shows the corresponding relationship between objects in the Kafka log storage structure:

To further view the ".index" offset index file, the ".timeindex" timestamp index file, and the ".log" log data file, execute the following command to convert the binary segmented index and log data file contents to character files:

# 1. Execute the following command to dump the contents of the log data file. / kafka-run-class.sh kafka.tools.DumpLogSegments-- files / apps/svr/Kafka/kafkalogs/kafka-topic-01-0/00000000000022372103.log-- print-data-log > 00000000000022372103_txt.log#2, Specific log data content Dumping / apps/svr/Kafka/kafkalogs/kafka-topic-01-0/00000000000022372103.logStarting offset: 22372103offset: 22372103 position: 0 CreateTime: 1532433067157 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 5d2697c5-d04a-4018-941d-881ac72ed9fdoffset: 22372104 position: 0 CreateTime: 1532433067159 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec NONE producerId: :-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 0ecaae7d-aba5-4dd5-90df-597c8b426b47offset: 22372105 position: 0 CreateTime: 1532433067159 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 87709dd9-596b-4cf4-80fa-d1609d1f2087.offset: 22372444 position: 16365 CreateTime: 1532433067166 isvalid true: Keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 isTransactional: false headerKeys: [] key: 1 payload: 8d52ec65-88cf-4afd-adf1-e940ed9a8ff9offset: 22372445 position: 16365 CreateTime: 1532433067168 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 5f5f6646-d0f5-4ad1-a257-4e3c38c74a92offset: 22372446 position: 16365 CreateTime: 1532433067168 isvalid: true Keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 51dd1da4-053e-4507-9ef8-68ef09d18ccaoffset: 22372447 position: 16365 CreateTime: 1532433067168 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: 80d50a8e-0098-4748-8171-fd22d6af3c9b. Offset: 22372785 position: 32730 CreateTime: 1532433067174 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload: db80eb79-8250-42e2-ad26-1b6cfccb5c00offset: 22372786 position: 32730 CreateTime: 1532433067176 isvalid: true keysize: 4 valuesize: 36 magic: 2 compresscodec: NONE producerId:-1 producerEpoch:-1 sequence:-1 isTransactional: false headerKeys: [] key: 1 payload 51d95ab0-ab0d-4530-b1d1 -05eeb9a6ff00.#3, Similarly The specific offset index content Dumping / apps/svr/Kafka/kafkalogs/kafka-topic-01-0/00000000000022372103.indexoffset: 22372444 position: 16365offset: 22372785 position: 32730offset: 22373467 position: 65460offset: 22373808 position: 81825offset: 22374149 position: 98190offset: 22374490 position: 114555.04, Dumping / apps/svr/Kafka/kafkalogs/kafka-topic-01-0/00000000000022372103.timeindextimestamp: 1532433067174 offset: 22372784timestamp: 1532433067191 offset: 22373466timestamp: 1532433067206 offset: 22373807timestamp: 1532433067214 offset: 22374148timestamp: 1532433067222 offset: 22374489timestamp: 1532433067230 offset: 22374830.

The specific contents of the offset index file and log data file from the above dump can be analyzed. The offset index file stores a large number of index metadata, and the log data file stores a large number of fields in the message structure and the value of the message body itself. The metadata postion field in the index file points to the actual location (that is, the physical offset address) of the message in the corresponding log data file.

The following table first lists the descriptions of several main fields in the body structure of the Kakfa message:

Each field of the Kafka message field describes the offset message offset message size message total length CRC32CRC32 encoding checksum attributes represents as a stand-alone version, or identifies the compression type, or the encoding type magic represents the length of the Kafka service program protocol version number key length message Key the actual data length of the valuesize message the actual data length of the playload message the actual data summary of the playload message

From the perspective of the full text, Kafka efficient data storage design is characterized by the following points:

(1) Kafka divides a partition in the topic into multiple small file segments. Through multiple small file segments, it is easy to find messages according to the offset, regularly clear and delete data files that have been consumed, and reduce the occupation of disk capacity.

(2) the offset index file of the log is built by sparse index storage and mapped to memory to improve the efficiency of finding messages and reduce disk IO operations.

(3). Kafka changes the operation logic of message append into the sequential writing of log data files, which greatly improves the performance of disk IO.

For any student who uses Kafka, if you can master its data storage mechanism, it will be of great benefit to the performance tuning and problem location of large-scale Kafka clusters.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.