Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of Kafka message specification, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.

As a message queue, Kafka has its own defined message format. Messages in Kafka use ByteBuf, and ByteBuf, a tight binary storage format, is adopted because it saves a lot of space. After all, defining a message object in the format of the Java class will waste a lot of space (there is some Header and some complement for the Java object in addition to the space occupied by its own properties).

V2 message format

The message format of Kafka has gone through V0, V1, and V2 versions. V0 does not have a timestamp field, making it difficult to judge expired messages. There are many fixed-length fields in V0 and V1, which often occupy very little in practice, resulting in waste, so V2 designs many of the fields with defined length as variable length.

The variable length design borrows from the Zig-zag encoding format, and the highest bit is used to indicate whether the current byte is already the last byte of a number encoding (1 for no, 0 for yes).

Total message length: the length of the entire message to facilitate the traversal of the message and to obtain its total length

Attribute: reserved field, temporarily inactive

Timestamp increment: the increment of the message from the Batch timestamp. Instead of using a fixed 8-byte timestamp, this field will greatly reduce the storage space of the message.

Displacement increment: increment of message distance Batch displacement

Key length: the length of the message key content

Key: the content of the message key

Value size: the length of the message content

Value: message content

Number of header:header

Heders: a specific header that is visible to users and can be used as a carrier for message routing or some message metadata.

V2 message batch format RecordBatch

A message batch consists of several messages. in fact, the log file of Kafka is composed of several message batches. Kafka does not operate directly at the message level, it is always written at the message batch level.

Start shift: offset in Kafka log partition

Length: the length of the message batch

Partition leader version number

Version number: currently this value is 2

CRC:CRC check code is used to confirm that the message will not be tampered with during transmission. This field is at the message level in V0 and V1, but CRC each message will result in a waste of CPU.

Attribute: this field also exists at the message level in the versions of V0 and V1. In V2, the lower three bits still represent the compression type of the message, and the fourth bit is still the timestamp type (one is the timestamp specified by the client, the other is the timestamp specified by kafka broker). Bits 5 and 6 represent the newly introduced transaction type and control type, respectively.

Start timestamp: the timestamp of the first message in the batch

Maximum timestamp: the timestamp of the last message in the batch

PID, producer epoch, starting sequence number: the introduction of sequence number in order to produce the idempotence of the message, Kafka uses it to determine whether the message has been submitted and to prevent repeated production of the message. The ID,producer epoch of PID stands for idempotent producer indicates the current version number carried by producer. Broker uses these two fields to determine whether producer is valid and to prevent expired producer production messages.

Thank you for reading this article carefully. I hope the article "sample Analysis of Kafka message Specification" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.