Shulou(Shulou.com)05/31 Report--

Xiaobian to share with you what kind of Kafka communication protocol is, I believe most people still do not know how, so share this article for your reference, I hope you have a lot of harvest after reading this article, let's go to understand it together!

Kafka's Producer, Broker and Consumer use a self-designed TCP layer protocol. Kafka's protocol is completely customized for Kafka's own business requirements, rather than implementing a general protocol similar to Protocol Buffer.

basic data types

Fixed length data types: int8,int16,int32 and int64, which correspond to byte, short, int and long in Java.

Variable length data types: bytes and string. A variable-length data type consists of two parts, a signed integer N(indicating the length of the content) and N bytes of content. Where N =-1 means null. The length of bytes is denoted by int32, and the length of string is denoted by int16.

Array: An array consists of two parts, an array of length N and N elements represented by numbers of type int32.

Basic structure of Request and Response

The basic unit of communication between two roles in Kafka is Request/Response. The basic structure of Request and Response is as follows:

RequestOrResponse => MessageSize (RequestMessage | ResponseMessage)

The meaning of each field is:

MessageSizeint32 indicates the length of RequestMessage or ResponseMessage RequestMessage/ResponseMessage-indicates the content of Request or Response, and its specific format will be described below.

This structure defines the basic structure of data exchanged between communicating parties. The process of communication can be simply expressed as: the client opens the Socket with the server, then writes an int32 number to the Socket indicating how many bytes are sent this time, and then continues to write the corresponding number of bytes to the Socket. The server reads an integer of int32 to obtain the size of the Request, and then reads the data corresponding to the number of bytes to obtain the specific content of the Request. After the server processes the request, it sends a response in the same way.

Structure of RequestMessage

The structure of RequestMessage is as follows:

RequestMessage => ApiKey ApiVersion CorrelationId ClientId Request Name Type Description ApiKeyint16 indicates the API number of this request ApiVersionint16 indicates the API version of the request. After the version is available, backward compatibility can be achieved CorrelationIdint32. A number specified by the client uniquely identifies the id of this request. After processing the request, the server will also write the same CorrelationId to the Response, so that the client can correspond to a request and response. ClientIdstring A string specified by the client to describe the client, which will be used for logging and monitoring, uniquely identifies a client. Details of the Request. Structure of ResponseMessage

ResponseMessage is structured as follows:

ResponseMessage => CorrelationId Response Name Type Description CorrelationIdint32 CorrelationId corresponding to Request. Response-Response to Request, response fields are different for different requests. Message

Kafka is a distributed messaging system where producers produce messages and push them to brokers, and consumers pull messages from brokers. The message produced by Producer is represented by Message. For users, it is a key-value pair. Let's take a look at its structure.

Message => Crc MagicByte Attributes Key Value Name Type Description CRCint32 indicates the check code of this message (excluding CRC field itself) MagicByteint8 indicates the version of the message format, used for backward compatibility, the current value is 0Attributeint 8 indicates the metadata of this message, the lowest two bits are currently used to indicate the compressed format Keybytes indicates the Key of this message, and nullValuebytes indicates the Value of this message. Kafka supports message nesting, which means putting one message inside another as a Value. MessageSet

MessageSet is used to combine multiple Messages. It adds Offset and MessageSize to each Message. Its structure is:

MessageSet => [Offset MessageSize Message]

MessageSet is an array. Each element of the array consists of three parts, Offset, MessageSize and Message. Their meanings are:

Name Type Description Offsetint64 It is used as the sequence number in log. Producer does not know what the specific value is when producing the message. You can fill in any number. MessageSizeint32 indicates the size of this Message Message-indicates the specific content of this Message. See the previous section for its format. Compression of Message

Kafka supports the following compression methods,

Compression encoding uncompressed 0Gzip1Snappy2LZ43

where the code is the value of the least significant two digits of the Message Attribute.

Because there may not be much duplication in a single message, multiple messages are usually put together to form a MessageSet, and then the MessageSet is put into a Message to improve the compression ratio.

Request/Response and Message/MessageSet

Request/Response is the structure of the communication layer, which is similar to the TCP layer in contrast to the seven-layer model of the network.

Message/MessageSet defines the structure of the business layer, similar to the HTTP layer in the 7-layer network model. Message/MessageSet is just a data structure in the payload of Request/Response.

The above is all the content of this article "What is the Kafka communication protocol?" Thank you for reading it! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.