Shulou(Shulou.com)06/01 Report--

Editor to share with you how to read and write copy of the Kafka message, I believe that most people do not know much about it, so share this article for your reference, I hope you will learn a lot after reading this article, let's go to know it!

AppendRecords- copy write

The logic for writing messages to the underlying log of the replica is implemented in ReplicaManager#appendRecords.

Scenarios in which Kafka needs to be written by a copy:

The producer writes a message to the Leader copy

The Follower copy is written to the copy after pulling the message.

Only this scenario calls the methods of the Partition object, and the other three are done by calling appendRecords.

Consumer groups write group information

The transaction manager writes transaction information (including transaction tags, transaction metadata, etc.)

The appendRecords method writes messages for a given set of partitions to the corresponding Leader replicas and selectively waits for other replicas to finish writing according to the settings of acks in the PRODUCE request. The specified callback logic is then invoked.

The process by which appendRecords writes a message to the copy log:

Execution process

Visible, appendRecords:

The way to write messages is appendToLocalLog

Method delayedProduceRequestRequired to determine whether it is necessary to wait for other copies to be written

AppendToLocalLog writes to the replica local log

Writing a message set using Partition#appendRecordsToLeader is written to the local log using the appendAsLeader method.

DelayedProduceRequestRequired

Determine whether you need to wait for other copies to be written successfully after the message set has been written to the log:

Private def delayedProduceRequestRequired (requiredAcks: Short, entriesPerPartition: Map [TopicPartition, MemoryRecords], localProduceResults: Map [TopicPartition, LogAppendResult]): Boolean = {requiredAcks = =-1 & & entriesPerPartition.nonEmpty & & localProduceResults.values.count (_ .substitution.isDefined) < entriesPerPartition.size}

If you wait for other copies to be written, you must meet at the same time:

RequiredAcks==-1

There is still data that has not yet been written.

At least one partition message has been successfully written to the local log

2 and 3 can be combined. If data writes to all partitions are unsuccessful, a serious error may occur, and instead of waiting, you should return the error directly to the sender.

On the other hand, some partitions are written successfully and some partitions fail, which may be caused by occasional instantaneous errors. At this point, you might as well put this write request into Purgatory and give it a chance to retry.

Copy reading: fetchMessages

ReplicaManager#fetchMessages is responsible for reading copy data. Regardless of:

Java consumers

APIFollower copy

The main way to pull a message is to send a FETCH request to Broker. After the broker receives the request, it calls fetchMessages to retrieve the message from the underlying Leader copy.

FetchMessages may also delay processing FETCH requests because the Broker side must accumulate enough data before returning the Response to the request sender.

The whole method is divided into:

Read local log

First of all, it is judged that the requestor who reads the message can determine the readable range.

FetchIsolation, read isolation level:

For the Follower copy, it can read all messages below the LEO value of the Leader copy

For an ordinary Consumer, you can only "see" messages below the high water level of the Leader copy.

Once the readable scope is determined, readFromLog is called to read the message data on the local log and assign the result to the logReadResults variable. ReadFromLog calls readFromLocalLog and in turn invokes the read method of its log object on the partition to be read to perform the actual message reading.

Determine the Response based on the reading result

Create the corresponding Response based on the reading result of the previous step:

According to the reading result obtained in the previous step, count the total number of bytes that can be read, and then determine whether Reponse can be returned immediately at this time.

The replica manager's two methods, appendRecords and fetchMessages, essentially call Log's append and read methods at the bottom to read and write local logs. After completing the read and write operation, these two methods also define the conditions for deferred processing. Once the delay processing condition is met, it is handed over to the corresponding Purgatory for processing.

From these two methods, we can see the trend of individual components merging together. Although the order in which we learn individual source files is top-down, the path to concatenating the functions of the main components of Kafka is bottom-up.

In the case of a copy write operation, the append method of the log object is called by the method in the Partition object at the previous level, which is further called by the method in the copy manager. We read the code of individual components such as copy manager, log object and so on from top to bottom to understand their independent functions.

Now start slowly merging them together to build the full call path of the Kafka operation partition copy log object. If you use these two ways to read the source code at the same time, you can understand the principle of Kafka more efficiently.

Summary

Kafka replica the core method of ReplicaManager read-write copy of the state machine class:

AppendRecords: write a message to the replica, and use the Log#append method and Purgatory mechanism to realize the data synchronization operation after the Follower replica obtains the message from the Leader copy.

FetchMessages: reads messages from replicas, used by normal Consumer and Follower replicas. When they send a FETCH request to Broker, the copy manager on Broker calls this method to get the specified message from the local log

The above is all the contents of the article "how to read and write copy message from Kafka". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.