Shulou(Shulou.com)06/02 Report--

This article shows you why Kafka does not support the separation of reading and writing, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

In Kafka, the operations of producer writing message and consumer reading message interact with leader copy, and the slave realizes a production and consumption model of main writer and main read. Database, Redis and so on all have the function of main writing and main reading, and at the same time, they also support the function of primary writing and subordinate reading, which is the separation of reading and writing. In order to correspond to the main writing and primary reading, here we call it as primary writing and reading.

Kafka does not support master-write-from-read. Why is that?

At the code level, although it increases the code complexity, this feature can be fully supported in Kafka. For this problem, we can do a specific analysis from the perspective of "income point". The master write slave read can make the slave node share the load of the master node and prevent the situation that the master node is overloaded while the slave node is idle. But master-write-slave reading also has two obvious disadvantages:

Data consistency issues. When the data is transferred from the master node to the slave node, there must be a delayed time window, which will lead to data inconsistency between the master and slave nodes. At some point, the value of A data in both the master node and the slave node is X, and then the value of An in the master node is changed to Y, so before the change is notified to the slave node, the application reads the value of A data in the slave node is not Y, which leads to the problem of data inconsistency.

Delay problem. For components like Redis, the process of data writing to synchronization to slave node needs to go through the stages of network → master node memory → network → slave node memory, and the whole process will take a certain amount of time. In Kafka, master-slave synchronization is more time-consuming than Redis. It needs to go through the stages of network → master node memory → master node disk → network → slave node memory → slave node disk. For delay-sensitive applications, the function of master-write-slave reading is not very suitable.

In reality, many applications can tolerate both a certain degree of delay and data inconsistencies over a period of time!

So in this case, is it necessary for Kafka to support the read-from-write function?

Master write and slave read can share a certain load but can not achieve complete load balancing. For example, in the case of high data write pressure and low read pressure, the slave node can only share a small amount of load pressure, and most of the pressure is still on the master node. On the other hand, load balancing can be achieved to a large extent in Kafka, and this balance is realized on the architecture of main writer and main read. Let's take a look at Kafka's production and consumption model, as shown in the following figure:

There are three partitions in the Kafka cluster, and each partition has three replicas, which happens to be evenly distributed on the three broker. Grey shadows represent leader replicas, non-gray shadows represent follower replicas, and dotted lines indicate that follower replicas pull messages from leader replicas. When the producer writes the message, the leader copy is written. In the case in the figure above, each broker has a message flowing in from the producer; when the consumer reads the message, it is also read from the leader copy; in the case of figure 8-23, each broker has a message flowing out to the consumer.

We can clearly see that the read and write load on each broker is the same, which means that Kafka can achieve load balancing that can not be achieved by primary write and slave read. The figure above shows an ideal deployment scenario, where the following situations (including, but not limited to) can cause a certain degree of load imbalance:

(1) the partition distribution on the broker side is uneven. When creating a theme, there may be more partitions assigned to some broker and fewer partitions in other broker, which naturally results in uneven leader replicas.

(2) the producer writes messages unevenly. Producers may only do a lot of writing to leader copies in some broker, while ignoring leader copies in other broker.

(3) the information of consumer consumption is uneven. Consumers may only pull a lot of leader copies in some broker and turn a blind eye to leader copies in other broker.

(4) the switching of leader copies is uneven. In practical applications, the switching of master-slave copies or the redistribution of partition replicas may be caused by broker downtime, which may lead to uneven distribution of leader replicas in each broker.

In this regard, we can take some preventive measures.

In the first case, the partition allocation should be balanced as much as possible when the theme is created. Fortunately, the corresponding allocation algorithm in Kafka is also striving for this goal. If the allocation is customized by the developer, you need to pay attention to this aspect. For the second and third cases, the main write-slave reading can not be solved. For the fourth case, Kafka provides the election of priority copies to achieve the balance of leader replicas, at the same time, it can also cooperate with the corresponding monitoring, alarm and operation and maintenance platform to achieve balanced optimization.

In practical application, with the ecological platform combined with monitoring, alarm and operation and maintenance, Kafka can achieve load balancing to a large extent in most cases.

Generally speaking, Kafka only supports primary writing and primary reading. There are several advantages:

It can simplify the implementation logic of the code and reduce the possibility of errors; the granularity of the load can be refined and evenly distributed, compared with the main write and slave reading, not only the load performance is better, but also it is controllable to the user; there is no effect of delay

When the copy is stable, there will be no data inconsistency. For this reason, why should Kafka realize the function of master, write and read, which is of no benefit to it? All this is due to the excellent architectural design of Kafka. In a sense, the master writer and slave reader is a stopgap measure due to design defects.

The above is why Kafka does not support the separation of reading and writing. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.