Shulou(Shulou.com)06/03 Report--

Today, I would like to talk to you about what is the second pulse of the governor of Kafka architecture design. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can gain something according to this article.

What on earth is the second pulse of Kafka architecture design?

Grasp this key point, I believe you will be able to better understand the architectural design of Kafka, and then grasp the core technical solutions of Kafka.

1. What are the technical difficulties of Kafka?

The previous article, "removing the mysterious veil of Kafka", explained two key messages:

1. Kafka is created for real-time log flow, and the amount of concurrency and data to be processed is very large. It can be seen that Kafka itself is a high concurrency system, and it is bound to encounter three typical high challenges in high concurrency scenarios: high performance, high availability and high scalability.

2. In order to simplify the complexity of the implementation, Kafka finally adopts a very ingenious message model: it persists all messages, allowing consumers to take whatever message they want, and can take any message they want. You only need to pass a message's offset to pull it.

Eventually Kafka degenerated itself into a "storage system". Therefore, the storage of massive messages is the biggest technical difficulty in the design of Kafka architecture.

2. The second pulse of Kafka architecture design.

Let's move on to the next analysis: how on earth does Kafka solve the storage problem?

In the face of huge amounts of data, the storage capacity and read-write performance of a single computer must be limited, so it is easy to think of a storage scheme: data storage in pieces. This kind of scheme is also very common in our practical work:

1. For example, in database design, when the amount of data in a single table reaches tens of millions or hundreds of millions, we will split it into multiple libraries or tables.

2. For example, in cache design, when the data volume of a single Redis instance reaches dozens of gigabytes, which causes performance bottlenecks, we will change the stand-alone architecture to sharding cluster architecture.

Similar split ideas can be seen in HDFS, ElasticSearch and other middleware.

Kafka is no exception, it also uses this horizontal split scheme. In Kafka terminology, the split data subset is called Partition (partition), and the data set of each partition is the full amount of data.

Let's take a look at exactly how Partition works in Kafka. To take a very vivid example, if we compare "Kafka" to "highway":

1. When you hear the Beijing-Guangzhou Expressway, you know that it is a highway from Beijing to Guangzhou. This is a logical term and can be understood as the Topic (theme) in Kafka.

2. A highway usually has multiple lanes for diversion, and the cars on each lane are directed to a destination (belonging to the same Topic). The lane here is Partition.

In this way, the flow path of a message is shown in the following figure, first take the topic route, then take the partition route, and finally decide which partition the message should be sent to.

The partition routing can be simply understood as a Hash function, and the producer can customize this function to determine the partition rules when sending messages. If the partition rules are set properly, all messages will be evenly distributed to different partitions.

Through this two-tier relationship, finally under the Topic, there is a new division unit: Partition. First, the messages are logically classified by Topic, and then physical slicing is further done through Partition. Finally, multiple Partition will be evenly distributed on each machine in the cluster, thus solving the problem of storage scalability.

Therefore, Partition is the most basic deployment unit of Kafka. In this article, Partition is called the second pulse of Kafka architecture design for the following two reasons:

1. Partition is the key to storage, and the core process of MQ "one send, one deposit and one consumption" must be carried out around it.

2. The three most difficult problems in Kafka high concurrency design can all be related to Partition.

Therefore, taking Partition as the root, we can naturally associate the various knowledge points in the Kafka architecture design and form a reliable knowledge system.

Next, please continue to follow my train of thought, take Partition as a clue, analyze the macro-structure of Kafka.

3. Macro architecture design of Kafka

Next, let's take a look at how Partition's distributed capabilities are implemented. How does it relate to the overall architecture of Kafka?

As mentioned earlier, Partition is a partition unit under Topic, it is the most basic deployment unit of Kafka, and it will determine how the Kafka cluster is organized.

Suppose there are two Topic, and two Partition are set for each Topic. If the Kafka cluster is two machines, the deployment architecture would be as follows:

It can be seen that the two Partition of the same Topic are distributed on different message servers, which can achieve the distributed storage of messages. However, for Kafka, a highly concurrent system, only storage scalability is not enough, and message pulling must also be parallel, otherwise it will encounter great performance bottlenecks.

So let's take a look at the consumer side, how does it combine with Partition and achieve parallel processing?

From the point of view of consumers, we must first meet two basic demands:

1. Broadcast consumption power: the same Topic can be subscribed by multiple consumers, and a message can be consumed multiple times.

2. Cluster consumption power: when the consumer is also a cluster, each message can only be distributed to one consumer in the cluster for processing.

In order to meet these two requirements, Kafka introduces the concept of consumption group. Each consumer has a corresponding consumption group. Broadcast consumption is carried out between groups and cluster consumption is carried out within groups. In addition, Kafka limits that each Partition can only be consumed by one consumer in the consumption group.

The final consumption relationship is shown in the following figure: assuming that topic A has four partitions and consumer group 2 has only two consumers, the two consumer groups will share the entire load equally and each consume messages from two partitions.

What should I do if I want to speed up the processing of messages? It is also very simple to add new consumers to consumer group 2, and Kafka will re-do load balancing in terms of Partition. When increased to four consumers, each consumer only needs to process 1 Partition, which will double the processing speed.

At this point, both problems of scalable storage and parallel processing of messages have been solved. However, in the design of high concurrency architecture, there is a very important problem: high availability design.

In a Kafka cluster, each machine stores some Partition, and once a machine goes down, isn't the data lost?

At this point, you must think of persistent storage of messages, but persistence can only solve part of the problem, it can only ensure that historical data is not lost after the machine is rebooted. However, this part of the data will not be accessible until the machine is restored. This is intolerable for highly concurrent systems.

Therefore, Kafka must have the ability to fail over, and the service can still be available when a machine goes down.

If you analyze any highly reliable distributed system, such as ElasticSearch and Redis Cluster, they all have a set of multi-copy redundancy mechanisms.

Yes, Kafka solves the high availability problem through Partition's multi-copy mechanism. In a Kafka cluster, each Partition has multiple copies, and different replicas of the same partition hold the same message.

The relationship between replicas is "one master and many slaves", in which leader replicas are responsible for read and write requests, and follower replicas are only responsible for synchronizing messages with leader replicas. When leader replicas fail, they have a chance to be elected as new leader replicas and provide services, otherwise they will always be on standby.

Now, I assume that there are four servers in the Kafka cluster, topic An and topic B each have two Partition, and each Partition has two copies, then the final multi-copy architecture will be shown in the following figure:

Obviously, the downtime of any machine in this cluster will not affect the availability of Kafka, and the data is still complete.

After understanding the above, let's take a look at the overall architecture of Kafka in turn:

1. Producer: the producer is responsible for creating the message and then delivering it to the Kafka cluster. When delivering the message, you need to specify the Topic to which the message belongs and determine which Partition to send.

2. Consumer: consumers will decide which Partition to pull messages from according to the Topic they subscribe to and the consumer group they belong to.

3. Broker: message server, which can be extended horizontally, responsible for partition management, message persistence, automatic fault transfer, etc.

4. Zookeeper: responsible for the metadata management and other functions of the cluster, such as which broker nodes and Topic are in the cluster, what Partition is each Topic, and so on.

Obviously, in the overall architecture of Kafka, Partition is the link between sending messages, storing messages, and consuming messages. After eating through it, and then to understand the overall structure, the context will be clearer.

Taking Partition as a starting point, this paper analyzes the overall structure of Kafka from a macro point of view, and then briefly summarizes:

1. Kafka degenerates itself into a massive message storage system through ingenious model design.

2. In order to solve the problem of storage scalability, Kafka splits the data horizontally and leads to Partition (partition), which is not only the basic unit of Kafka deployment, but also the minimum granularity of Kafka concurrent processing.

3. For a highly concurrent system, it is also necessary to achieve high availability. Kafka fails over through the multi-copy redundancy mechanism of Partition to ensure high reliability.

After reading the above, do you have any further understanding of what is the second pulse of the governor of Kafka architecture design? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.