Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to achieve the comparison of Pulsar and Kafka consumption models. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

Kafka

Kafka belongs to the consumption model of Stream. In order to support the consumption relationship of multi-partition, the concept of consumer group is introduced, and the dynamic reblance operation on the consumer side is supported at the same time. When multiple Consumer subscribes to the same Topic, the consumer subscription partition will be redistributed according to the partition policy. Whenever the relationship between consumer-group and topic changes, reblance operations are triggered dynamically, such as:

Increase or decrease the number of partition in topic

The number of consumer in consumer-group decreases

The subscription relationship between consumer-group and topic has changed

Wait

The advantage of introducing reblance is that when the subscription relationship changes, users can change the subscription relationship without restarting the system, which is equivalent to kafka delegating the right of this allocation from the server to the consumer in the client, so that users can customize their own allocation scheme.

Pulsar

Stream MQ like kafka is more suitable for offline business processing and analysis. Many online businesses use MQ of Queue such as Active MQ. In order to be compatible with these two consumption models at the same time, pulsar makes a layer of consumption abstraction, unifying the two consumption models of Queue and Stream.

Among them, Exclusive and Failover belong to the consumption model of Stream, Share belongs to the consumption model of Queue. At the time of this writing, the latest version of pulsar 2.3.1 Magi Keykeeper shared is a new subscription model added by pulsar. In later articles, we will share the Key_shared subscription model separately, which will not be discussed here.

Comparison of Stream support

Since kafka does not support a consumption model of type Queue, the form of Share is not compared here. Next, we will discuss the following consumption models of pulsar and kafka under Stream.

As shown in the figure below, the consumption of pulsar under Failover and Exclusive is on the left, and the consumption model of kafka on the right.

Suppose there is a topic,topic name of topic1 and five partition, respectively: topic1-p1,topic1-p2,topic1-p3,topic1-p4,topic1-p5. In kafka, consumer-group is used and there are three consumer under the group. As mentioned above, kafka supports the reblance mechanism, so when consumer-2 and consumer-3 join consumer-group, the consumption pressure of the previous consumer-1 will be dynamically shared, as shown in the right half of the above figure. Cousumer-1 consumption topic1-p1 and ropic1-p2,consumer-2 consumption topic1-p3 and topic1-p4,consumer-3 consumption topic1-p5. So when users continue to add consumer to consumer-group, using kafka's reblance mechanism, users can dynamically specify which consumer to consume which partition in topic1.

In pulsar, you can understand subscribe as consumer-group in kafka. If the specified subscribe-name is the same when a user starts consumer, it means that the two consumer belong to the same subscription group. The code example is as follows:

Consumer consumer1 = pulsarClient.newConsumer () .topic ("topic-1") .subscriptionName ("my-subscriber-name") .subscriptionType (SubscriptionType.Failover) .subscribe ()

Consumer consumer2 = pulsarClient.newConsumer () .topic ("topic-2") .subscriptionName ("my-subscriber-name") .subscriptionType (SubscriptionType.Failover) .subscribe ()

As shown in the example above, under the same subscription group, start three consumer. In pulsar, each consumer will subscribe to five partition in the topic1, so each consumer will launch five sub-consumer. Under the subscription model of failover, five partition will be assigned to three consumer for consumption in the form of hashcode. Pulsar regards the sub-consumer currently being consumed as a consumer in leader state, and the remaining sub-consumer that is not working as a slave node. When the consumer in the leader state fails to work for some reason, the sub-consumer in the slave state will replace the consumer of the leader and continue to work. It can be found that kafka joins the mechanism of reblance, allowing users to specify which consumer to consume which partition. In pulsar, this work is done by the mechanism of failover, which assigns consumer to different sub-consumer in the form of hash.

Now, verify what is described above.

Scene one

1. Start pulsar in the form of standalone

$docker run-it\-p 6650 apachepulsar/pulsar:2.3.0 6650\-p 8080 purl 8080\-v $PWD/pulsardata:/pulsar/data\ apachepulsar/pulsar:2.3.0\ bin/pulsar standalone

two。 The number of topic,partition created is 4

$. / bin/pulsar-admin topics mytopic1 create-partitioned-topic-p 4

With the subscription type of failover, start 3 consumer and specify them as the same subscription group, namely-s sub-1

$. / bin/pulsar-client consume mytopic1-s sub-1-n 0-t Failover

3. Start producer and send 10 pieces of data to mytopic1

. / bin/pulsar-client produce mytopic1-n 10-m "hello-pulsar"

As you can see, consumer1 received 2 messages, consumer2 received 5 messages, and consumer3 received 3 messages. The effect is consistent with what we expected.

This is because three consumer have been started to consume messages before producer is sent, so pulsar will distribute messages in the form of hash to three consumer for consumption.

Scene two

Start two consumer in the form of Exclusive subscription, and the results are as follows:

. / bin/pulsar-client consume mytopic1-s sub-1-n 0-t Exclusive

When starting consumer2, there will be an error Exclusive consumer is already connected, because in Failover's subscription mode, other consumer will exist in the form of "slave" consumer, but Exclusive only allows one consumer to subscribe to one topic.

The above is the editor for you to share how to achieve Pulsar and Kafka consumption model comparison, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.