Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what are the characteristics and advantages of Pulsar". In daily operation, I believe that many people have doubts about the characteristics and advantages of Pulsar. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts about "what are the characteristics and advantages of Pulsar?" Next, please follow the editor to study!

Why choose Pulsar?

The current situation and pain points before using Pulsar, the architecture used for graffiti is basically shown in the following figure. After the message enters the access layer, the message distribution transformation is carried out through Kafka. This message cluster mainly does some functions of message distribution and routing. It is then delivered to a third party through HTTP.

There are some business pain points in the above architectural patterns:

1. The delivery mode of HTTP is not flexible, so it is easy to lose messages.

Due to network reasons, when the size of the company's server is not large enough to support the business, the phenomenon of message loss may occur during the restart process. If you want to meet this requirement, you need to do additional processing for message persistence.

2. The number of Kafka topic is increasing day by day, and the cost of operation and maintenance is high. With the increase in the number of access manufacturers and developers, the pressure on operation and maintenance of Kafka will be high, and the cost of manpower and time will be high.

3. Some pain points of Kafka itself, such as Rebalance mechanism

If the business cluster needs to be upgraded frequently, consumer will often be disconnected. In the case of disconnection, the process of Rebalance is very long, resulting in an increase in the accumulation of messages and a decline in user experience. At the same time, the restart after accumulation will put great pressure on the consumer side in the case of a large number of clusters.

4. Tenants will interact with each other if one tenant dies and does not deal with it in time, it will always accumulate on the Kafka processor, delaying the follow-up process, reducing the performance of message reporting, and affecting other tenants.

The characteristics and advantages of Pulsar Apache Pulsar is a flexible publish-subscribe messaging system, using hierarchical slicing architecture.

1. Rich delivery / subscription policy Pulsar unifies the queue model and the flow model. At the Topic level, only one piece of data needs to be saved, and the same data can be consumed multiple times. Calculating different subscription models by streaming, queuing, etc., greatly improves the flexibility.

two。 It is less difficult for operation and maintenance (compared with Kafka), and the rapid response of Rebalance mechanism is mainly reflected in cross-regional replication. Pulsar uses a native cloud architecture that separates computing from storage. Data is moved away from Broker and stored inside shared storage. The upper layer is the stateless Broker, which replicates message distribution and services; the lower layer is the persistent storage layer Bookie cluster.

Pulsar storage is fragmented, and this architecture can avoid restrictions on capacity expansion and achieve independent expansion and fast recovery of data.

3. Multi-tenant isolation advantage tenant and namespace (namespace) are two core concepts that Pulsar supports multi-tenancy.

At the tenant level, Pulsar reserves appropriate storage space for specific tenants and applies authorization and authentication mechanisms. At the namespace level, Pulsar has a series of configuration policies (policy), including storage quotas, flow control, message expiration policies, and isolation policies between namespaces.

At this stage, the structure is just these three characteristics, corresponding to the pain points faced by graffiti before, so under the fit, we began to use Pulsar instead of Kafka.

At present, the architecture of Pulsar has been applied to the graffiti intelligent platform, becoming a leading message queue, and some secondary development and peripheral services are also being built around Pulsar.

In the past, the delivery of information would have a delay of 5-6 s, but now it is only about 1 s. The overall improvement and improvement is very large.

Of course, the process of replacing Kafka with Pulsar also has some disadvantages, such as high cost. This process requires urging third-party developers to replace SDK and supporting two systems during the transition period.

It just so happens that StreamNative recently opened up KoP, which makes the migration between the two easier and solves this problem.

The figure above is a schematic diagram of the Pulsar architecture within BookKeeper, where zk refers to ZooKeeper. For more introduction to the concepts of Broker, Bookie, and Proxy, you can refer to the previous review of TGIP-CN: what information transmission looks like in Message Lifecycle:Pulsar.

At the same time, through the storage middleware features of the storage side BookKeeper, the current storage separation architecture of Pulsar does not add additional complexity.

Here, Proxy provides a TCP-like agent that provides "addressing" functionality for Consume. Consumer does not need to care about the real Broker address. After connecting to Proxy, it will directly pull data such as the location of Topic from ZooKeeper. In this process, Consumer only needs to ensure a stable connection.

However, Proxy does not have the function of load balancing, which is carried out on Broker. This has also been mentioned in previous TGIP-CN live broadcasts.

Of course, for some use scenarios of graffiti, they have also made some extensions on Proxy.

Application of Dubbo-go + Pulsar as an example

The figure above shows the architecture at the beginning of using Pulsar. After the producer delivers the message, it uses in Topic route to parse it and deliver it. However, in this process, multiple Pulsar clusters need to be built, resulting in an increase in the degree of operation and maintenance.

Now add a "source" to the Pulsar architecture, which can be used to collect the previous topic and pass it along with the function function.

After that, Dubbo-go consumer is embedded in the function component to pull some complex routing rules through Dubbo-go consumer. At the same time, the overall cluster management of Pulsar broker is borrowed to reduce the business pressure of operation and maintenance.

Dubbo-go for some demo presentation or usage of Dubbo, please refer to the following website.

Dubbo Chinese website:

Http://dubbo.apache.org/zh-cn/

GitHub Warehouse:

Https://github.com/apache/dubbo-go

For the architecture diagram of Dubbo-go, please refer to the following:

At this point, the study on "what are the characteristics and advantages of Pulsar" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.