Shulou(Shulou.com)05/31 Report--

This article shows you how Tencent uses Apache Pulsar in cross-city practice. The content is concise and easy to understand, which can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Introduction: the following describes how to use Pulsar as a MQ component for application, hoping to help you have a deeper understanding of the application type of Pulsar as message middleware.

01

Tencent Billing introduction

The platform carries the company's daily revenue market of hundreds of millions, and provides services for 180 + countries (regions), 10,000-level business codes and 100W + settlement merchants. The total amount of escrow accounts is more than 30 billion. It is an omni-directional, one-stop billing platform.

As a 100 billion-level online payment platform, Tencent Billing (Masters) needs to focus on optimization and upgrading in the following areas.

1. Data consistency

Billing scenarios require that no data can be lost, which is the most basic requirement. After all, it belongs to the category of financial assets. Although there are many virtual accounts, they are still traded with real gold and silver. SQL is extracted to ensure high reliability and consistency at the data level.

two。 Fluency, availability

The operation process of the platform is somewhat similar to that of some e-commerce platforms, that is, adding shopping carts, placing orders, and shipping goods. The difference is that most e-commerce platforms require users to participate in these processes manually, while Tencent billing can be done automatically with a click in the background.

Especially for some free projects, there will be higher requirements for overall usability. It needs to have the ability of disaster recovery and can be repaired automatically in abnormal circumstances.

3. Performance level

At the logical level, because the whole system covers more than 300 different business products, there is a high demand for performance stability in the face of massive growth of data.

For the magnitude of Tencent business, the effect of disaster recovery is expected to be at least city level. There are tens of thousands of businesses within Tencent, which are deployed everywhere, and the services of the billing platform are basically deployed in Shenzhen and Shanghai. It is necessary to provide the service of "living more in different places" here.

Of course, in most cases, we will continue to deploy services in the same city. When making a business request, choosing the nearest gateway can reduce unnecessary situations. Of course, in the case of different places, it is also necessary to ensure the state of fast switching.

02

Cross-city Challenge in Billing scenario

At the disaster recovery level, according to the region, it can be divided into "living more in the same city" and "living in different places". At the logical level, it is easier to operate in both states, but it is more difficult at the data level.

In the case of multiple active users in the same city, there will be multiple replicas in the data layer, which can be deployed across data centers or IDC. If there is a problem, you can immediately switch to another server room / IDC for deployment to achieve lossless switching. Although this way of operation is relatively simple, but it can not avoid cross-city, some extreme situations.

In the case of multiple activities in different places, the deployment needs to be carried out across cities and places. When one city's server goes down, another city can continue to support the product to reduce losses.

However, it is very difficult to achieve complete lossless, unless each step is synchronized to a different location and the operation is successful, to ensure that the data is consistent at all times. However, in terms of performance, the efficiency is greatly reduced.

In terms of overall performance, living in other places is more suitable for Tencent's billing product model.

Single living in different places and multiple living in different places

In the past, Tencent adopted the mode of living alone in different places. At that time, the business was not as complex as it is now, and most of the requests were completed in the main city / main database. The reading of the database can be read in the main city or backup city according to the different needs of different products.

The advantage of this architecture is that the data is highly consistent and error-free, because all the data is integrated in the same place to write. The disadvantage is that in the event of a city-level network disaster, there is no device for write operations.

With the expansion and enrichment of Tencent's business, living in other places has become an inevitable choice to provide high-quality and stable services. This involves the synchronization of the two data centers.

At present, the mainstream in the industry is the CDC asynchronous mode, which can be applied to the message level through message queues. Not only the massive stacking capacity of MQ can be used to store operation pipelining, but also multi-consumption processing capacity can be used to support the efficiency of data replication.

For MQ deployment, you can also use intra-city or cross-city deployment.

Same city: deploy a separate MQ cluster in the DB center. When producing, it is carried out in the MQ of the same city, and when consuming, the data is read across. The advantage of this is high production efficiency, but the disadvantage is that cross-provincial access is required during consumption, and it does not have the ability to "cross-province".

Cross-city: MQ is deployed in the same city, with cross-provincial capability, which can further reduce data loss. The disadvantage is that in production configuration, cross-city data calls are required, and consumer copies need to be bound in multiple locations.

The second method is often used in daily use because of the need to minimize the risk of data loss.

Therefore, the requirements for distributed message queues in Tencent billing system are as follows:

Consistency requirements: billing scenarios require that a single piece of data cannot be lost, which is the most basic requirement.

High availability requirements: disaster tolerance is required, and it can be repaired automatically in abnormal cases.

Mass storage demand: in the era of mobile Internet, the generation of a large number of transaction data requires massive stacking capacity.

Fast response requirements: in hundreds of millions of payment scenarios, MQ is required to provide smooth response time and be controlled within 10ms as much as possible.

03

Pulsar cross-city capability

Therefore, after investigating and trying, Pulsar is chosen as the message queue choice of Tencent billing platform. Compared with Kafka, which is widely used in the industry, its main scenario is big data log processing, which is rarely used in financial scenarios. RocketMQ is not very friendly to Topic operation, especially does not support deleting failure messages by Topic, and does not have downtime Failover capability.

Pulsar has native high consistency, provides highly available storage services based on BookKeeper, adopts a storage and service separation architecture to facilitate capacity expansion, and supports a variety of consumption modes and multi-domain deployment models.

At the message replication level, Pulsar provides two replication methods. One is cross-region asynchronous replication, eliminating the need for additional deployment of asynchronous replication. It is very suitable for some data statistics scenarios within Tencent, especially those business in cross-provincial or even cross-border scenarios.

The second is synchronous replication, which itself is a cross-city-level business, with copies stored in multiple locations. We also explained a lot about Pulsar's read-write architecture model in the previous TGIP-CN series. You can click "album" at the beginning of the article to check it out.

For more information on Pulsar's efficient data replication mode, check out the 25:00-34:20 period of video playback.

04

Optimization of Tencent Billing across cities

There will be a problem in cross-city production and consumption. Does the consumer have to go to the main city to get the news, because it is mentioned above that the control of a topic belongs to only one broker.

In fact, the saying of belonging to broker is only at the "write" level, while the reading level can be operated with other broker. To avoid tens of thousands of consumers subscribing to the same topic, we can deploy a read-only broker according to the "nearest principle".

Read-only broker is very helpful for Tencent billing platform with large flow. In deployment, cross-city storage is common. BookKeeper provides some ledger-level storage objects that can create read-only data.

Here are two issues:

1. How do I know what data to read?

Because read-only broker and broker are deployed in isolation, you need to synchronize topic metadata from the written broker so that you know what the ledger collections corresponding to topic are, and you can tell the data is read.

two。 How do I read the latest data?

Topic consists of an ordered set of ledger streams, with only one ledger in each group in the open state and the rest in the closed state and cannot be changed.

If the read-only broker is reading the ledger in the off state, it can be done easily. If you read the ledger of the open status, you need to confirm whether there is a new message through LastConfirm id.

After the message is read correctly, the problem of consumer offset synchronization needs to be solved. If you don't consider the problem of message deletion, you don't have to set it.

Because in the Pulsar system, messages will be deleted by default after consumption, and those with no consumers will also be deleted. In this way, the production cluster is only responsible for production and no consumption.

In this way, when you split the ledger, you may delete the previously unconsumed ledger. This only requires synchronizing the read-only consumer offset to the broker to be written. This is a complete design idea of read-only broker. The above is how Tencent uses Apache Pulsar in cross-city practice. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.