Shulou(Shulou.com)06/02 Report--

RocketMQ message queue, professional message middleware, can not only provide distributed application systems with the ability of asynchronous decoupling and peak cutting and valley filling, but also have the characteristics of massive message accumulation, high throughput and reliable retry required by Internet applications. It is a necessary technology to cope with the peak time of enterprise business.

Due to the business characteristics of cloud film, message queues are used very frequently, so the cloud film service number will release a series of articles on "actual combat of cloud film RocketMQ" starting from this tweet, describing the actual combat experience of cloud film using RocketMQ message queue according to the characteristics of SMS service.

This tweet, "past Life and present Life of Stargate", is provided by Zhou Kaifan, a senior Java development engineer in Yunpian.

This article has 3025 words and is expected to take 20 minutes to read.

Because of its business characteristics, message queues are frequently used. Here, take the cloud SMS service as an example. The logic of SMS service is very simple. We only look at the main process. The essence is to accept user requests, find appropriate channels, and submit them to operators using cmpp/smpp protocol.

We can find that users have to wait for a response from the operator for each request, so the main problems are:

The server of the cloud needs to maintain the http connection all the time when the service provider returns.

The processing rate of the operator directly limits the processing rate of the cloud.

The maximum concurrency of cloud is the sum of concurrency of all vendors.

Under such circumstances, we are unable to provide stable services.

In fact, users do not care about the specific process, they just need to submit SMS messages to the cloud, so we can deal with these sending processes asynchronously, confirm the receipt of SMS messages, and then return the results to users to improve the response speed.

On a daily basis, our system traffic will be a relatively stable value of X, so we provide the spending power of current traffic that can be met, so that there is no backlog of messages.

However, with the increase of traffic, the actual traffic will eventually exceed our consumption capacity, so there will be SMS delivery delay. We do not want to see the dotted line on the right side of the picture.

So we increase the consumption capacity of the system before the traffic reaches the dotted line.

We can see that the cloud is heavily dependent on the message queue, which makes it impossible to find a useful annotation component suitable for the cloud during the micro-service. at that time, the SpringCloud framework did not support the relevant components for RocketMQ, but there was only an immature project on the official github of RocketMQ.

For the current SMS service which is heavily dependent on RocketMQ, we need an easy-to-use annotation that is compatible with the code in our old project, as well as meet the different needs of each team, so we started a component called Stargate to support our follow-up service promotion.

StargateProducerpublic interface TestProducter {@ StargateMapper ("testaaa") SendResult test (@ StargateBody TestVO message);}

@ StargateConsumerpublic class TestConsumer {@ StargateMapper ("testaaa") public void test (@ StargateBody TestVO message) {/ / TODO}}

Why does the Stargate component appear?

In fact, before we had Stargate, there was an encapsulation of RocketMQ SDK in our project, and it did solve a lot of problems, but in the face of more and more producers and consumers, it became more and more difficult to maintain, and even encountered unfamiliar code, I once spent half an hour trying to find out where a producer's consumer is, and this consumer is inherited and rewritten by various kinds of inheritance, and the producer's topic is generated by an extremely complex rule. And various configuration files are scattered in every corner of the code.

Then it is very difficult for the old components to migrate to the new micro-service projects, so that some business lines begin to repackage a set of components themselves, and the messages of their producers are difficult to be consumed by other teams. So I want to make a set of components that can avoid these bad usage habits and provide strong compatibility for everyone to migrate.

What is the goal of designing Stargate?

Therefore, the following points should be considered when designing Stargate:

Simple and easy to use

The way of annotations is clearer, faster and easier to use than using RocketMQ directly, avoiding all kinds of bad writing.

Strong expansibility

The extension provided by SG1 enriches the functionality of Stargate, and the development capabilities of this plug-in are open, which will be mentioned later. In fact, the plug-in functionality was added in 2.0 because I found that after limiting the way you use it, I needed to customize a lot of annotations to be compatible with a variety of usage scenarios, so I opened this part of the ability to allow people to selectively develop and use the features they need.

Compatible with all kinds of old projects

Through the codec we can be compatible with a variety of old projects, without the need to modify the code of the old project.

Unit testing is more convenient

In addition, in the unit testing and development phase, mock can be easily carried out without external dependencies.

The value of Stargate components

At first, I simply thought that the value of this component was to provide a more convenient way to use RocketMQ, but later on, I slowly found that this was not the case. for now, I think the two most valuable points are:

Specification of Asynchronous invocation between Services

Because its scalability and compatibility have been adopted by various business line teams, it seems to have become a "specification", and communication between services is an option. We can provide our own StargateProducer interface definition to others in a jar package.

The sharing center of "experience of use"

The development ability of the plug-in enables people to package their usage patterns into an annotation and publish them, so that many very ingenious usage methods will be released, and these are all available out of the box. Users only need to know what this annotation can achieve, and let developers document the release requirements, so as to form an ecology and precipitate everyone's experience.

In addition, Stargate is also of great help in code structure, and now we can quickly find out where a producer's consumer is, and even consider providing IDE plug-ins to better maintain the code.

Initialization of Stargate

So let's take a look at how these design goals of Stargate are achieved. First, let's take a look at the entry of the component, how we initialize the Stargate.

When the application starts, we process partial annotations to get a Bean configuration information, then register these Bean with Spring, we generate proxy classes for Producer, create Consumer client to listen for messages, and call StargateConsumer to process these messages.

The entry for generating these Bean starts with a factory class, and usually a StargateProducer | StargateConsumer is created through the following steps:

In fact, Stargate is not responsible for initializing these producers and consumers bean,Stargate only provides the creation process, we register these bean with Spring and then provide a factory method for spring to create these bean and maintain these bean at the appropriate time.

So we have examples of these producer interfaces in spring, and we can inject them anywhere and use them to send messages. The consumer who monitors him invokes the pre-configured StargateConsumer.

Codec

Every sender and consumer will encode and decode before sending and receiving messages, which is the key to compatibility with the original project. You can think about it, all projects use RocketMQ, essentially directly call the Send method of SDK to send messages, but the old project on how to turn a message into a binary array is different, so we provide codec interface so that you can replace the implementation of these conversion processes.

Extended interface

However, message coding and decoding can only achieve compatibility, but the need for scalability can not be met, so we abstract six interfaces in the process of re-initialization and sending consumption, so that users can expand their logic.

These interfaces are mainly used to handle "annotation parsing", "RocketMQ Client creation" and "message processing". As we can see in the following figure, the factory invokes the implementation of these interfaces before returning a bean. The corresponding implementation is also called in the message sending and receiving phase.

Typically, we have several processes for implementing a new annotation @ DemoModel:

Implement the annotation processor to process the annotation data

Implement a Client processor to process Client according to the data parsed by annotations

Implement a message processor to process messages according to the data parsed by annotations

Context

There is another problem at this time. Our usual process of parsing annotations needs to save the data to the other two processors. Of course, we do not want to let users process the data themselves, which will increase the cost of users.

So we define a concept of context, which has these characteristics:

Each generated consumer has its own context

The context is inherited.

For example, if we now add a common prefix to the topic of all producers, then we only need to add this prefix to the topic in the context of the producer root, and all producers will have this prefix.

In fact, after Stargate2.0 started providing extension functionality, I stopped iterating over the functionality of the core project, all the new features were released as plug-ins in a project called SG1, and each user can expand their own plug-ins to upload to SG1, hoping to create an ecology.

In addition, after the launch of the international version of YCloud, we have some new challenges. YCloud mainly serves overseas customers. Our server is not in China, but our message needs to be submitted back to the domestic node for consumption, and the MQ is deployed in China. The delay from Hong Kong node to domestic node is unacceptable, because this delay is perceived by users, so my goal is to give this delay to consumers.

If you simply deploy a cluster, it may be the fastest to implement, but the cost is very high, and the load of consumer 1 and consumer 2 is usually uneven, if there is a third computer room, does each consumer have to deploy 3 sets?

So our direction is to deploy a broker on the Hong Kong node and rewrite the client's queue selector so that the producer finds the queue in the nearest broker, while the consumer consumes all the queues.

In fact, the same is true for consumers here, we can change the consumer's client-side load balancer to consume the specified queue, so that we can achieve an isolated environment.

In subsequent developments, we will use the same way for consumers to specify queues on the specified broker, thus creating an isolated environment like this in the figure, which has the following advantages:

Multiple environments are isolated and normal messages do not communicate with each other to achieve the goal of isolation.

Broker in multiple environments is still integrated, and if one consumer fails, another consumer can replace it.

If we find that consumer 1 is abnormal, we can temporarily let consumer 2 replace consumer 1 to ensure that the function is normal, but at present, this part of the function still can not be realized, because we need a command center to tell consumers 2 to go to consumer environment 1.

In order to achieve collaboration between producers and consumers, we need a command center to collect and coordinate the work of all Stargate applications.

StargateCommand will act as a command center, but it is only a coordination mechanism, and without it, Stargate applications will still run in the way their prototypes are set.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.