Shulou(Shulou.com)06/02 Report--

I have used activemq and rabbitmq message queue in the actual production environment before, and there are some problems that are difficult to solve in the process of using. This paper analyzes the advantages and characteristics of rocketmq through product selection, network architecture and core features.

Product selection

When we select the middleware, we usually select the product type through the following points:

1. Performance

two。 Degree of functional support

3. Development language (is there any member of the team who is familiar with the development language of this middleware, and whether developers of this language are easy to recruit on the market)

4. How many companies have actually used it in the production environment, and how effective is it?

5. How is the support of the community?

6. Whether the learning level of middleware is simple and whether the documentation is detailed.

7. Stability.

8. Whether the cluster function is complete

...

If I choose a message queue from the above 8 points, as a programmer familiar with java, when I encounter the scenario of re-selecting message queue, I will not hesitate to choose rocketmq,rocketmq except for the slightly poor performance on point 5 (few documents, high learning cost) and unfriendly monitoring and management functions, in other ways, it is really a very good message queue middleware.

Network architecture

(the picture comes from the official document)

The main part of rocketmq is made up of four kinds of clusters: namesrv cluster, broker cluster, producer cluster and consumer cluster.

Namesrv cluster: that is, the registry. Rocketmq does not use third-party middleware in the registry, but writes its own code. The number of lines of code is only 1000 lines. Producer, broker and consumer all need to register with namesrv at startup, and there is no communication between namesrv services.

Broker cluster: broker provides the functions of message management, storage, distribution and so on. It is the core component of message queue. Rocket's cluster on broker provides two main solutions, one is master-slave synchronous scheme, in which messages are written to master and slave servers at the same time as messages are sent successfully; the other is asynchronous scheme, in which slave's asynchronous service is responsible for reading master data, and I prefer asynchronous scheme when choosing.

Producer cluster: the producer of messages, each producer needs to belong to a group,producer group concept in addition to play some role in transactional messages, but other times, it is more of a virtual concept.

Consumer cluster: the consumer of messages, there are two main consumer:DefaultMQPullConsumer and DefaultMQPushConsumer, and after diving into the code, you can find that rocket's consumer uses the pull pattern to process messages. Under the configuration of cluster messages, messages are distributed evenly among the services in the cluster. When one of the servers goes down, the messages assigned to it will continue to be allocated to other consumer.

Core characteristics

1. The number of read queues and write queues can be inconsistent: when we use the updateTopic command to create a topic, we will find that there are 8 write columns and 8 read columns by default under the newly created topic (depending on the configuration), and the number of read queues and write queues can be different. Why? Is the read-write queue at the bottom physically separated? With this problem in mind, I analyzed the relevant source code and found that the underlying code refers to the same queue for read and write queues, where the number of write queues is for producer and the number of read queues is for consumer:

a. Suppose there are eight write queues and four read queues, then the messages generated by producer will be written to the eight queues by rotation, but consumer can only consume the first four queues. Only when the read queues are reset to 8, consumer can continue to consume historical messages from the last four queues.

b. Suppose there are 4 write queues and 8 read queues, then the messages generated by producer will be written to 4 queues in rotation, but consumer can consume 8 queues, but the last 4 queues have no messages to consume.

two。 It is stored as a file storage mode, and supports synchronous flushing and asynchronous flushing. I prefer asynchronous flushing. After all, the probability of broker hanging up is relatively small, and most business scenarios can tolerate the loss and a small amount of messages in extreme cases.

3. Support message backtracking and periodically delete historical messages

4. The cluster solution is much better than activemq, and supports multi-master and multi-slave solutions. For example, under the 2-master and 2-slave asynchronous architecture, master,as,bs b is slaver. When a machine goes down, producer will send all messages to machine b, and consumer will consume messages on as,b and bs. In theory, only millisecond messages will be lost, which will not affect the normal use of the business. It can be said that the cluster solution of rocketmq is better than that of activemq. In many cases, we do not have high requirements for the performance of asynchronous queues, but the availability of clusters must be very high. Here are three clustering scenarios for activemq:

a. Disk array is a general scheme because of its high cost.

b. Using jdbc to realize unified storage of messages, not only the performance is a problem, but also the problem is just thrown to the database, which does not solve the single machine problem of the cluster.

c. By using the master selection function of zookeeper registry, the data is synchronized between various services, and in the actual use, it is found that the host drifts automatically, the data confusion caused by incomplete synchronous data and the service can not be started, but it is not as stable as a stand-alone machine.

5. The number of queues is more than 10000 on a single machine.

6.consumer supports cluster function and can consume messages averagely. When one consumer goes down, other PCs will continue to share equally.

7.consumer consumes messages in the way of pull, and its performance is not lower than that of push, which is also one of the reasons why broker has strong parallel ability. It devolves the initiative to consumer, which reduces the amount of computation and thread switching cost of broker.

8. Support sequential messages. When sending a message, you can use the hash mode of the selector mechanism to achieve which broker and which queue the message falls on. When a Broker goes down, the queue will be automatically switched due to the change of the modulus value.

Synchronous return, asynchronous return and oneway are supported when 9.producer sends messages.

10.broker guarantees that each message is delivered to consumer at least once, so the business of consumer needs to support idempotency.

11. The ability of message accumulation is amazing, and one of the functions of message queues is to prevent the flood peak from directly destroying the back-end business.

twelve。 Support to query messages according to message id and message key. I like the function of querying messages according to key very much. For example, in the business of issuing an order, order id can be used as key to facilitate analysis of the processing process of abnormal orders in the system.

13. Support for message filtering

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.