Shulou(Shulou.com)06/01 Report--

This article mainly shows you the "common news middleware RocketMQ what is the use", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "what is the use of common message middleware RocketMQ" this article.

Preface

RocketMQ is a distributed, queue-based message middleware developed by Alibaba, which is suitable for high concurrency, high reliability and massive data scenarios. The early open source 2.x version was named MetaQ; iteration 3.x in 2015 and renamed RocketMQ. It was contributed to Apache in 2016 and eventually became one of Apache's top open source projects after more than a year of incubation. RocketMQ is developed on the basis of Kafka, and its birth reference draws on Apache Kafka (I will introduce Kafka separately in a later article). The reason is that with the development of Alibaba's business, they found that the support of Kafka for specific business scenarios was not perfect, so Alibaba's team drew lessons from the design ideas of Kafka, and combined with their own "Singles Day" scenario, developed a RocketMQ that is more suitable for their business scenarios, and reasonably extended Kafka and enriched API. The message routing, storage, cluster partition and other design ideas of RocketMQ are very similar to Kafka, the only difference is that RocketMQ supports business characteristics better, so it is more suitable for business scenarios.

1 professional terminology

Each technical framework has its own technical terms, and the technical terms of RocketMQ are as follows:

1) Producer: the message producer, which is responsible for generating messages. Generally, the business system is responsible for generating messages.

2) Consumer: message consumer, which is responsible for consuming messages. Generally, the backend system is responsible for asynchronous consumption.

3) A kind of Pull Consumer:Consumer that needs to actively request Broker to pull messages.

4) A type of Push Consumer:Consumer that needs to register listening with the Consumer object.

5) Producer Group: a collection of producers, generally used to send a class of messages.

6) Consumer Group: a collection of consumers, which is generally used to accept a class of messages for consumption.

7) Broker:MQ message service (technical secondary school role for message storage and production consumption forwarding).

2 ability and support

1) support cluster model, load balancing and horizontal scalability, such as the cluster architecture we will talk about below.

2) billion-level message stacking capacity.

3) adopt the principle of zero copy, write disk sequentially and read randomly (index file).

4) rich use of API.

5) the code is excellent, and the underlying communication framework adopts Netty NIO framework.

6) NameServer replaces Zookeeper.

7) emphasize that the cluster has no single point, is scalable, is highly available at any point, and is horizontally scalable.

8) message failure retry mechanism, message can be queried.

9) the open source community is active and mature enough (after the test of Singles Day).

3 core source package and function description

As shown in the figure below, let's take a look at the composition of the RocketMQ source package, which is helpful for us to learn more in the future.

1) rocketmq-broker 's main business logic, message sending and receiving, master-slave synchronization, pagecache

2) rocketmq-client client interfaces, such as producers and consumers

3) rocketmq-common common data structure, etc.

4) rocketmq-distribution compilation module, compilation output, etc.

5) rocketmq-example examples, such as producers and consumers

6) rocketmq-fliter transmits uninterested messages filtered by Broker to reduce bandwidth pressure

7) related to rocketmq-logappender and rocketmq-logging logs

8) rocketmq-namesrv Namesrv service for service coordination

9) rocketmq-openmessaging provides external services

10) rocketmq-remoting remote call API, encapsulating the underlying communication of Netty

11) rocketmq-srvutil provides some common tool methods, such as parsing command line arguments

12) rocketmq-store message Storage Core package

13) rocketmq-test provides some test packages

14) rocketmq-tools management tools, such as the famous mqadmin tool

4 Cluster architecture

RocketMQ provides us with a wealth of cluster architecture models, including single-point mode, master-slave mode, double-master mode and double-master mode (or multi-master and multi-slave mode) that are most widely used in production. Let's take a look at the most classic dual-master and double-slave mode, as shown below:

1) NameServer cluster

As an ultra-lightweight configuration center, NameServer cluster stores all the Broker information of the current cluster and the corresponding relationship between Topic and Broker. Each NameServer records complete routing information, provides equivalent read and write services, and supports fast storage expansion. NameServer only does cluster metadata storage and heartbeat work, which has simple function and high stability. There is no communication between multiple NameServer, so there is no need to ensure strong data consistency between nodes. In other words, NameServer cluster is a concept of multi-machine hot backup, and the outage of a single NameServer does not affect other NameServer work. It should be noted that even if the entire NameServer cluster goes down, the Producer, Consumer and Broker that are already working properly will still work properly, but the new Producer, Consumer and Broker will not work.

NameServer uses a heartbeat mechanism, as shown below:

A. A single Broker keeps heartbeat with all NameServer requests. The heartbeat interval is 30 seconds. The heartbeat request includes all the Topic information of the current Broker. It should be noted that when Broker sends a heartbeat to Namesrv, it will bring with it all the Topic information it is currently responsible for. If there are too many Topic (ten thousand), it will lead to a heartbeat, only tens of megabytes of Topic data. If the network condition is poor, the network transmission will fail and the heartbeat will fail, resulting in Namesrv mistakenly thinking that Broker heartbeat failed.

B. NameServer will reverse check the heartbeat information of Broer. If a Broker has no heartbeat within 2 minutes, the Broker is considered offline and the corresponding relationship between Topic and Broker is adjusted. However, NameServer will not take the initiative to notify Producer and Consumer of Broker downtime at this time.

C, Consumer and Broker have a long connection and will send heartbeat messages to Broker every 30 seconds. The Broker side checks the current surviving Consumer every 10 seconds. If it finds that a Consumer has no heartbeat within 2 minutes, it disconnects the Consumer and sends a notification to other instances of the consumer group, triggering the load balancing (rebalance) of the consumer cluster.

D. The producer obtains the mapping relationship between Topic and Broker from Namesrv every 30 seconds and updates it to local memory. Then establish a long-term connection with all the Broker involved in Topic and send a heartbeat every 30 seconds. The currently registered Producer is also scanned every 10 seconds on the broker side, and if a Producer is found to have no heartbeat for more than 2 minutes, it is disconnected.

2) Producer cluster

A Producer cluster is a message producer cluster, which is in the same producer group Producer Group. Producer establishes a persistent connection with one of the nodes in the Name Server cluster (randomly selected), periodically takes Topic routing information from the Name Server, establishes a long connection to the Master that provides Topic services, and regularly sends a heartbeat to the Master. Producer is completely stateless and can be deployed in clusters.

3) Consumer cluster

Consumer clusters are message consumers, and they are in the same consumer group Consumer Group. Consumer establishes a persistent connection with one of the nodes in the Name Server cluster (randomly selected), periodically takes Topic routing information from Name Server, establishes a long connection to Master and Slave that provide Topic services, and regularly sends heartbeats to Master and Slave. Consumer can subscribe to messages from either Master or Slave, and the subscription rules are determined by the Broker configuration.

4) Broker cluster

For Broker, usually Master and Slave are a set of services, which are master and slave nodes to each other, and expose a unified cluster portal through NameServer and external Client. Broker is the core MQ service of message storage.

5 Summary

As the top message middleware in China, the performance of RockerMQ mainly depends on the natural distributed Topic/Queue, and its memory and disk will store message data, drawing lessons from the concept of "air relay" of Kafka, which means that the data does not necessarily land on the ground. RocketMQ provides the characteristics of synchronous / asynchronous double writing and synchronous / asynchronous replication. In a real production environment, you should choose a configuration that suits your business. The following describes the high performance and bottlenecks of RocketMQ:

1) the main bottleneck faced in the actual production environment will eventually fall on IOPS, that is, disk read and write ability. When the peak period comes, the IOPS of sending and receiving messages reaches 10W + messages per second. In the cloud environment, there is obviously a big gap between the SSD physical storage of the cloud environment and the SSD of the self-built computer room. Whether from the disk performance of the database or the disk performance of the search service (ElasticSearch), we can give an accurate bottleneck. The performance bottleneck of cloud storage SSD is that the stand-alone IOPS reaches about 10,000. Here we also see the effect of "barrel short board principle". In real production, the work of CPU is mainly waiting for IO operation, and CPU resources are close to the limit under high concurrency, but IOPS still can not achieve the effect we want.

2) the performance of RocketMQ is good enough, but in many cases, our business will have some non-core message delivery, which can split the business of message middleware, and adopt Kafka's asynchronous sending mechanism for unimportant messages (messages that allow message loss and unreliable delivery), and take advantage of Kafka's strong throughput and message stacking capacity to do business diversion, so as to alleviate the performance bottleneck of RocketMQ.

The above is all the content of the article "what is the use of RocketMQ in Common message Middleware". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.