Shulou(Shulou.com)06/02 Report--

RocketMQ briefly introduces what it means, and I believe that many inexperienced people have nothing to do about it. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

I. brief introduction

RocketMQ is a distributed messaging middleware developed by Ali Open Source, which meets the needs of online message accumulation in Shanghai. It was donated to the Apache Open Source Foundation as an incubation project at the end of 2016 and officially became a top Apache project in 2017. Is a pure Java message middleware, known for its reliability, low latency, extensibility, and ease of use.

1. Background introduction

In the early days, Ali also built its message middleware based on ActiveMQ 5.x distributed message middleware, but with the increase of topics and queues, it was slowly found that the ActiveMQ IO module encountered a bottleneck. We try our best to solve this problem through throttling, circuit breaker or downgrade, but the effect is not good. Therefore, we began to pay attention to Kafka, the popular messaging solution at that time. Unfortunately, Kafka does not meet our requirements, especially in terms of low latency and high reliability. Kafka is a distributed streaming media platform, which originates from the log aggregation case. It doesn't need too much concurrency. In some large cases of Alibaba, we found that the original model could not meet our actual needs.

RocketMQ originated from Ali's Metaq (Metamorphosis) experienced Metaq 1.x, Metaq 2.x, RocketMQ3.x changed its name to RocketMQ in version 3.0, and now the latest version is RocketMQ4.5.x.

2. Use the scene

It is not so much the usage scenario of RocketMQ as the usage scenario of MQ (message queuing). There are many products of MQ, such as ActiveMQ,RabbitMQ,ZeroMQ,Kafka,MetaMQ,RocketMQ and so on.

MQ can be used in many fields, including asynchronous communication decoupling, enterprise solutions, financial payment, telecommunications, e-commerce, express logistics, advertising marketing, social networking, instant messaging, mobile games, video, Internet of things, car Internet and so on.

Here are some common scenarios:

One-to-many, many-to-many asynchronous decoupling, based on publish and subscribe model, asynchronous decoupling of distributed applications to increase the horizontal scalability of applications.

When peak-cutting and valley-filling are suddenly hit, MQ can buffer the sudden traffic and prevent the downstream subscription system from collapsing due to sudden traffic.

Log monitoring, as a monitoring communication pipeline for important logs, minimizes the impact of application log monitoring on system performance.

Message push, which provides point-to-point push and one-to-many broadcast push for social applications and Internet of things applications.

Financial message, send financial message, achieve financial quasi-real-time message transmission, reliable and secure.

Telecommunication signaling, which encapsulates telecommunication signaling into messages and transmits them to each control terminal to realize quasi-real-time control and information transmission.

3. Comparison of mainstream MQ

Comparison of mainstream MQ: RocketMQ, ActiveMQ, Kafka

Message products client SDK protocol and specification message storage server triggers retransmission of broadcast message high availability and failover message tracking ActiveMQJava,.NET,C + + and other push models, supports OpenWire,STOMP,AMQP,MQTT,JMS using JDBC and high performance logs (such as levelDB,kahaDB) supports very fast persistence does not support, according to storage, if kahadb is used, it is required that the ZooKeeper server does not support pull models such as KafkaJava,Scala Support TCP high-performance file storage is not supported, need ZooKeeper server does not support RocketMQJava,C + +, Go pull model, support TCP,JMS,OpenMessaging high-performance and low-latency file storage support Master-Slave model, there is no other suite to support II, overview 1, RocketMQ cluster deployment structure

Name Server

Name Server is an almost stateless node that can be deployed in clusters without any information synchronization between nodes.

Broker

The deployment of Broker is relatively complex. Broker is divided into Master and Slave. A Master can correspond to multiple Slave, but a Slave can only correspond to one Master,Master and Slave. The corresponding relationship between Master,Master and Slave is defined by specifying the same Broker Name and different BrokerId. If BrokerId is 0, it means Master. Non-zero means Slave. Master can also deploy multiple.

Each Broker establishes persistent connections with all nodes in the Name Server cluster and registers Topic information to all Name Server at regular intervals (every 30s). Name Server scans all surviving broker connections regularly (every 10 seconds). If the Name Server does not receive a heartbeat for more than 2 minutes, the Name Server disconnects from the Broker.

Producer

Producer establishes a persistent connection with one of the nodes in the Name Server cluster (randomly selected), periodically takes Topic routing information from the Name Server, establishes a long connection to the Master that provides Topic services, and regularly sends a heartbeat to the Master. Producer is completely stateless and can be deployed in clusters.

Producer gets the latest status of all topic queues from Name server every 30s (by ClientConfig's pollNameServerInterval), which means that if Broker is not available, Producer can perceive at most 30s, and all messages sent to Broker during this period will fail.

Producer sends heartbeats to all associated broker every 30s (as determined by heartbeatBrokerInterval in ClientConfig). Broker scans all surviving connections every 10s. If Broker does not receive heartbeat data within 2 minutes, close the connection with Producer.

Consumer

Consumer establishes a persistent connection with one of the nodes in the Name Server cluster (randomly selected), periodically takes Topic routing information from Name Server, establishes a long connection to Master and Slave that provide Topic services, and regularly sends heartbeats to Master and Slave. Consumer can subscribe to messages from either Master or Slave, and the subscription rules are determined by the Broker configuration.

Consumer gets the latest queue status of topic from Name server every 30 seconds, which means that when Broker is not available, Consumer needs up to 30 seconds to perceive it.

Consumer sends heartbeats to all associated broker every 30s (determined by heartbeatBrokerInterval in ClientConfig), and Broker scans all surviving connections every 10s. If a connection does not send heartbeat data within 2 minutes, it closes the connection; and sends a notification to all Consumer of the Consumer Group, the Consumer in the Group reassigns the queue, and then continues to consume.

When Consumer is notified of master downtime, it turns to slave consumption. Slave cannot guarantee that all master messages are synchronized, so a small number of messages will be lost. But once the master is restored, messages that are not synchronized to the past will eventually be consumed.

2. Message sending type 1) synchronous messages (reliable)

Synchronous sending, thread blocking, delivery completes blocking ending

If the delivery fails, it will be retried within the default timeout of 3 seconds, with a maximum of 2 retries

Delivery of completes does not mean delivery is successful. It is up to check SendResult.sendStatus to judge whether delivery is successful or not.

2) Asynchronous messages (reliable)

To send an asynchronous message, you need to specify the callback function after the message is sent successfully. The API that calls the message to send the message will return immediately. The thread of the message sender will not block until the end of the run, and the callback task of successful or failed message delivery will be executed in the new thread.

3) one-way message (unreliable)

The message is unreliable and has high performance. It is only responsible for sending a message to the server. It does not retry and does not care whether it is sent successfully.

The process of sending messages in this way takes a very short time, usually at the microsecond level.

3. Consumption pattern (cluster / broadcast)

RocketMQ physical deployment diagram:

Terminology interpretation

Message subject (Topic)

Topic is the category in which producers are sending messages and consumers are pulling messages. The relationship between Topic and producers and consumers is very loose. Specifically, a Topic may have 0, and one or more producers send messages to it; instead, a producer can send messages of different types of Topic. Similarly, a consumer group can subscribe to one or more topics, as long as the instances of the group keep their subscriptions consistent.

One topic corresponds to multiple queues (4 by default)

Messages are stored in different queues

Consumer Group (ConsumerGroup)

The collection name of a class of Consumer. The Consumer in the group usually consumes a class of messages, which is logically consistent.

A consumerGroup corresponds to only one topic.

The Topic and Tag of consumer subscriptions in the same ConsumerGroup must be the same (consumption will be problematic in cluster mode)

A message queue corresponds to only one consumer

Production group (ProducerGroup)

The collection name of a class of Producer. The Producer in the group usually sends a class of messages with the same logic.

1) broadcast mode

A message is consumed by multiple consumer. Even if these consumer belong to the same ConsumerGroup, the message will be consumed once by each Consumer in the ConsumerGroup. The concept of ConsumerGroup in broadcast consumption can be considered meaningless in message division.

2) Cluster mode (default)

Consumption messages are shared equally among Consumer instances in a ConsumerGroup. For example, if a Topic has 9 messages and one ConsumerGroup has 3 instances (possibly 3 processes or 3 machines), only some of them are consumed by each instance, and the consumed messages cannot be consumed by other instances.

After reading the above, have you mastered the method of what RocketMQ brief introduction refers to? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.