Shulou(Shulou.com)05/31 Report--

This article mainly introduces what features RocketMQ has, which has certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article. Let Xiaobian take you to understand it together.

A nameserver

The stability of nameserver is relatively high. There are two reasons:

1 nameservers are independent of each other and have no communication relationship with each other. If a single nameserver is suspended, it will not affect other nameservers. Even if all nameservers are suspended, it will not affect the use of business systems. This is similar to the zookeeper of dubbo.

2 nameserver does not have frequent reads and writes, so the performance overhead is very small and the stability is very high.

Two brokers

1 Relationship with nameserver

connected

Single broker and all nameservers maintain long connections

heartbeat

Heartbeat Interval: Heartbeat is sent to all nameservers every 30 seconds (this time cannot be changed), heartbeat contains its own topic configuration information.

Heartbeat timeout: nameserver scans all surviving broker connections every 10 seconds (this time cannot be changed), and disconnects if a connection does not send heartbeat data within 2 minutes (the difference between the current time and the last update time exceeds 2 minutes, this time cannot be changed).

disconnect

Timing: broker hangs; heartbeat timeout causes nameserver to actively close the connection

Action: Once the connection is disconnected, nameserver will immediately sense and update the correspondence between topc and queue, but will not notify producers and consumers.

2 Load Balancer

A topic is distributed on multiple brokers, and a broker can configure multiple topics, which are many-to-many relationships.

If a topic has a large message volume, it should be configured with several queues and distributed on different brokers as much as possible to reduce the pressure on a broker.

When topic messages are evenly distributed, the more queues there are on a broker, the greater the pressure on the broker.

3 Availability

Because messages are distributed across brokers, once a broker goes down, message reading and writing on that broker will be affected. Therefore, rocketmq provides a master/slave structure, salve synchronizes data from master regularly, and if master goes down, slave provides consumption services, but cannot write messages. This process is transparent to applications and solved internally by rocketmq.

There are two key points here:

Once a broker master goes down, how long does it take for producers and consumers to discover it? Limited by rocketmq's network connection mechanism, it takes up to 30 seconds by default, but this time can be shortened by application configuration parameters. During this time period, all messages sent to the broker fail, and the broker's messages cannot be consumed, because the consumer does not know that the broker has been hung up.

After the consumer receives the master outage notification, it turns to slave consumption, but slave cannot guarantee that 100% of the master's messages are synchronized, so there will be a small number of messages lost. But messages are never lost, and once the master is restored, unsynchronized messages are consumed.

4 Reliability

All messages sent to broker have synchronous and asynchronous brushing mechanisms. Generally speaking, the reliability is very high.

When the disk is swiped synchronously, the message written to the physical file will return successfully, so it is very reliable.

Asynchronous disk brushing, only machine downtime, message loss will occur, broker hang may occur, but machine downtime crash is rare, unless sudden power failure

5 Message Cleaning

scan interval

Default 10 seconds, determined by broker configuration parameter cleanResourceInterval

space threshold

Physical files cannot be stored on disk indefinitely. When disk space reaches the threshold, messages will no longer be accepted. Broker prints logs. Message sending fails. The threshold is fixed at 85%.

cleaning timing

Default 4 a.m. daily, determined by broker configuration parameter deleteWhen; or disk space reaches threshold

File retention time

Default 72 hours, determined by broker configuration parameter fileReservedTime

6 Reading and writing performance

File memory mapping mode to operate files, avoid read/write system calls and real-time file reading and writing, very high performance

Always one file in writing, others in reading

Sequential writing, random reading

Use linux sendfile mechanism to output message content directly to sokect pipeline, avoiding system call

7 System characteristics

Large memory, the larger the memory, the higher the performance, otherwise the system swap will become a performance bottleneck

IO intensive

CPU load is high and usage is low, because CPU is occupied most of the time after IO WAIT

Disk reliability requirements are high, in order to balance safety and performance, RAID10 array is adopted

Disk read speed requirements fast, high speed requirements large capacity disk

third consumer

1 Relationship with nameserver

connected

A single consumer maintains a long connection with a nameserver and queries topic configuration information regularly. If the nameserver hangs up, the consumer will automatically connect to the next nameserver until there is an available connection and can automatically reconnect.

heartbeat

No heartbeat with nameserver

polling time

By default, consumers get queue updates for all topics from nameserver every 30 seconds, which means that if a broker goes down, it takes up to 30 seconds for the client to notice. This time is determined by the pollNameServerInteval parameter of DefaultMQPushConsumer and can be manually configured.

2 Relationship with broker

connected

A single consumer maintains a long connection with all brokers associated with that consumer.

heartbeat

By default, consumers send heartbeats to all brokers every 30 seconds, which is determined by the HeartbeatBrokerInterval parameter of DefaultMQPushConsumer and can be manually configured. broker scans all surviving connections every 10 seconds (this time cannot be changed). If a connection does not send heartbeat data within 2 minutes (the difference between the current time and the last update time exceeds 2 minutes, this time cannot be changed), the connection will be closed and a notification will be sent to all consumers in the consumer group. Consumers in the group will be reassigned to the queue to continue spending.

disconnect

Timing: consumer hangs up; heartbeat timeout causes broker to actively close connection

Action: Once the connection is disconnected, the broker will immediately sense it and send a notification to all consumers in the consumer group, and the consumers in the group will redistribute the queue to continue spending.

3 Load Balancer

In cluster consumption mode, multiple machines in a consumer cluster consume multiple queues of a topic together, and a queue is consumed by only one consumer. If a consumer drops out, other consumers in the group will take over from the dropped consumer.

4 Consumption mechanisms

local queue

Consumers continuously pull messages from brokers, messages are pulled to local queues, and then local consumer threads consume local message queues, just an asynchronous process, pulling threads do not wait for local consumer threads, this mode is very real-time. There is a protection for consumers on local queues, so local message queues cannot be infinite, otherwise they may take up a lot of memory. The size of local queues is controlled by the pullThreshold ForQueue attribute of DefaultMQPushConsumer. The default is 1000, which can be manually set.

polling interval

How often does the message pull thread pull? The interval time is controlled by the pullInterval property of DefaultMQPushConsumer, which defaults to 0 and can be set manually.

Number of messages consumed

How many messages does the listener receive from the local queue at a time? This parameter is controlled by the consummeMessageBatchMaxSize property of DefaultMQPushConsumer. It defaults to 1 and can be set manually.

5 Consumption Progress Storage

Store the consumption progress of each queue to the corresponding broker every once in a while. The time is controlled by the persistConsumerOffsetInterval attribute of DefaultMQPushConsortium. The default is 5 seconds and can be manually set.

If a topic has 3 queues on a broker and a consumer consumes these 3 queues, how many connections does the consumer have with the broker?

A connection, consumption unit is related to queue, consumption connection is only related to broker, in fact, consumer will pull all queue message tasks to the local queue, pull one by one, pull the task to the end of the queue, and then execute the next pull task

4 Producers

1 Relationship with nameserver

connected

A single producer maintains a long connection with a nameserver, queries topic configuration information regularly, and if the nameserver hangs, the producer will automatically connect to the next nameserver until there is an available connection, and can automatically reconnect.

polling time

By default, producers get queue updates for all topics from nameserver every 30 seconds, which means that if a broker goes down, it takes up to 30 seconds for producers to notice, during which time messages to that broker fail to be sent. This time is determined by the pollNameServerInteval parameter of DefaultMQProducer and can be manually configured.

heartbeat

No heartbeat with nameserver

2 Relationship with broker

connected

A single producer maintains a long connection with all brokers associated with that producer.

heartbeat

By default, producers send heartbeats to all brokers every 30 seconds, which is determined by DefaultMQProducer's heartbeatBrokerInterval parameter and can be manually configured. broker scans all surviving connections every 10 seconds (this time cannot be changed), and closes a connection if it does not send heartbeat data within 2 minutes (the difference between the current time and the last update time is more than 2 minutes, this time cannot be changed).

connection is broken

Remove producer information from broker

3 Load Balancer

Producer time doesn't matter, each producer takes turns sending messages to the queue

Thank you for reading this article carefully. I hope that the article "What are the characteristics of RocketMQ" shared by Xiaobian will be helpful to everyone. At the same time, I hope that everyone will support you more and pay attention to the industry information channel. More relevant knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.