Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about the technical terms in rocketmq and the problems that need to be solved in message queuing. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something from this article.

Technical terms:

Producer: message producer, responsible for generating messages, usually generated by business systems

Consumer: message consumer, responsible for consuming messages. Generally, the backend system is responsible for asynchronous consumption.

Push Consumer:

A type of Consumer in which an application usually registers a Listener interface with a Consumer object. As soon as a message is received, the Consumer object immediately calls back the Listener interface method

Pull Consumer:

A type of Consumer that usually actively calls Consumer's pull message method to pull messages from broker. The initiative is controlled by the application.

Producer Group:

The collection name of a class of Producer, which usually sends a class of messages with the same sending logic

Consumer Group:

The collection name of a class of Consumer, which usually sends a class of messages with the same sending logic

Broker:

Message transfer role, responsible for storing messages, forwarding messages, commonly known as Server. It is called Provider in the JMS specification.

Broadcast consumption:

A message is consumed by multiple Consumer, and even if these Consumer belong to the same Consumer Group, the message will be consumed once by each Consumer in the Consumer Group. The concept of Consumer Group in broadcast consumption can be considered meaningless in message division. In the CORBA Notification specification, the consumption mode belongs to broadcast consumption in the JMS specification, which is equivalent to JMS publish/subscribe model.

Cluster consumption:

The instances in a Consumer Group share the consumption messages equally. For example, if a topic has nine messages and one Consumer Group has three instances (possibly three processes or three machines), each instance can only consume three of the messages.

Sequential messages:

The order of consuming messages should be the same as the order in which messages are sent. In RocketMq, it mainly refers to the local order, that is, a class of messages must be sent sequentially by Producer single thread and sent to the same queue in order to satisfy the sequence, so that Consumer can consume messages according to the order sent by Producer.

Normal sequential messages:

A kind of sequential messages, normally can guarantee complete sequential messages, but once a communication exception occurs, Broker restarts, due to the change in the total number of queues, the queues located after hashing will change, resulting in temporary message queue sequence inconsistency.

If the business can tolerate temporary disordering of messages in the event of a cluster exception (such as a Broker outage or restart), it is more appropriate to use a normal domestic demand method.

Strictly sequential messages:

A kind of sequential messages that can guarantee the order regardless of normal abnormal conditions, but at the expense of the distributed FailOver feature, that is, if a machine in the Broker cluster is unavailable, the whole cluster will be unavailable and the service availability will be greatly reduced.

If the server is deployed in synchronous double write mode, this defect can be avoided by switching the standby machine automatically, but the service will still be unavailable for a few minutes. (relying on synchronous double writing, automatic switching between master and slave, automatic switching has not been implemented.) currently, only database binlog synchronization strongly relies on strict sequential messages. Most other applications can tolerate temporary disorder, so it is recommended to use ordinary sequential messages.

Message Queue

In RocketMq, all message queues are persistent and unlimited in length. The so-called infinite length means that each storage unit in the queue is of a fixed length, and the storage unit accessing it is accessed by offset. Offset is a java long type, 64-bit, and theoretically will not overflow within 100 years, so it is considered that the length is infinite. In addition, only the data of the last few days is stored in the queue, and the previous data will be deleted according to the expiration time. You can also think of Message Queue as an infinite array, and offset is the subscript.

What problems need to be solved in message queuing: 1. Publish/Subscribe

Publish and subscribe message is the most basic function of message middleware, and it is also relative to traditional RPC.

2. Message Priority (message priority)

The priority described in the specification means that in a message queue, each message has a different priority, which is generally described by an integer. The message with high priority is delivered first. If the message is completely in a memory queue, then it can be sorted according to priority before delivery, so that the high priority message can be delivered first. Because all messages in Rocketmq are persistent, the overhead will be very high if sorted by priority, so RocketMq does not specifically support message priority, but similar functions can be achieved by workarounds, that is, a high priority queue and a normal priority queue can be configured separately, and different priorities can be sent to different queues. Priority issues can be classified into two categories:

1) as long as the goal of priority is achieved, it is not strictly a priority, but is usually divided into high, medium, low, or several more levels. Each priority can be represented by a different topic, and a different topic can be specified to represent the priority, which can solve part of the priority problem, but compromise the priority accuracy of the business.

2) strict priority, which is expressed as an integer, for example: 0,65535, this kind of priority problem is generally very inappropriate to be solved with different topic. If you let MQ solve this problem, it will have a very big impact on the performance of MQ. It is important to ensure that this strict priority is needed in the business, and how much impact will it have on the business if the priorities are compressed into several?

3. Message Order (message order)

Message ordering refers to a kind of message consumption that can be consumed in the order in which it is sent. For example, an order produces three messages, namely, order creation, order payment, and order completion. When consuming, it is necessary to spend in this order in order to be meaningful. But at the same time, orders can be consumed in parallel. RocketMq can strictly guarantee the order of messages.

4 、 Message Filter

Broker-side message filtering: in Broker, filtering is done according to the requirements of Consumer, which has the advantage of reducing the network transmission of Consumer useless messages. The disadvantage is that it increases the burden of Broker, and the implementation is relatively complex.

1) Taobao Notify supports multiple filtering methods, including filtering by message type and flexible syntax expression filtering

2) Taobao RocketMQ supports filtering by simple MessageTag, and also by Message Header and body

5. Message Persistence (message persistence)

Several persistence methods are commonly used in message middleware:

1) persist to a database, such as mysql

2) persist to kv storage, such as kv storage systems such as levelDB and Berkeley DB

3) persistence of file records, such as kafka,rocketmq

4) make a persistent image of the in-memory data, such as beanstalkd,VisiNotify

1) 2) 3) all of the three persistence methods have the ability to expand the memory queue Buffer. 4) it is only a memory image that can still recover the previous memory data when the broker is hung up and rebooted.

Rocketmq refers to the persistence mode of kafka and makes full use of Linux file system memory cache to improve performance.

6. Message Reliablity (reliability of messages)

Several situations that affect the reliability of the message: (1) Broker shuts down normally

(2) Broker exception Crash (crash)

(3) OS Crash

(4) the machine is powered off, but the power supply can be restored immediately.

(5) the machine cannot boot (may be caused by damage to key equipment such as cpu, motherboard, memory, etc.)

(6) the disk device is damaged.

1) 2) 3) 4) all four cases are situations in which hardware resources can be recovered immediately. Rocketmq can guarantee that messages will not be lost in these four cases, and the latter can lose a small amount of data (depending on whether the disk flushing method is synchronous or asynchronous)

5) 6) it is a single point of failure and cannot be recovered. once it occurs, all messages on this single point are lost. In these two cases, through asynchronous replication, RocketMQ can ensure that 99% of the messages will not be lost, but there will still be a very small number of messages that may be lost. A single point can be avoided through synchronous double-writing technology, and synchronous double-writing is bound to affect performance. It is suitable for situations that require high message reliability, such as Money-related applications.

The rocketmq3.0 version supports synchronous double writes.

7. Low Latency Messaging (low message latency)

If the message does not pile up and arrives at the Consumer Rocketmq immediately after the message reaches the big broker, the long polling Pull method is used to ensure that the message is very real-time and the real-time performance of the message is not lower than that of Push.

8. At Least One (at least once)

Each message must be delivered once. RocketMq Consumer first pull messages locally, and then returns ack to the server after consumption is completed. If there is no consumption, ack messages will not be returned, so Rocket MQ supports this feature very well.

9. Exactly Only One (only once)

1) message sending phase. Duplicate messages are not allowed to be sent.

2) consumption message phase, duplicate messages are not allowed to be consumed

In order to achieve these two points, in the distributed system environment, it is inevitable to incur huge overhead, so Rocketmq does not guarantee this feature, but requires de-duplication in the business, that is, to consume messages to achieve idempotency.

Although rocketmq cannot strictly guarantee non-repetition, under normal circumstances, repeated push and consumption will rarely occur. Only network anomalies, Consumer start and stop and other abnormal circumstances will lead to consumption duplication. The essential cause of this problem is the uncertainty of network calls, that is, the third state of neither success nor failure, which leads to the problem of message repeatability.

10. What if broker's buffer is full?

There is no concept of memory buffer in Rocketmq, the queues of rocketmq are persistent disks, and the data is cleared periodically.

For the solution to this problem, rocketmq is different from other mq. The internal buffer of rocketmq is abstracted into an infinite length queue, which can be loaded no matter how much data it has. This infinity has a premise. Broker will delete expired data regularly. For example, if the brokr is saved for only 3 days, the buffer will be deleted from the end of the queue even though the length of the buffer is unlimited.

11. Retrospective consumption

Retrospective consumption refers to the message that consumer has consumed successfully, which needs to be re-consumed due to business needs. To support this feature, after broker delivers a success message to consumer, the message still needs to be retained, and the re-consumption is generally based on the time dimension.

RocketMQ supports backtracking of consumption by time, and the time dimension is accurate to milliseconds, either backwards or backwards.

12. Message accumulation

The main function of message middleware is asynchronous decoupling. An important function of message middleware is to block the data flood peak at the front end and ensure the stability of the back-end system, which requires message middleware to have a certain ability to accumulate messages. Message heap points are as follows:

(1) messages are accumulated in the memory Buffer. Once the memory Buffer is exceeded, messages can be discarded according to certain discarding policies, as described in the CORBA Notification specification. It is suitable for businesses that can tolerate discarding messages. In this case, the stacking capacity of messages mainly depends on the memory Buffer size, and after messages are accumulated, the performance will not degrade too much, because the amount of data in memory has a limited impact on the external access capacity provided.

(2) messages are piled into persistent storage systems, such as DB,KV storage, file records. When a message cannot be hit by a memory Cache, it is inevitable to access the disk, resulting in a large number of read IO, and the throughput of the read IO directly determines the access capability after message accumulation.

There are four main points to evaluate the ability to accumulate messages:

(1)。 How many messages can be stacked and how many bytes can be accumulated? That is, the stacking capacity of messages.

(2)。 After the message is accumulated, will the throughput of the message be affected by the accumulation?

(3)。 Will the normal consumption of Consumer be affected after messages accumulate?

(4)。 After messages are piled up, how much do you swallow when accessing messages that are stacked on disk?

13. Distributed transaction

Several known distributed transaction specifications, such as XA,JTA. Among them, XA specification is widely supported by major database manufacturers. Such as: Orcal, Mysql and so on.

XA is a two-phase commit protocol, which is divided into the following two phases: the first phase: the transaction coordinator requires each database involving a transaction to pre-commit (precommit) this operation and reflects whether it can be committed.

The second phase: the transaction coordinator requires each database to submit data. The way Rocketmq implements transactions is not through kv storage, but through offset access to messages. There is a significant drawback, that is, using offset to change data will cause too many dirty pages in the system and require special attention.

14. Timing message

Timed message means that after a message is sent to broker, it cannot be consumed by Conusmer immediately. It can only be consumed at a specific point in time or after waiting for a specific time. Rocketmq supports timing messages, but does not support any time precision, and supports specific level, such as timing 5smeme 10srec 1m and so on.

15. Message retry

After Consumer fails to consume messages, a retry mechanism is provided to make the messages be consumed again. Failure to consume messages is generally considered to have the following situations:

1) due to the message itself, such as deserialization failure, the message data itself cannot be processed. This kind of error usually needs to skip this message and consume other messages, and even if this failed message is retried immediately, 99% will not succeed, so it is best to provide a regular retry mechanism, that is, to retry after 10 seconds.

2) due to the unavailability of dependent downstream application services, such as unavailable db connection, unreachable external network, etc. In this case, even if you skip the current failed message, consuming other messages will also report an error. In this case, it is recommended that sleep consume the next message in 30 seconds, thus reducing the pressure on broker to retry the message.

After reading the above, do you have any further understanding of the technical terms in rocketmq and the problems that message queuing needs to solve? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.