Shulou(Shulou.com)05/31 Report--

This article is to share with you about the characteristics of RocketMQ. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

RocketMQ is a distributed, queuing model message middleware with the following characteristics:

1. Can guarantee strict message order

two。 Provide rich message pull mode

3. Efficient subscriber horizontal scalability

4. Real-time message subscription mechanism

5. 100 million-level message accumulation capability

I. characteristics of RocketMQ network deployment

(1) NameServer is an almost stateless node that can be deployed in clusters without any information synchronization between nodes.

(2) Broker deployment is relatively complex. Broker atmosphere Master and Slave, one Master can correspond to multiple Slaver, but one Slaver can only correspond to one Master,Master and Slaver. The corresponding relationship between Master,Master and Slaver is defined by specifying the same BrokerName and different BrokerId. If BrokerId is 0, it means Master, and non-zero means Slaver. Master can deploy multiple. Each Broker establishes persistent connections with all nodes in the NameServer cluster and registers Topic information to all NameServer regularly.

(3) Producer establishes a persistent connection with one of the nodes in the NameServer cluster (randomly selected), periodically takes Topic routing information from NameServer, establishes a long connection to Master that provides Topic services, and regularly sends heartbeats to Master. Produce is completely stateless and can be deployed in clusters

(4) Consumer establishes a persistent connection with one of the nodes in the NameServer cluster (randomly selected), periodically takes Topic routing information from NameServer, establishes a long connection to Master and Slaver providing Topic services, and regularly sends heartbeats to Master and Slaver. Consumer can subscribe to messages either from Master or from Slave. The subscription rules are determined by Broker configuration.

II. Storage characteristics of RocketMQ

(1) Zero copy principle: Consumer consumes the message process, using zero copy, zero copy includes the following 2 methods, RocketMQ uses the first method, because the effect of small block data transmission is better than sendfile mode

A) use the mmap+write method

Advantages: even if it is called frequently, it is very efficient to use small file blocks to transfer.

Disadvantages: can not make good use of DMA, will consume more CPU resources than sendfile, memory security control is complex, need to avoid JVM Crash problems

B) use the sendfile method

Advantages: DMA mode can be used, less CPU resources are consumed, high efficiency of transferring large chunks of files, and new security problems without memory

Disadvantages: the efficiency of small block files is lower than that of mmap mode, so it can only be transferred by BIO mode, and cannot be used by NIO.

(2) data storage structure

III. Key features of RocketMQ

1. Stand-alone supports more than 1W persistence queues

(1) all data are stored separately in commit Log, written sequentially and read randomly.

(2) for the queue displayed by the end user, only the location information of the message in the Commit Log is actually stored, and the disk is flushed in a serial way.

The benefits of this:

(1) the queue is lightweight, and the amount of data in a single queue is very small.

(2) access to disk in serial jargon to avoid disk competition, and IOWait will not increase due to the increase of queues

Each scheme has its advantages and disadvantages, and its disadvantages are:

(1) although writing is written sequentially, reading becomes random reading.

(2) if you read a message, you will read Consume Queue first, and then Commit Log, which increases the overhead.

(3) to ensure that Commit Log is completely consistent with Consume Queue, which increases the complexity of programming.

How to serve the customer with the above shortcomings:

(1) Random read, let the read hit pagecache as much as possible, and reduce IO operations, so the larger the memory, the better. If there are too many messages in the system, will the read data access the hard disk lead to a sharp decline in system performance due to random reading? the answer is no.

A) when accessing pagecache, even if you only access 1K messages, the system will pre-read more data in advance and may hit pagecache the next time you read it.

B) random access to Commit Log disk data, and the system IO scheduling algorithm is set to NOOP mode, which will change the complete random read into sequential hopping mode to a certain extent, and the performance of sequential hopping mode is 5 times higher than that of complete random read.

(2) because the amount of Consume Queue storage is very small and read sequentially, the read performance of Consume Queue is almost always with memory in the case of pagecache and read, even in the case of stacking. So it can be considered that Consume Queue will not hinder read performance at all.

(3) all the meta-information is stored in Commit Log, including the message body, which is similar to the redolog of MySQl and Oracle, so as long as Commit Log exists, Consume Queue can still recover it even if it loses data.

two。 Brushing strategy

All messages in rocketmq are persistent. Write to the system pagecache first, and then brush the disk to ensure that both memory and disk have a piece of data. When accessed, it can be read directly from memory.

2.1 Asynchronous swiping disk

In the RAID card, SAS 15000 disk test sequence to write files, the speed can reach about 300m per second, and online network cards are generally gigabit network card, write disk speed is significantly faster than the data network entry speed, then can you write memory back to the user, by the background thread to brush the disk?

(1)。 Since the speed of the disk is faster than that of the network card, the progress of flushing the disk can certainly keep up with the writing speed of the message.

(2)。 In case, due to excessive system pressure at this time, messages may be piled up, including not only writing IO, but also reading IO. If there is a lag in disk reading, will it lead to system memory overflow? the answer is no for the following reasons:

A) when writing messages to PAGECACHE, if there is not enough memory, try to discard clean PAGE and free up memory for new messages. The policy is LRU.

B) if there are not enough clean pages, writing to PAGECACHE will be blocked at this time, and the system will try to flush some of the data, about 32 PAGE at a time, to find more clean PAGE.

To sum up, memory overflows will not occur.

2.2 synchronously refresh disk:

The only difference between synchronous flushing and asynchronous flushing is that PAGECACHE is returned directly after asynchronous flushing, while synchronous flushing needs to wait for the completion of flushing. The synchronous flushing process is as follows:

(1) after writing the PAGECACHE, the thread waits and notifies the flushing thread to flush the disk.

(2) after brushing the disk, the thread wakes up the front end waiting thread, which may be a batch of threads.

(3) the front-end waiting thread returns success to the user.

3. Message query

3.1 query messages by MessageId

The MsgId has a total of 16 bytes, including the message storage host address and the message Commit Log Offset. Parse the Broker address and Commit Log offset address from MsgId, and then parse the location message buffer into a complete message according to the storage format.

3.2 query messages by Message Key

1. Get the specific slot location according to the hashcode%slotNum of the queried key (slotNum is the maximum number of slots contained in an index file, such as the slotNum=500W shown in the figure)

two。 Find the last item in the list of index items according to slotValue (the value of the corresponding position of slot) (in reverse order, slotValue always points to the latest index entry)

3. Traversing the list of index items returns the result set within the query time range (the default maximum of 32 records returned at a time)

4.Hash conflict, finding the slot location of key is equivalent to executing two hash functions, once the hash of key and the hash mode of key, so there are two conflicts. First, the hash value of key is different but the modulus is the same. In this case, the query will compare the hash value of the first key (each index entry holds the hash value of key) and filter out the situation where the hash value does not want to wait. Second, the hash value is equal and key does not want to wait. For performance reasons, conflict detection is placed on the client side (the original value of key is stored in the message file to avoid parsing the data file), and the client compares whether the key of the message body is the same.

5. Storage, in order to save space, the time stored in the index item is the time difference (the storage time-start time, the start time is stored in the index file header), the whole index file is fixed in length and the structure is fixed.

4. Server message filtering

The message filtering method of RocketMQ is different from other message middleware. When you subscribe, you filter again. Let's first take a look at the Consume Queue storage structure.

1. For Message Tag comparison on the broker side, first traverse the Consume Queue. If the stored Message Tag does not match the subscribed Message Tag, skip it, continue to compare the next one, and transmit it to the Consumer if it matches. Note that Message Tag is in the form of a string, and what is stored in Consume Queue is its corresponding hashcode. When comparing, it is also comparing hashcode.

When 2.Consumer receives the filtered message, it also performs the operation on the broker side, but compares the real Message Tag string, not the hashcode.

Why does filtering do this?

1.Message Tag stores hashcode in order to save space in Consume Queue fixed-length storage.

two。 Commit Log data will not be accessed in the filtering process, which can ensure efficient filtering in the case of accumulation.

3. Even if there is a hash conflict, it can be corrected on the consumer side to ensure that it is foolproof.

5. A single JVM process can also take advantage of machine memory.

1.Producer sends messages that enter the java heap from socket

2.Producer sends messages from the java heap to pagecache, physical memory

3.Producer sends messages, which are flushed by asynchronous threads, and messages are brushed from pagecache to disk.

4.Consumer pulls messages (normal consumption). Messages are transferred directly from pagecache (data is in physical memory) to socket and to Consumer without going through the java heap. This consumption scenario is the most, with 96 GB of physical memory online. According to 1K messages, 100 million messages can be physically cached.

5.Consumer pulls messages (abnormal consumption). Messages are transferred directly from pagecache to socket.

6.Consumer pull message (abnormal consumption). Because socket accesses virtual memory, resulting in page fault, disk IO will be generated, from disk Load message to pagecache, and then sent directly from socket.

7. Same as 5

8. With 6

6. The solution to the problem of message accumulation

1 message stacking capacity, depending on disk size

(2) the throughput of messages will be affected to a certain extent in the case of no Slave, but not in the case of Slave.

(3) whether the Consumer of normal consumption will be affected, whether there is no Slave, it will be affected to a certain extent, and there will be Slave, but it will not be affected.

4 when accessing messages stacked on disk, how big the throughput is, which is related to the concurrency of access, will eventually drop to about 5000.

In the case of Slave, once Master finds that Consumer accesses data accumulated on disk, recall that Consumer issued a redirect instruction to Consumer to pull data from Slave, so that normal sending messages and normal consumption will not be affected by accumulation, because the system divides stacking scenarios and non-stacking scenarios into two different nodes for processing. The question that arises here is whether the write performance of Slave will degrade, and the answer is no. Because Slave message writing only pursues throughput, not real-time performance, as long as the overall throughput is high, and Slave always pulls a batch of data from Master, such as 1m. This batch sequential writing method makes the stacking situation relatively small, but the impact on overall throughput is relatively small, but writing to RT will be longer.

Thank you for reading! This is the end of this article on "what are the characteristics of RocketMQ?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.