Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what the broker message storage architecture is like in RocketMQ". It is easy to understand and well organized. I hope it can help you solve your doubts. Let me lead you to study and learn "what the broker message storage architecture is like in RocketMQ".

I. the overall architecture of message storage

All topic messages under a broker are sequentially stored in the same data log file (commit log file). At the same time, broker also maintains an index file (index file) that records the reverse index relationship of msg key, which is used to find all associated msgs from key. For the consumer side, broker creates a logical consumption queue (ConsumeQueue) for each subqueue under topic. The element stored in ConsumeQueue has a fixed length of 20 bytes and is a pointer to the message entity in commitlog. The consumption progress offset of each consumer is essentially the index of the elements in the ConsumeQueue, and the specific message body can be obtained through commitlog offset.

When broker receives a message sent by producer, it will first add the message to the end of the commitlog file and refresh the disk synchronously or asynchronously. There will be a thread of ReputMessageService to build ConsumeQueue and IndexFile asynchronously. The core storage file is shown below:

2. Hierarchical structure of storage model

Broker's message storage architecture can be divided into the following levels according to dependencies:

Among them, MappedFileQueue/MappedFile encapsulates the underlying file read-write interface. RocketMQ uses MappedByteBuffer/FileChannel in JDK NIO to use the operating system's file mapping service to map a large file directly to a memory address space. The read and write operation of the file is realized through the reading and writing of memory, which avoids the copy of data from kernel state to user mode. MappedFile represents a log file that is sliced into a fixed size (the default shard size for commitlog is 1G, and the file name is the offset of the first byte of the shard throughout the commitlog). Multiple MappedFile form a MappedFileQueue to form a complete view of the log file.

Third, core implementation class

The core implementation class of message storage is shown below:

DefaultMessageStore is the entrance of the whole storage subsystem, which provides the method of message writing / reading / querying, and constructs two key data structures, ConsumeQueue and IndexFile, asynchronously through backstage threads.

These are all the contents of the article "what is the broker message storage architecture in RocketMQ?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.