Shulou(Shulou.com)06/01 Report--

I. the way messages are stored

ActiveMQ supports persistent and non-persistent messages in the JMS specification.

Persistent messages are often used to ensure that messages are consumed by consumers, whether or not consumers are online. When the message is confirmed to be consumed, it will be deleted from the storage

Non-persistent messages are usually used to send notifications and real-time data, usually requiring performance priority, and message reliability is not necessary.

MQ supports pluggable message storage, such as memory, files, and relational databases.

Storage of Queue message Model in ActiveMQ

Using first-in, first-out (FIFO) storage, a message can only be consumed by one consumer, and will not be deleted until the message is confirmed to be consumed.

Topic message model (for persistent subscriptions)

The message obtained by each subscriber is actually a copy of the message, and only one copy of the message is stored. MQ provides a pointer to the message store and distributes copies of the message to the subscriber, and the message cannot be deleted until all persistent subscribers are received.

Persistent storage method:

KahaDB message Store

AMQ message Store

JDBC message Store

Memory message storage

Second, KahaDB storage mode

KahaDB is the default persistence plug-in since ActiveMQ 5. 4. KahaDb takes much less time to recover than its predecessor AMQ and uses fewer data files, so the persistence mechanism that can completely replace AMQ,kahaDB is also based on log files, indexes, and caches.

(1). The main characteristics of KahaDB:

Store messages in log form

The message index is stored in B-Tree structure and can be updated quickly.

Full support for JMS transactions

Support for multiple recovery mechanisms

(2) applicable scenarios:

High-throughput applications

Store messages with a large amount of data

(3) configuration method conf/activemq.xml:

(4) KahaDB storage principle:

When there is an active consumer, for temporary storage, the message is sent to the consumer and arranged to be stored, and if the message is acknowledged in time, it does not need to be written to disk. A data message written to disk that, in subsequent message activities, is marked as deletable if the message is successfully sent. The system periodically clears or archives log files.

1. The internal structure of KahaDB

Data logs: message log contains message log and some commands

Cache: when there is an active consumer, for temporary storage, the message will be sent to the consumer and arranged to be stored. If the message is confirmed in time, it does not need to be written to disk.

Btree indexes (message index): used to refer to the message log (message id), which is stored in memory so that it can be quickly located. MQ periodically saves the message index in memory to metadata store to avoid taking up too much memory space when a large number of messages are not sent.

Redo log is used to maintain index integrity in the event of an abnormal shutdown.

2. Directory structure:

Db log files: used to store messages (default size 32m). When the log log is full, a new one will be created. When all the messages in the log log are deleted, the log file will be deleted or archived.

Archive directory: when datalog is no longer needed by kahadb, it is archived (controlled by the archiveDataLogs attribute).

Db.data: store Btree indexs.

Db.redo: stores redo file, which is used to restore Btree indexs.

III. AMQ message storage

When a message is written, the message is written to the log file, and the performance is high because it is sequentially appended. To improve performance, create a message primary key index and provide a caching mechanism to further improve performance. The size of each log file is limited (the default is 32m, which can be configured by yourself). When this size is exceeded, the system will re-create a file. When all messages are consumed, the system deletes the file or archives (depending on the configuration). The main disadvantage is that AMQ Message creates an index for each Destination, and if you use a lot of Queue, the size of the index file takes up a lot of disk space. And because the index is huge, once the Broker crashes, the speed of rebuilding the index will be very slow.

Features: similar to KahaDB, it also includes transaction logs. Each destination contains an index file. AMQ is suitable for high-throughput application scenarios, but not for multi-queue scenarios.

Configuration method conf/activemq.xml:

1. Internal structure of AMQ:

Data logs: message log contains message log

Cache: for fast message retrieval

Reference store indexes: used to reference messages in datalogs, associated through message ID

2. Directory structure:

Lock: ensure that only one borker accesses the file directory at a time

Temp-storag: used to store non-persistent messages (when not stored in memory), such as waiting for slow consumers to process messages

Kr-store: used to store reference message log data

Journal directory: contains message files, message logs, and message control information

Archive: archived data log

IV. JDBC storage

Support for storing messages to relational databases through JDBC, which is not as good as file storage in performance, and can query the information of messages through relational databases.

Databases supported by MQ: Apache Derby, MYsql, PostgreSQL, Oracle, SQLServer, Sybase, Informix, MaxDB.

Storage table structure:

A, ACTIVEMQ_MSGS: used to store messages, Queue and Topic are stored in this table:

ID: self-increasing database primary key

CONTAINER: the Destination of the message

MSGID_PROD: the primary key of the message sender client

MSG_SEQ: the order in which messages are sent. MSGID_PROD+MSG_SEQ can make up the MessageID of JMS.

EXPIRATION: the expiration time of the message, which stores the number of milliseconds from 1970-01-01 to the present

MSG: Java of the message ontology serializes the binary data of the object

PRIORITY: priority, from 0 to 9. The higher the number, the higher the priority.

B, ACTIVEMQ_ACKS: used to store subscription relationships. If the Topic is persisted, the subscription relationship between the subscriber and the server is saved in this table:

The main database fields are as follows:

CONTAINER: the Destination of the message

SUB_DEST: if you are using a Static cluster, this field will contain information about other systems in the cluster

CLIENT_ID: each subscriber must have a unique client ID to distinguish between

SUB_NAME: subscriber name

SELECTOR: selector, you can choose to consume only messages that meet the criteria. Conditions can be implemented with custom attributes and support multi-attribute AND and OR operations

LAST_ACKED_ID: the ID that records consumed messages.

C, ACTIVEMQ_LOCK (message lock, which ensures that only one broker can access these table structures at a time):

Table activemq_lock is useful in a clustered environment, where only one Broker can get a message called Master Broker, and the rest can only be used as backups when Master Broker is not available before it can become the next Master Broker. This table is used to record which Broker is the current Master Broker.

Configuration method:

1. Configure the data source conf/acticvemq.xml file:

2. Configure persistenceAdapter in broke:

DataSource specifies whether the bean,createTablesOnStartup of the persistent database creates a data table at startup. The default value is true, so that the data table will be created every time it starts. It is usually set to true at the first startup, and then changed to false.

Ps: the database activemq needs to be created manually.

5. Memory message storage

In-memory message storage, all persistent messages will be stored in memory. You must pay attention to JVM usage and memory limitations. It is suitable for small messages that can be consumed quickly. When MQ shuts down or goes down, unconsumed memory messages will be emptied.

The configuration method sets the broker attribute value persistent= "false":

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.