Shulou(Shulou.com)06/01 Report--

1. Recently, the company is doing the technical selection of queues. After investigation and research, we decided to use rocketmq as the queue layer of the entire architecture. Our previous company used RabbitMQ, cluster deployment refer to my previous article: RabbitMQ cluster deployment; rocketmq cluster is composed of NameServer and Broker roles. NameServer is stateless and can be deployed horizontally to achieve the purpose of eliminating a single point. Broker is divided into three deployment scenarios: multi-master, multi-master multi-slave synchronous, and multi-master multi-slave async. Generally, production environments use multi-master and multi-slave async. The advantages and disadvantages of these three solutions are compared as follows:

Multi-Master mode (2m-noslave):

A cluster has no Slave and is all Master, such as 2 Master or 3 Master

Advantages: simple configuration, single Master downtime or restart maintenance will not affect the application. When the disk is configured as RAID10, even if the machine outage is unrecoverable, messages will not be lost because the RAID10 disk is very reliable (a small amount of messages will be lost in asynchronous flushing, and none will be lost in synchronous flushing). The highest performance.

Disadvantages: during the downtime of a single machine, messages that are not consumed on this machine cannot be subscribed until the machine is restored, and the real-time performance of messages will be affected.

Multi-Master multi-Slave mode, asynchronous replication (2m-2s-async):

Each Master is configured with a Slave. Multiple pairs of Master-Slave,HA are replicated asynchronously, and the master / slave has a short message delay of millisecond.

Advantages: even if the disk is damaged, very few messages are lost, and the real-time performance of messages will not be affected, because after Master downtime, consumers can still consume from Slave, this process is transparent to the application. No human intervention is required. The performance is almost the same as the multi-Master mode.

Disadvantages: Master downtime, disk damage, will lose a small number of messages.

Multi-Master and multi-Slave mode, synchronous double write (2m-2s-sync)

Each Master is configured with a Slave. Multiple pairs of Master-Slave,HA are synchronously double-written. Both master and slave are successfully written, and success is returned to the application.

Advantages: no single point of data and service, no message delay in case of Master downtime, high service availability and data availability.

Cons: performance is slightly lower than asynchronous replication mode, about 10% lower, and the RT for sending a single message is slightly higher. At present, after the master downtime, the slave cannot be automatically switched to the host, and the automatic switching feature will be supported later.

II. Environment

1. Version

JDK version: openjdk version "1.8.0mm 161"

Rocketmq version: 4.2.0

Rocketmq-console-ng version: 1.0.0

2. Role

Rocketmq-console-ng (Monitoring and Management):

192.168.54.26:8080

NameServer:

192.168.47.183:9876

192.168.47.184:9876

Broker:

192.168.47.183:10911 master broker-a

192.168.47.184:10911 slave broker-a

192.168.47.185:10911 master broker-b

192.168.47.186:10911 slave broker-b

III. Deployment

1. Download the binary package and decompress it

Mkdir / chj/app/rocketmq-psetCD / chj/app/rocketmqwget http://mirrors.shu.edu.cn/apache/rocketmq/4.2.0/rocketmq-all-4.2.0-bin-release.zip

Unzip rocketmq-all-4.2.0-bin-release.zip

2. Write configuration files. The default configuration can not meet our needs, so we need to add customized parameters. Since we are deploying an asynchronous solution with multi-master and multi-slave, we can modify or add the configuration files under the / chj/app/rocketmq/conf/2m-2s-async directory (it is recommended that all configuration files be written on one node of broker and distributed to all nodes. Keep the configuration files of all nodes consistent, and different nodes can use different configuration files when starting service, which is convenient for later maintenance)

Differential configuration of An and 4 files

Vim broker-a.propertiesbrokerName=broker-abrokerId=0brokerRole=ASYNC_MASTERvim broker-a-s.propertiesbrokerName=broker-abrokerId=1brokerRole=SLAVEvim broker-b.propertiesbrokerName=broker-bbrokerId=0brokerRole=ASYNC_MASTERvim broker-b-s.propertiesbrokerName=broker-bbrokerId=1brokerRole=SLAVE

B. at the beginning of the configuration above, each configuration file is different, and the configuration below is the same

# brokerClusterName=rocketmq-cluster-test#broker name of the cluster to which it belongs. Note that different configuration files are entered here. # brokerName=broker-a#0 indicates Master, > 0 indicates Slave#brokerId=0#nameServer address, and semicolon divides namesrvAddr=192.168.47.183:9876. 192.168.47.184Broker 987mm automatically creates topic that does not exist on the server when sending messages. Whether the number of queues created by default defaultTopicQueueNums=4# allows Broker to automatically create Topic, it is recommended that offline enable, online shutdown autoCreateTopicEnable=true# allows Broker to automatically create subscription groups, recommended offline enable, online closed autoCreateSubscriptionGroup=true#Broker external service listening port listenPort=10911# delete file time point, default deleteWhen=04# file retention time at 4: 00 am Default 48-hour fileReservedTime=120#commitLog size per file default 1GmapedFileSizeCommitLog=1073741824#ConsumeQueue 30W per file by default Adjust mapedFileSizeConsumeQueue=300000#destroyMapedFileIntervalForcibly=120000#redeleteHangedFileInterval=120000# detection physical file disk space diskMaxUsedSpaceRatio=88# storage path storePathRootDir=/chj/data/rocketmq/store#commitLog storage path storePathCommitLog=/chj/data/rocketmq/store/commitlog# consumption queue storage path storePathConsumeQueue=/chj/data/rocketmq/store/consumequeue# message index storage path storePathIndex=/chj/data/rocketmq/store/index#checkpoint file storage path storeCheckpoint=/chj/data/rocketmq/store/checkpoint# Abort file storage path abortFile=/chj/data/rocketmq/store/abort# limited message size maxMessageSize=65536#flushCommitLogLeastPages=4#flushConsumeQueueLeastPages=2#flushCommitLogThoroughInterval=10000#flushConsumeQueueThoroughInterval=60000#Broker role #-ASYNC_MASTER asynchronous replication Master#- SYNC_MASTER synchronous double write Master#- SLAVE#brokerRole=ASYNC_MASTER# flush mode #-ASYNC_FLUSH asynchronous flush disk #-SYNC_FLUSH synchronous flush flushDiskType=ASYNC_FLUSH#checkTransactionMessageEnable=false# number of sending thread pool # sendMessageThreadPoolNums=128# pull message thread pool # pullMessageThreadPoolNums=128

3. Start the service

NameServer startup (both nodes start in the same way)

Cd / chj/app/rocketmq/&& nohup. / bin/mqnamesrv &

Start Broker on 192.168.47.183

Cd / chj/app/rocketmq/&& nohup sh / chj/app/rocketmq/bin/mqbroker-c / chj/app/rocketmq/conf/2m-2s-async/broker-a.properties > / chj/log/rocketmq/broker-a.log > / dev/null 2 > & 1 &

Start Broker on 192.168.47.184

Cd / chj/app/rocketmq/&& nohup sh / chj/app/rocketmq/bin/mqbroker-c / chj/app/rocketmq/conf/2m-2s-async/broker-a-s.properties > / chj/log/rocketmq/broker-a-s.log > / dev/null 2 > & 1 &

Start Broker on 192.168.47.185

Cd / chj/app/rocketmq/&& nohup sh / chj/app/rocketmq/bin/mqbroker-c / chj/app/rocketmq/conf/2m-2s-async/broker-b.properties > / chj/log/rocketmq/broker-b.log > / dev/null 2 > & 1 &

Start Broker on 192.168.47.186

Cd / chj/app/rocketmq/&& nohup sh / chj/app/rocketmq/bin/mqbroker-c / chj/app/rocketmq/conf/2m-2s-async/broker-b-s.properties > / chj/log/rocketmq/broker-b-s.log > / dev/null 2 > & 1 &

IV. Rocketmq-console-ng deployment

Rocketmq-console-ng is a visual monitoring tool of rocketmq, project address: github address, it can be deployed in the traditional way or docker deployment mode, we choose docker this way of deployment.

Docker pull styletang/rocketmq-console-ngdocker run-itd-e "JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.47.183:9876-Dcom.rocketmq.sendMessageWithVIPChannel=false"-p 8080 JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.47.183:9876 8080-t styletang/rocketmq-console-ng

Visit 192.168.54.26 cluster 8080 to view its status

PS: reference link

Http://blog.csdn.net/lovesomnus/article/details/51769977

Http://blog.csdn.net/jayjjb/article/details/70140667

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.