Shulou(Shulou.com)06/01 Report--

This article mainly explains "the deployment process of RocketMQ". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the deployment process of RocketMQ".

1. RocketMQ introduction 1.1. Brief introduction

RocketMQ is a distributed, queuing model message middleware with the following characteristics:

It can guarantee strict message order, provide rich message pull mode, efficient subscriber level expansion ability, real-time message subscription mechanism, 100 million message stacking ability.

Reasons for selection:

It is emphasized that the cluster has no single point, can be expanded, any point is highly available, and can be expanded horizontally.

Massive message accumulation ability, after message accumulation, write low latency.

Support tens of thousands of queu

Message failure retry mechanism

Messages can be queried

The open source community is active

Maturity (after the test of Singles Day)

1.2. Key concepts

1.2.1. Theme and label

Topic Tpoic: first-level message type, book title label Tags: second-level message type, book catalog, message filtering based on Tag

For example:

Topic: order transaction

Label: order transaction-create order transaction-payment order transaction-complete

1.2.2. Send and subscribe groups

Production group: used for sending messages.

Consumer group: used for subscription processing of messages.

Production group and consumer group, easy to expand and shrink the machine, increase or decrease processing capacity, the name of the cluster group, used to mark a member on the way. Only one member of each cluster will be sent randomly at a time.

2. RocketMQ cluster mode

Several Broker cluster deployment methods are recommended. The Slave here is not writable but readable, similar to the Mysql master / slave mode.

Operating system: 64bit OS, Linux/Unix/Mac is recommended

Memory at least: 8 GB recommended 16 GB

2.1. Single Master

This method is risky. Once the Broker is restarted or downtime, the entire service will be unavailable, and it is not recommended to use it in the online environment.

2.2. Multi-Master mode

A cluster has no Slave and is all Master, such as 2 Master or 3 Master

Advantages: simple configuration, single Master downtime or restart maintenance will not affect the application. When the disk is configured as RAID10, even if the machine outage is unrecoverable, the disk by and RAID10 is very reliable, and the message will not be lost (a small amount of messages will be lost in asynchronous flushing, and none will be lost in synchronous flushing)

. The highest performance.

Disadvantages: during the downtime of a single machine, messages that have not been consumed on this machine cannot be subscribed until the machine is restored, and the real-time performance of messages will be affected.

# start NameServer first

# launch the first Master on machine A

# launch the second Master on machine B

2.1. Multi-Master and multi-Slave mode, asynchronous replication

Each Master is configured with one Slave with multiple pairs of Master-Slave,HA

Using asynchronous replication mode, the master and standby have short message delay, millisecond level. Advantages: even if the disk is damaged, very few messages are lost, and the real-time performance of messages will not be affected because

After the Master downtime, consumers can still access the Slave

Consumption, this process is transparent to the application. No human intervention is required. The performance is almost the same as the multi-Master mode. Disadvantages: Master downtime, disk damage, will lose a small number of messages.

# start NameServer first

# launch the first Master on machine A

# launch the second Master on machine B

# launch the first Slave on machine C

# launch the second Slave on machine D

2.2. Multi-Master and multi-Slave mode, synchronous double write

Each Master is configured with one Slave with multiple pairs of Master-Slave,HA

Using synchronous double-write mode, both master and backup are successfully written, and success is returned to the application.

Advantages: no single point of data and service, no message delay in case of Master downtime, high service availability and data availability.

Disadvantages: slightly lower performance than asynchronous replication mode, about 10% lower, RT that sends a single message

It will be a little higher. At present, after the master downtime, the slave cannot be automatically switched to the host, and the automatic switching feature will be supported later.

# start NameServer first

# launch the first Master on machine A

# launch the second Master on machine B

# launch the first Slave on machine C

# launch the second Slave on machine D

The above Broker and Slave pairs are paired by specifying the same brokerName parameters. The BrokerId of Master must be 0 and the BrokerId of Slave must be large and 0. Multiple Slave can be mounted under another Master, and multiple Slave under the same Master can be distinguished by specifying different BrokerId.

3. RocketMQ deployment [dual Master] 3.1. Server environment

Serial number

IP

User name

Password

Role

Pattern

one

192.168.100.24

Root

NameServer1,brokerServer1

Master1

two

192.168.100.25

Root

NameServer2,brokerServer2

Master2

3.2. Hosts add information [two machines] # vi / etc/hosts [root@cs11-cluster-2 rocketmq] # cat / etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.100.24 rocketmq-master1 rocketmq-nameserver1192.168.100.25 rocketmq-master2 rocketmq-nameserver2

IP

NAME

192.168.100.24

Rocketmq-nameserver1

192.168.100.24

Rocketmq-master1

192.168.100.25

Rocketmq-nameserver2

192.168.100.25

Rocketmq-master2

3.3. Upload and decompress [two machines] installation package acquisition method 1, the latest code: download git code: https://github.com/apache/incubator-rocketmq.git installation: 64bit JDK 1.8upload / Maven 3.2.x enter the project to execute mvn compilation package: cd incubator-rocketmqmvn-Prelease-all-DskipTests clean install-Ucd distribution/target/ will push the packaged code to two servers: apache-rocketmq.tar.gz or apache-rocketmq.zip

Installation package acquisition method 2: download git code: https://github.com/apache/incubator-rocketmq.git into the project package: cd incubator-rocketmq/: distribution package, push to two servers. 3.4. Create a storage path [two machines] # mkdir / usr/local/rocketmq/store# mkdir / usr/local/rocketmq/store/commitlog# mkdir / usr/local/rocketmq/store/consumequeue# mkdir / usr/local/rocketmq/store/index3.5. RocketMQ profile [two machines]

# vim / usr/local/rocketmq/conf/2m-noslave/broker-a.properties

# vim / usr/local/rocketmq/conf/2m-noslave/broker-b.properties

# brokerClusterName=rocketmq-cluster#broker name of the cluster to which it belongs. Note that different broker-an is entered in different configuration files here | broker-bbrokerName=broker-a#0 indicates Master, > 0 indicates Slave brokerId=0#nameServer address, and semicolon is divided into namesrvAddr=rocketmq-nameserver1:9876. When rocketmq-nameserver2:9876# sends messages, it automatically creates topic for which the server does not exist. Whether the number of queues created by default defaultTopicQueueNums=4# allows Broker to automatically create Topic, it is recommended that offline enable, online shutdown autoCreateTopicEnable=true# allows Broker to automatically create subscription groups, recommended offline enable, online shut down autoCreateSubscriptionGroup=true#Broker external service listening port listenPort=10911# delete file point, default 4: 00 am deleteWhen=04# file retention time Default 48 fileReservedTime=120#commitLog size per file default 1G 1024 * 1024 * 1024mapedFileSizeCommitLog=1073741824# detect physical file disk space diskMaxUsedSpaceRatio=88# storage path storePathRootDir=/usr/local/rocketmq/store#commitLog storage path storePathCommitLog=/usr/local/rocketmq/store/commitlog# consumption queue storage path storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue# message index storage path storePathIndex=/usr/local/rocketmq/store/index#checkpoint file storage path storeCheckpoint=/usr/local/rocketmq/ Store/checkpoint#abort file storage path abortFile=/usr/local/rocketmq/store/abort# limited message size maxMessageSize=65536#flushCommitLogLeastPages=4#flushConsumeQueueLeastPages=2#flushCommitLogThoroughInterval=10000#flushConsumeQueueThoroughInterval=60000#Broker role #-ASYNC_MASTER asynchronous replication Master#- SYNC_MASTER synchronous double write Master#- SLAVEbrokerRole=ASYNC_MASTER# flush mode #-ASYNC_FLUSH asynchronous flush disk #-SYNC_FLUSH synchronous flush flushDiskType=ASYNC_FLUSH#checkTransactionMessageEnable=false# number of sending thread pool # sendMessageThreadPoolNums=128# pull message thread pool # pullMessageThreadPoolNums=128

3.6. Modify log configuration file [two machines]

# mkdir-p / usr/local/rocketmq/logs# cd / usr/local/rocketmq/conf & & sed-I's xml3.7 ${user.home} # / usr/local/rocketmq#g' *. Xml3.7. Modify the startup script parameter [two machines] (heap memory is set to at least 1g, otherwise errors will occur)

# vim / usr/local/rocketmq/bin/runbroker.sh

# = Development Environment JVM Configuration#====JAVA_OPT= "${JAVA_OPT}-server-Xms1g-Xmx1g-Xmn512m-XX:PermSize=128m-XX:MaxPermSize=320m"

# vim / usr/local/rocketmq/bin/runserver.sh

JAVA_OPT= "${JAVA_OPT}-server-Xms1g-Xmx1g-Xmn512m-XX:PermSize=128m-XX:MaxPermSize=320m" 3.8. Start NameServer [two machines] [root@cs11-cluster-1 rocketmq] # cat shell/start_namesvr.sh #! / bin/shnohup sh / usr/local/rocketmq/bin/mqnamesrv 2 > & 11 > / usr/local/rocketmq/logs/start_namesvr.log & # netstat-ntlp # jps

3.9. Start BrokerServer A [192.168.100.24]

[root@cs11-cluster-1 rocketmq] # cat shell/start_broker-a.sh #! / bin/shnohup sh / usr/local/rocketmq/bin/mqbroker-c / usr/local/rocketmq/conf/2m-noslave/broker-a.properties 2 > & 11 > / usr/local/rocketmq/logs/start_broker-a.log & # netstat-ntlp# jps# tail-f-n 500 / usr/local/rocketmq/logs/rocketmqlogs/broker.log# tail-f -n 500 / usr/local/rocketmq/logs/rocketmqlogs/namesrv.log3.10. Launch BrokerServer B [192.168.100.25] [root@cs11-cluster-1 rocketmq] # cat shell/start_broker-b.sh #! / bin/shnohup sh / usr/local/rocketmq/bin/mqbroker-c / usr/local/rocketmq/conf/2m-noslave/broker-b.properties 2 > & 11 > / usr/local/rocketmq/logs/start_broker-b.log & # netstat-ntlp# jps# tail-f-n 500 / usr/local/rocketmq/ Logs/rocketmqlogs/broker.log# tail-f-n 500 / usr/local/rocketmq/logs/rocketmqlogs/namesrv.log data transmission Accept verification: Produce: > export NAMESRV_ADDR=192.168.100.24:9876 > sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer SendResult [sendStatus=SEND_OK, msgId=... Consummer: > export NAMESRV_ADDR=192.168.100.24:9876 > sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer ConsumeMessageThread_%d Receive New Messages: [MessageExt...3.11.RocketMQ Console

Deploy rocketmq-console.war in tomcat

3.12. Data cleaning

# cd / usr/local/rocketmq/bin# sh mqshutdown broker# sh mqshutdown namesrv#-- waiting to stop # rm-rf / usr/local/rocketmq/store# mkdir / usr/local/rocketmq/store# mkdir / usr/local/rocketmq/store/commitlog# mkdir / usr/local/rocketmq/store/consumequeue# mkdir / usr/local/rocketmq/store/index#-- restart NameServer and BrokerServer by following the above steps

3.13. Abnormal error

# the following exception occurred: 2017-10-09 22:04:59 INFO main-load / opt/rocketmq/apache-rocketmq/store/config/consumerOffset.json OK2017-10-09 22:04:59 INFO main-load / opt/rocketmq/apache-rocketmq/store/config/consumerFilter.json OK2017-10-09 22:04:59 INFO main-shutdown thread PullRequestHoldService interrupt false2017-10-09 22:04:59 INFO main-join thread PullRequestHoldService eclipse time (ms) 09000 possible cause: incorrect configuration path Or store file file creation error, Note: store creation can only manually create the following folder: commitlog consumequeue index thank you for reading, the above is the "RocketMQ deployment process" of the content, after the study of this article, I believe you have a deeper understanding of the deployment process of RocketMQ, the specific use of the need for you to practice and verify. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.