Shulou(Shulou.com)06/03 Report--

RocketMQ introduction

Overview

Apache RocketMQ is a distributed message flow processing platform with low latency, high performance, high reliability, trillions of capacity, flexible and scalable. It consists of four parts: name servers, brokers, producers and consumers. All of them can be scaled horizontally to avoid a single point of failure, as shown in the figure above.

Name service group service NameServer cluster

NameServer services provide lightweight service discovery and routing. Each NameServer service records complete routing information, provides consistent read and write services, and supports rapid storage expansion

Agent service cluster Broker Cluster

Broker handles message storage by providing lightweight topics and queuing mechanisms. They support Push and Pull models, include fault-tolerant mechanisms (2 or 3 replicas), provide extremely strong peak processing capabilities and the ability to store millions of messages in chronological order. In addition, agents provide disaster recovery, rich metrics, statistics and alarm mechanisms that are lacking in traditional messaging systems.

Producer cluster Producer Cluster

Produce supports distributed deployment, and distributed produce sends messages to the broker cluster through various load balancing strategies provided by the broker cluster. The sending process supports fast failure with low latency.

Consumer cluster Consumer Cluster

Consumers also support distributed deployment in push or pull mode, and it also supports cluster consumption and message broadcasting. Provide real-time message subscription mechanism, which can meet the needs of most consumers. RocketMQ's website provides a simple quick start guide for interested users.

Name Service NameServer

NameServer is a full-featured server with two main functions:

Managed by ⊙ broker, nameserver accepts registration information from the broker cluster and provides a heartbeat to detect whether they are available.

⊙ routing management, where each nameserver holds all routing information about broker clusters and queues and is used to provide queries to clients.

We know that the rocketMQ client (producer / consumer) will query the routing information of the queue from nameserver. How does the client know the address of nameserver?

There are four ways for the client to obtain the address of nameserver:

⊙ through the program, like this producer.setNamesrvAddr ("ip:port")

⊙ java configuration item, so use rocketmq.namesrv.addr

⊙ environment variable NAMESRV_ADDR

⊙ HTTP Endpoint

For more details on nameserver address discovery, please refer to here

Agent Service broker server

Broker server is responsible for message storage and delivery, message query, ensuring high availability, and so on.

As shown in the following figure, broker server has some very important submodules:

The ⊙ remoting (remote) module, the entrance to broker, handles requests initiated from the client.

⊙ client manager (client management), managing individual clients (producers / consumers) and maintaining consumer topic subscriptions.

⊙ store (storage service), which provides a simple api to hold or query messages on disk.

⊙ HA highly available service that provides data synchronization between master and slave broker.

⊙ index (indexing Service), which provides fast message query for message indexing.

Differences between RocketMQ and other message queuing services

As shown in the figure below, the differences among ActiveMQ, Kafka and RocketMQ are listed, as detailed in the official documentation.

Server planning

Prepare two servers as master and slave to each other in the following table:

RocketMQ environment preparation

1. Set the host hosts

Echo "192.168.2.177 rocketmq-nameserver1" > > / etc/hostsecho "192.168.2.177 rocketmq-master1" > / etc/hostsecho "192.168.2.178 rocketmq-nameserver2" > > / etc/hostsecho "192.168.2.178 rocketmq-master2" > > / etc/hosts

2. Prepare the JAVA environment

Download the JAVA installation package: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Extract the downloaded jdk package to the / usr/local directory and configure the environment variables

[root@rocketmq-nameserver1 ~] # vim .bash _ profileexport JAVA_HOME=/usr/local/jdk1.8.0_201export PATH=$PATH:$JAVA_HOME/bin

3. Prepare the Maven environment

Download the Maven installation package: http://maven.apache.org/download.cgi

Download the bin package, extract it directly to the / use/local directory, and then modify the environment variables

[root@rocketmq-nameserver1 ~] # vim .bash _ profileexport JAVA_HOME=/usr/local/jdk1.8.0_201export M2_HOME=/usr/local/maven3export PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin

RocketMQ deployment

Download the RocketMQ installation package: http://rocketmq.apache.org/dowloading/releases/

Extract the downloaded RocketMQ installation package and compile it

[root@rocketmq-nameserver1 rocketmq-all-4.4.0] # mvn-Prelease-all-DskipTests clean install-U

This process is limited by the network condition and may be very slow. Please wait patiently.

After the compilation is complete, go to the distribution/target/ directory and copy the entire apache-rocketmq directory under that directory to the / usr/local directory

[root@rocketmq-nameserver1 rocketmq-all-4.4.0] # cd distribution/target/ [root@rocketmq-nameserver1 target] # cp-raf apache-rocketmq / usr/local/rocketmq

Enter the / usr/local/rocketmq/conf directory, modify the log configuration file, and change the relevant log path in the configuration file to the path set by yourself.

[root@rocketmq-nameserver1 conf] # vim logback_ broker.xml [root @ rocketmq-nameserver1 conf] # vim logback_ namesrv.xml [root @ rocketmq-nameserver1 conf] # vim logback_tools.xml

Then go to the 2m-2s-async directory (Note: there are three directories here. 2m-2s-async is a multi-master, multi-asynchronous replication preconfiguration file directory, 2m-2s-sync is a multi-master, multi-slave synchronous double write preconfiguration file directory, and 2m-noslave is a double master preconfiguration file directory), and modify the broker configuration file.

[root@rocketmq-nameserver1 2m-2s-async] # vim broker-a.propertiesbrokerClusterName=rocketmq-clusterbrokerName=broker-abrokerId=0brokerIP1=192.168.2.177brokerIP2=192.168.2.177listenPort=10911storePathRootDir=/wdata/rocketmq/storestorePathCommitLog=/wdata/rocketmq/store/commitlognamesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876deleteWhen=04fileReservedTime=48brokerRole=ASYNC_MASTERflushDiskType=ASYNC_FLUSH

Save exit

[root@rocketmq-nameserver1 2m-2s-async] # vim broker-a-s.propertiesbrokerClusterName=rocketmq-clusterbrokerName=broker-abrokerId=1brokerIP1=192.168.2.178brokerIP2=192.168.2.178listenPort=10920storePathRootDir=/wdata/rocketmq/store-a-sstorePathCommitLog=/wdata/rocketmq/store-a-s/commitlognamesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876deleteWhen=04fileReservedTime=48brokerRole=SLAVEflushDiskType=ASYNC_FLUSH

Save exit

[root@rocketmq-nameserver1 2m-2s-async] # vim broker-b.propertiesbrokerClusterName=rocketmq-clusterbrokerName=broker-bbrokerId=0brokerIP1=192.168.2.178brokerIP2=192.168.2.178listenPort=10911storePathRootDir=/wdata/rocketmq/storestorePathCommitLog=/wdata/rocketmq/store/commitlognamesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876deleteWhen=04fileReservedTime=48brokerRole=ASYNC_MASTERflushDiskType=ASYNC_FLUSH

Save exit

[root@rocketmq-nameserver1 2m-2s-async] # vim broker-b-s.propertiesbrokerClusterName=rocketmq-clusterbrokerName=broker-bbrokerId=1brokerIP1=192.168.2.177brokerIP2=192.168.2.177listenPort=10920storePathRootDir=/wdata/rocketmq/store-b-sstorePathCommitLog=/wdata/rocketmq/store-b-s/commitlognamesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876deleteWhen=04fileReservedTime=48brokerRole=SLAVEflushDiskType=ASYNC_FLUSH

Save exit

Note: the brokerId format here is mainly 0 from 1.

Enter / usr/local/rocketmq/bin to modify the memory occupied by the JAVA virtual machine (here it mainly depends on the server configuration. If the server configuration is poor, it is recommended to modify it)

[root@rocketmq-nameserver1 bin] # vim runbroker.sh

[root@rocketmq-nameserver1 bin] # vim runserver.sh

Change it to what is shown in the red box in the image above

Edit the configuration file for the namesrv service (this step can be ignored, using the system default configuration)

[root@rocketmq-nameserver1 conf] # vim / wdata/rocketmq/namesrv/namesrv.confrocketmqHome=/usr/local/rocketmqkvConfigPath=/wdata/rocketmq/namesrv/kvConfig.jsonconfigStorePath=/wdata/rocketmq/namesrv/namesrv.propertiesproductEnvName=centerclusterTest=falseorderMessageEnable=falselistenPort=9876serverWorkerThreads=8serverCallbackExecutorThreads=0serverSelectorThreads=3serverOnewaySemaphoreValue=256serverAsyncSemaphoreValue=64serverChannelMaxIdleTimeSeconds=120serverSocketSndBufSize=65535serverSocketRcvBufSize=65535serverPooledByteBufAllocatorEnable=trueuseEpollNativeSelector=false

Mainly modify the relevant directories and port numbers, and don't worry about others.

Description: the meaning of the fields involved in the above configuration. For more information, please see the official documentation. There are too many explanations here.

Copy a file

Copy the entire / usr/local/rocketmq directory on 192.168.2.177 to 192.168.2.178.

[root@rocketmq-nameserver1 local] # scp-r rocketmq root@192.168.2.178:/usr/local/ [root@rocketmq-nameserver1 local] # scp-r / wdata/rocketmq root@192.168.2.178:/wdata/

For convenience, if you are not at ease, you can do the same thing on another server.

Start RocketMQ

Operate on 192.168.2.177

[root@rocketmq-nameserver1 ~] # nohup / usr/local/rocketmq/bin/mqnamesrv-c / wdata/rocketmq/namesrv/namesrv.conf 2 > & 1 & [root@rocketmq-nameserver1 ~] # nohup / usr/local/rocketmq/bin/mqbroker-c / usr/local/rocketmq/conf/2m-2s-async/broker-a.properties > / wdata/rocketmq/logs/rocketmqlogs/broker-a.log 2 > & 1 & [root@rocketmq-nameserver1 ~] # nohup / usr/local/rocketmq/bin/mqbroker-c / Usr/local/rocketmq/conf/2m-2s-async/broker-b-s.properties > / wdata/rocketmq/logs/rocketmqlogs/broker-b-s.log 2 > & 1 &

Operate on 192.168.2.178

[root@rocketmq-nameserver2 ~] # nohup / usr/local/rocketmq/bin/mqnamesrv-c / wdata/rocketmq/namesrv/namesrv.conf 2 > & 1 & [root@rocketmq-nameserver2 ~] # nohup / usr/local/rocketmq/bin/mqbroker-c / usr/local/rocketmq/conf/2m-2s-async/broker-b.properties > / wdata/rocketmq/logs/rocketmqlogs/broker-b.log 2 > & 1 & [root@rocketmq-nameserver2 ~] # nohup / usr/local/rocketmq/bin/mqbroker-c / Usr/local/rocketmq/conf/2m-2s-async/broker-a-s.properties > / wdata/rocketmq/logs/rocketmqlogs/broker-a-s.log 2 > & 1 &

At this point, the deployment of the RocketMQ cluster is complete, and to see the status of the cluster, we can use the graphical open source tool rocketmq-console on github.

Download rocketmq-console: https://github.com/apache/rocketmq-externals

Extract and go to the rocketmq-console directory to compile

[root@rocketmq-nameserver1 opt] # cd rocketmq-externals-master/rocketmq-console/

Before compiling, you need to modify the pom.xml file in this directory

[root@rocketmq-nameserver1 rocketmq-console] # vim pom.xml

Change the 4.4.0-SNAPSHOT in the red box in the above figure to 4.4.0. If you don't modify it here, you may get an error during compilation.

Compile

[root@rocketmq-nameserver1 rocketmq-console] # mvn clean package-Dmaven.test.skip=true

This process is relatively long and needs to wait patiently.

After the compilation is complete, go to the target/ directory

[root@rocketmq-nameserver1 rocketmq-console] # cd target/

Run the rocketmq-console-ng-1.0.0.jar package

[root@rocketmq-nameserver1 target] # java-jar rocketmq-console-ng-1.0.0.jar

As shown in the figure above, the appearance of Tomcat started on port (s): 8080 (http) indicates that the service has been started successfully.

Then we type 192.168.1.177pur8080 in the browser to see

As shown in the figure, two masters and two slaves appear, indicating that our cluster configuration is successful

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.