Shulou(Shulou.com)06/02 Report--

One: the description of message middleware

1. Introduction

Message middleware can also be called message queue, which refers to the platform-independent data exchange with efficient and reliable message delivery mechanism, and the integration of distributed systems based on data communication. By providing messaging and message queuing models, the communication of processes can be extended in a distributed environment.

At present, the mainstream message middleware are RabbitMQ, Kafka, ActiveMQ, RocketMQ and so on.

2. The main functions of message middleware

Redundancy (storage)

Expansibility

Recoverability

Sequence guarantee

Buffer

Asynchronous communication

Peak clipping: a common scenario in message queues, which is widely used in second kill or grab activities. Generally, because the traffic is too large, the application system configuration can not carry this instantaneous traffic, resulting in the system to hang up directly, that is, the legendary phenomenon of "downtime". To solve this problem, we will reject that huge amount of traffic at the top of the system, that is, transfer it to MQ instead of flooding directly into our interface.

two。 Brief introduction of RabbiMQ, one of the message queues

1. RabbiMQ is developed by Erang, and clusters are easy to use, because Erlang is a distributed language, but it does not maintain load balancing, supports high concurrency and supports scalability. Support AJAX, persistence, used to store and forward messages in distributed systems, and perform well in terms of ease of use, scalability, high availability and so on.

2. Characteristics of RabbiMQ

Reliability.

Expansibility

High availability

Multiple protocols

Multilingual client

Management interface

Plug-in mechanism

3. What is message queuing

The full name of MQ is Message Queue, message queue. Is an application-to-application communication method. Applications communicate by reading and writing messages in and out of the queue (for the application's data) without the need for dedicated connections to link them.

Messaging refers to the communication between programs by sending data in a message, rather than by calling each other directly.

4.RabbiMQ mode

Note: RabbitMQ modes are generally divided into the following three categories:

(1) single server mode (one host deploys rabbitmq)

(2) normal mode (default cluster mode).

(3) Image mode (the required queue is made into an image queue and exists on multiple nodes, which belongs to the HA license case of RabbiMQ, which is "more suitable" in situations where there is a high demand for business reliability. To implement the mirror mode, you need to build a normal cluster mode, and then configure the mirror mode on the basis of this mode to achieve "yes". You can set a policy in the mirror mode to synchronize the data in the main rabbitmq to other nodes in the cluster.

+ RabbitMQ cluster base (normal mode) +

One: basic deployment, operations that need to be implemented by all hosts in the cluster

1. If there are three servers as three nodes, and all of them are connected, turn off the firewall, selinux, and edit the domain name resolution / etc/hosts (ip should correspond well with the host name, otherwise it will easily affect the experiment)

[root@rabbitmq-1 ~] # vim / etc/hosts

192.168.50.138 rabbitmq-1

192.168.50.139 rabbitmq-2

192.168.50.140 rabbitmq-3

2, install rabbitmq software in three nodes configuration

Installation dependencies:

[root@rabbitmq-1 ~] # yum install-y gcc-c++ unixODBC unixODBC-devel openssl-devel ncurses-devel

[root@rabbitmq-1 ~] # wget-- content-disposition https://packagecloud.io/rabbitmq/erlang/packages/el/7/erlang-20.3-1.el7.centos.x86_64.rpm/download.rpm

[root@rabbitmq-1] # rpm-ivh erlang-20.3-1.el7.centos.x86_64.rpm

Test

[root@rabbitmq-1 ~] # erl

Install rabbitmq:

[root@rabbitmq-1 ~] # wget https://dl.bintray.com/rabbitmq/all/rabbitmq-server/3.7.5/rabbitmq-server-3.7.5-1.el7.noarch.rpm

[root@rabbitmq-1 ~] # yum install rabbitmq-server-3.7.5-1.el7.noarch.rpm

3. Start

Startup mode 1:

[root@rabbitmq-1 ~] # systemctl daemon-reload

[root@rabbitmq-1 ~] # systemctl start rabbitmq-server

[root@rabbitmq-1 ~] # systemctl enable rabbitmq-server

[root@rabbitmq-1 ~] # systemctl status rabbitmq-server

Startup mode 2:

[root@rabbitmq-1 ~] # / sbin/service rabbitmq-server status-View status

[root@rabbitmq-1] # / sbin/service rabbitmq-server start-start

4. Open the web access interface of rabbitmq:

[root@rabbitmq-1 ~] # rabbitmq-plugins enable rabbitmq_management

5. Create a user:

Note: to create a user, you only need to add it to a host, that is, you can create it as an administrator.

(1) add users and passwords who are in

[root@rabbitmq-1 ~] # rabbitmqctl add_user soho 123

Creating user "soho"...

... done.

(2) this is for the administrator

[root@rabbitmq-1 ~] # rabbitmqctl set_user_tags soho administrator

Setting tags for user "soho" to [administrator]...

... done.

(3) View users

[root@rabbitmq-1 ~] # rabbitmqctl list_users

Listing users...

Guest [administrator]

Soho [administrator]

... done.

(4) permissions

[root@rabbitmq-1 ~] # rabbitmqctl set_permissions-p "/" soho "."

Setting permissions for user "soho" in vhost "/".

... done.

Pay attention to'. 'when setting permissions here There needs to be a space between three'. * 'which represent conf permission, read permission and write permission, for example, when not given

Soho does not have the permission to query the queue before setting these three permissions, and it is not visible in the ui interface.

6. All machines operate: enable remote login for users:

[root@rabbitmq-1 ~] # cd / etc/rabbitmq/

[root@rabbitmq-1 rabbitmq] # cp / usr/share/doc/rabbitmq-server-3.7.5/rabbitmq.config.example / etc/rabbitmq/rabbitmq.config

[root@rabbitmq-1 rabbitmq] # ls

Enabled_plugins rabbitmq.config

[root@rabbitmq-1 rabbitmq] # vim rabbitmq.config

The modifications are as follows:

7. All three machines operate the restart service:

[root@rabbitmq-1 ~] # systemctl restart rabbitmq-server

Check that all machines operate on the port: enable remote login for users:

[root@rabbitmq-1 ~] # cd / etc/rabbitmq/

[root@rabbitmq-1 rabbitmq] # cp / usr/share/doc/rabbitmq-server-3.7.5/rabbitmq.config.example / etc/rabbitmq/rabbitmq.config

[root@rabbitmq-1 rabbitmq] # ls

Enabled_plugins rabbitmq.config

[root@rabbitmq-1 rabbitmq] # vim rabbitmq.config

The modifications are as follows:

8. All three machines operate the restart service:

[root@rabbitmq-1 ~] # systemctl restart rabbitmq-server

View Port

4369-erlang Discovery Port

5672-Program connection Port

15672-Management Interface ui Port

25672-server internal communication port

9. If it is a CVM, remember to add a security group port for release.

Web visit: 192.168.50.138purl 15672

It is important to note here:

Rabbitmq default administrator user: guest password: guest

The newly added user is: soho password: 123

+ RabbitMQ cluster positive sign (normal mode) +

Second, formally deploy the cluster.

1. First, create a data storage directory and a log storage directory.

[root@rabbitmq-1] # mkdir-p / data/rabbitmq/data

[root@rabbitmq-1] # mkdir-p / data/rabbitmq/logs

[root@rabbitmq-1 ~] # chmod 777-R / data/rabbitmq

[root@rabbitmq-1] # chown rabbitmq.rabbitmq / data/-R

2, create a profile:

[root@rabbitmq-1 ~] # vim / etc/rabbitmq/rabbitmq-env.conf (add the following)

RABBITMQ_MNESIA_BASE=/data/rabbitmq/data

RABBITMQ_LOG_BASE=/data/rabbitmq/logs

[root@rabbitmq-1 ~] # cat / etc/rabbitmq/rabbitmq-env.conf

3. Restart the service

[root@rabbitmq-1 ~] # systemctl restart rabbitmq-server

4. Copy erlang.cookie

Reason: each node in the cluster is implemented through a single cookie, so the cookie of each node must be ensured, otherwise the nodes will communicate with each other. (location of cookie: the rabbitmq installed and deployed in decompressed mode is stored in / home/.erlang.cookie. / var/lib/rabbitmq installed in rpm and other installation packages)

[root@rabbitmq-1 ~] # cat / var/lib/rabbitmq/.erlang.cookie

HOUCUGJDZYTFZDSWXTHJ

The "scp" type copies the value of the .erlang.cookie of the rabbitmq-1 node to the other two nodes.

[root@rabbitmq-1 ~] # scp / var/lib/rabbitmq/.erlang.cookie root@192.168.50.139:/var/lib/rabbitmq/

[root@rabbitmq-1 ~] # scp / var/lib/rabbitmq/.erlang.cookie root@192.168.50.140:/var/lib/rabbitmq/

3. Use mq-2 and mq-3 as memory nodes and .mq-1 in the mq-1 node cluster as disk nodes. In the RabbitMQ cluster, there must be "few" disk nodes, otherwise the queue metadata will be written to the cluster. When the disk node goes down, the cluster will write new queue metadata information.

Execute the following commands in mq-2 and mq-3:

[root@rabbitmq-2 ~] # rabbitmqctl stop_app # stop node, remember not to stop service

[root@rabbitmq-2 ~] # rabbitmqctl reset # if there is data that needs to be reset, not if there is no data

[root@rabbitmq-2 ~] # rabbitmqctl join_cluster-- ram rabbit@rabbitmq-1 # is added to the disk node

Clustering node 'rabbit@rabbitmq-2' with' rabbit@rabbitmq-1'...

[root@rabbitmq-2 ~] # rabbitmqctl start_app # launch node

Starting node 'rabbit@rabbitmq-2'... 3.1.0 the solution to the node problem here:

(1) if an error occurs when executing the rabbitmqctl stop_app command: it needs to be executed.

# chmod 400.erlang.cookie packages are 400 permissions

# chown rabbitmq.rabbitmq .erlang.cookie

(2) as a result of changing the hostname file, every time rabbitmqctl stop or rabbitmqctl cluster_status, etc., as long as it is a command of rabbitmq, an error will be reported. The prompt is about as follows.

Cluster status of node rabbit@web2...

Error: unable to connect to node rabbit@web2: nodedown

.

At this point, first ps aux | grep mq, then kill-9 all mq processes, and then rabbitmq-server-detached can be solved. (that is, kill first, then restart)

(3) after restarting with rabbitmqctl stop,rabbitmq-server-detached, the previously added user admin and virtual host coresystem are all lost and need to be re-added.

(4) check domain name resolution when rabbitmqctl join_cluster-ram rabbit@rabbitmq-1 reports an error

(5) other questions about starting rabbitmq with the rabbitmq-server-detached command, the following prompt appears: Warning: PID file not written;-detached was passed, use at this time

The rabbitmqctl status prompt service has been started, so you can see that this issue does not need to be resolved.

3.1.1 Node add:

(1) by default, rabbitmq is the disk node after startup. Under this cluster command, mq-2 and mq-3 are memory nodes.

Mq-1 is a disk node.

(2) if you want to make mq-2 and mq-3 both disk nodes, you can remove the-ram parameter.

(3) if you want to change the node type, you can use the rabbitmqctl change_cluster_node_type disc command (ram), provided that you must stop the rabbit response.

Note:

# if necessary, use disk nodes to join the cluster

[root@rabbitmq-2 ~] # rabbitmqctl join_cluster rabbit@rabbitmq-1

[root@rabbitmq-3 ~] # rabbitmqctl join_cluster rabbit@rabbitmq-1

4. Check the cluster status and execute rabbitmqctl cluster_status on any node to see if the cluster configuration is successful.

[root@rabbitmq-1 ~] # rabbitmqctl cluster_status

Each machine shows three nodes, indicating that it has been added successfully!

5. Log in to the rabbitmq web Management console and create a new queue

Open a browser and enter http://192.168.50.138:15672,

Enter the default Username:guest

Enter the default Password:guest

Create a queue according to the boundary prompt

+ from the above RABBITMQ normal mode + mirror mode +

Third, the reasons for adding mirror mode:

The default cluster mode of RabbitMQ has been completed above, but the high availability of the queue is not guaranteed and the contents of the queue will not be copied. If the queue node downtime directly causes the queue cannot be applied, you can only wait for restart. Therefore, if you want to apply the queue node downtime or failure normally, you must copy the queue content to each node in the cluster, and you must create a mirror queue. The mirror queue is based on the normal cluster mode.

1. Create an image cluster: three machines operate in the same way

Rabbitmq set_policy: setting policy

[root@rabbitmq-1 ~] # rabbitmqctl set_policy ha-all "^"'{"ha-mode": "all"}'

Setting policy "ha-all" for pattern "^" to "{" ha-mode ":" all "}" with priority "0" for vhost "/".

2. Check again that the queue has been synchronized to the other two nodes:

3. The image queue is set successfully. The deployment is complete, and all queues are set as mirror queues, that is, the queues are copied to each node, and the state of each node is consistent. ``

+ rabbitmq default cluster mode + mirror mode + HA+ load balancing +

The RabbitMQ high availability cluster is built, and the last step is to build the equalizer.

Simply install and configure the load balancer HA

Note: if you use Aliyun, you can use Aliyun's private network slb to achieve load balancing without having to build your own HA.

Install and configure the load balancer HA

1. Install HAProxy in 192.168.50.138

Yum-y install haproxy

2. Modify / etc/haproxy/haproxy.cfg

[root@rabbitmq-1 ~] # cp / etc/haproxy/haproxy.cfg / etc/haproxy/haproxy.cfg.bak

[root@rabbitmq-1 ~] # vim / etc/haproxy/haproxy.cfg

Global

Log 127.0.0.1 local2

Chroot / var/lib/haproxypidfile / var/run/haproxy.pidmaxconn 4000user haproxygroup haproxynbproc 4daemon# turn on stats unix socketstats socket / var/lib/haproxy/stats

#-

Defaults

Mode http

Log global

Retries 3

Timeout connect 10s

Timeout client 1m

Timeout server 1m

Timeout check 10s

Maxconn 2048

#-

# # Monitoring and viewing local status #

Listen admin_stats

Bind: 80

Mode http

Option httplog

Option httpclose

Log 127.0.0.1 local0 err

Stats uri / haproxy

Stats auth xingdian:123 (user + password)

Stats refresh 30s

# #

# reverse Monitoring

Frontend server

Bind: 5670

Log global

Mode tcp

# option forwardfor

Default_backend rabbitmq

Maxconn 3

Backend rabbitmq

Mode tcp

Log global

Balance roundrobin

Server rabbitmq1 192.168.50.138:5672 check inter 2000s rise 2 fall 3

Server rabbitmq2 192.168.50.139:5672 check inter 2000s rise 2 fall 3

Server rabbitmq3 192.168.50.140:5672 check inter 2000s rise 2 fall 3

[root@rabbitmq-1 ~] # systemctl restart haproxy

The browser enters http://192.168.50.138/haproxy to view the status of rabbitmq.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.