Shulou(Shulou.com)06/02 Report--

1. Rabbitmq introduction

RabbitMQ is an open source service implemented by AMQP protocol, the server side is written in Erlang language, supports a variety of clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., supports AJAX. It is used to store and forward messages in distributed systems, and performs well in terms of ease of use, scalability, high availability and so on.

AMQP, namely Advanced Message Queuing Protocol, advanced message queuing protocol, is an open standard of application layer protocol and is designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, and the sender of the message does not need to know the existence of the message consumer, and vice versa.

It enables the corresponding client (client) to interact with the corresponding message middleware (broker). Message middleware receives messages from publishers (publisher) (applications that publish messages, also known as producer) and forwards them to consumers (consumers, applications that process messages). Because AMQP is a network protocol, publishers, consumers, and message middleware can be deployed to different physical machines.

2. The concept of message queue

Message is the carrier of information. In order for both the message sender and the message receiver to understand the information carried by the message (the message sender needs to know how to construct the message; the message receiver needs to know how to parse the message), they need to describe the message according to a unified format, which is called message protocol. Therefore, a valid message must have a certain format; a message without a format is meaningless.

3. Scenarios of message queuing applications

Message queuing is a common usage scenario in practical applications. Four scenarios: asynchronous processing, application decoupling, traffic sharpening and message communication

3.1 Asynchronous processing

Scenario description: if a user needs to send an email and a registered SMS message after registering, there are two traditional methods

1) Serial mode: after the registration information is successfully written into the database, send the registration email, and then send the registration SMS. When all the above three tasks are completed, return to the client.

2) parallel mode: after the registration information is successfully written into the database, the registration email is sent and the registration SMS is sent at the same time. After the above three tasks are completed, return to the client. The difference with serial is that parallel approach can improve processing time.

Assuming that each of the three business nodes uses 50 milliseconds, regardless of other overhead such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.

As described in the above case, the traditional system performance (concurrency, throughput, response time) will have bottlenecks. How to solve this problem?

The introduction of message queuing will not be a necessary business logic for asynchronous processing. The modified structure is as follows:

According to the above convention, the response time of the user is equal to the time that the registration information is written to the database, that is, 50 milliseconds. Register the mail, send a short message to write to the message queue, and return directly, so the speed of writing to the message queue is very fast and can be ignored, so the user's response time may be 50 milliseconds.

3.2 Application decoupling

Scenario description: after the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system

Disadvantages of the traditional model: if the inventory system is not accessible, the order reduction will fail, resulting in order failure, and the order system is coupled with the inventory system.

How to solve the above problems? The solution after introducing the application message queue, as shown below:

Order system: after the user places an order, the order system completes the persistence processing, writes the message to the message queue, and returns the user's order to be issued successfully.

Inventory system: subscribe to the order message and use the pull / push method to obtain the order information. The inventory system carries out inventory operation according to the order information.

If: the inventory system does not work properly when placing an order. It also does not affect the normal issuance of the order, because after the order is issued, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of order system and inventory system

3.3, flow peaking

Traffic sharpening is also a common scenario in message queues, which is widely used in second kill or group robbery activities.

Application scenario: flash sale activity, usually because the traffic is too large, the traffic will surge and the application will fail. In order to solve this problem, it is generally necessary to join the message queue at the front end of the application.

A. The number of people who can control the activity

B, it can relieve the application of high flow collapse in a short time.

After receiving the user's request, the server first writes to the message queue. If the length of the message queue exceeds the maximum, the user request is discarded directly or jumps to the error page. The second kill service does subsequent processing according to the request information in the message queue.

3.4. Log processing

Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of mass log transmission. The architecture is simplified as follows

The log collection client is responsible for log data collection, and writes regularly to the Kafka queue.

Kafka message queue, which is responsible for receiving, storing and forwarding log data

Log processing applications: subscribe to and consume log data in kafka queues

3.5. Message communication

Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. Such as implementing peer-to-peer message queues or chat rooms

Point-to-point communication:

Client An and client B use the same queue for message communication.

Chat room newsletter:

Client A, client B and client N subscribe to the same topic to publish and receive messages. Achieve a chat room-like effect

4. Common message queue products

4.1 、 redis

Is a Key-Value NoSQL database, development and maintenance is very active, although it is a Key-Value database storage system, but it supports MQ functions, so the completion can be used as a lightweight queue service. For the entry and exit operations of RabbitMQ and Redis, each performed 1 million times, and the execution time was recorded every 100000 times. The test data are divided into four different sizes: 128Bytes, 512Bytes, 1K and 10K. The experimental results show that when joining the team, when the data is relatively small, the performance of Redis is higher than that of RabbitMQ, but if the data size exceeds 10K, the performance of Redis is very good, while the performance of RabbitMQ is much lower than that of Redis.

4.2 、 mecahceq

Persistent message queuing (mcq) is a lightweight message queue with the following features:

Simple and easy to use

Fast processing speed

Multiple queu

Good performance of concurrency

Compatible with memcache's protocol. It means that as long as the extension of the former is installed, no additional plug-ins are needed.

It is very convenient to use php development framework in zend framework.

4.3 、 MSMQ

This is the only thing that is considered valuable in Microsoft's products. The key is that it is not complex, except for receiving and sending. It has some rigid limitations, such as the maximum message volume is 4MB.

4.4 、 ZeroMQ

ZeroMQ is a very lightweight message system, known as the fastest message queuing system, specially developed for high-throughput / low-latency scenarios, and it can often be found in financial applications.

Compared to RabbitMQ, ZeroMQ supports many advanced messaging scenarios and can implement advanced / complex queues that RabbitMQ is not good at, but you have to implement various blocks in the ZeroMQ framework (such as Socket or Device, etc.).

ZeroMQ has a unique non-middleware model

You do not need to install and run a messaging server or middleware because your application will play this service role. You can simply reference the ZeroMQ library, install it using NuGet (the .net platform developed by Microsoft), and then you can happily send messages between applications.

But ZeroMQ only provides non-persistent queues, that is, there is no place to see if there is a problem, that is, if the downmachine, the data will be lost.

4.5 、 Jafka/Kafka

Jafka/Kafka (the ability to distribute messages to different nodes) is a distributed MQ system developed and open source by LinkedIn in December 2010. it is now an incubation project of Apache and a high-performance cross-language distributed Publish/Subscribe message queuing system, while Jafka is hatched on Kafka, that is, an upgraded version of Kafka. Has the following characteristics:

1) Fast persistence, which allows message persistence under O (1) system overhead.

2) High throughput, which can reach the throughput rate of 10W/s on an ordinary server

3) complete distributed system, Broker, Producer and Consumer all support distribution automatically and realize complex equilibrium automatically.

4) support parallel loading of Hadoop data, unify online and offline message processing, for log data and offline analysis systems like Hadoop, but require real-time processing limitations, this is a feasible solution.

5) compared with ActiveMQ, it is a very lightweight messaging system. In addition to its very good performance, it is also a well-functioning distributed system.

4.6 、 Apache ActiveMQ

ActiveMQ resides between (RabbitMQ&ZeroMQ), similar to ZemoMQ, supports advanced messaging scenarios and data persistence.

ActiveMQ is known as the backbone of the Java world. It has a long history and is widely used. It is also cross-platform and provides a natural integrated access point for products that are not on Microsoft platform.

However, it can only be considered if it runs past the MSMQ. To configure ActiveMQ, you need to install the Java environment on the target machine.

Similar to RabbitMQ, it is easy to implement advanced scenarios at a low cost. It is known as the "Swiss Army knife" of message middleware.

4.7 、 RabbitMQ

RabbitMQ is an open source message queue written in Erlang, which supports many protocols: AMQP (Advanced message queuing Protocol), XMPP (Extensible message processing Field Protocol), SMTP (simple Mail transfer Protocol), STONP (simple text oriented message Protocol), which makes it very heavyweight and more suitable for enterprise development.

It implements the Broker architecture, which means that messages can be queued on a central node before being sent to the client. This feature makes RabbitMQ easy to use and deploy, and is suitable for many scenarios such as routing, load balancing, or message persistence, which can be done with message queuing with only a few lines of code. However, this makes it poor scalability and slow speed, because the central node increases the delay and the message is larger after encapsulation. To configure RabbitMQ, you need to install the Erlang environment on the target machine

Finally, the above similar products:

1. Each has its own client API or supports multiple programming languages

2. There are a lot of documents.

3. Have provided active support.

4. ActiveMQ, RabbitMQ, MSMQ and Redis all need to start service processes, which can be monitored and configured, but there are problems with the others.

5. Both provide relatively good reliability (consistency), scalability, load balancing and, of course, performance

5. Basic concepts of Rabbitmq

Application scenario framework

RabbitMQ Server: also known as broker server, it's not a food truck, it's a delivery service. It's RabbitMQ isn't a food truck, it's a delivery service. His role is to maintain a route from Producer to Consumer to ensure that data can be transmitted in a specified way. But this guarantee is not a 100% guarantee, but it is sufficient for ordinary applications. Of course, for commercial systems, you can do another layer of data consistency guard (monitoring whether the implementation is successful), you can completely ensure the consistency of the system.

Client A & B: also known as Producer, the sender of the data. Create messages and Publish (Send) them to a broker server (RabbitMQ). A Message has two parts: Payload (payload) and Label (label). Payload is the data transferred as its name implies. Label is the name of Exchange or a tag, which describes payload, and RabbitMQ also uses this label to decide which Consumer to send the Message to. AMQP only describes label, and RabbitMQ determines the rules for how to use this label.

Client 1, 2, 3: also known as Consumer, the receiver of the data. Consumers attach to a broker server (RabbitMQ) and subscribe to a queue. Compare queue to a mailbox with a name. When a Message arrives at a mailbox, RabbitMQ sends it to one of its subscribers, Consumer. Of course, it is possible to send the same Message to many Consumer. In this Message, only payload,label has been deleted. For Consumer, it doesn't know who sent this message. Is that the agreement itself does not support it. But of course, if the payload sent by Producer contains information about Producer, it's a different matter.

There are three more concepts that need to be clarified for the correct transfer of data from Producer to Consumer: exchanges and queues and bindings.

Exchanges: message switch, which specifies the rules by which messages are routed to which queue

Queues: message queue carrier, where each message is thrown into one or more queues

Bindings: its function is to bind exchange and queue according to routing rules

Routing Key: routing keyword according to which exchange delivers messages

Connection: it's just a TCP connection.

Both Producer and Consumer are connected to RabbitMQ Server through TCP. As we can see later, the starting point of the program is to establish this TCP connection.

Channel: virtual connection. It is established in the TCP connection above. Data flow takes place in Channel. In other words, generally speaking, the program starts to establish a TCP connection, and the second step is to establish the Channel.

Vhost: virtual host, multiple vhost can be opened in a broker, which can be used to separate permissions for different users. Each virtual host is essentially a RabbitMQ Server, with its own queue,exchagne, bings rule, and so on. This ensures that you can use RabbitMQ in many different application.

6. The choice of Channel

So why use Channel instead of directly using TCP connections?

For OS, it is costly to establish and close TCP connections. Frequent establishment and closure of TCP connections have a great impact on the performance of the system, and the number of TCP connections is also limited, which also limits the ability of the system to deal with high concurrency. However, there is no such cost in establishing a Channel in an TCP connection. For Producer or Consumer, multiple Channel can be used concurrently for Publish or Receive.

7. Message queue execution process

The client connects to the message queuing server and opens a Channel.

The client declares an Exchange and sets the relevant properties.

The client declares a Queue and sets the relevant properties.

The client uses Routing key to establish a binding relationship between Exchange and Queue.

The client delivers the message to Exchange.

After Exchange receives the message, it routes the message and delivers the message to one or more queues according to the key of the message and the Binding that has been set.

8. Message persistence

RabbitMQ supports persistence of messages, that is, data is written on disk. For the sake of data security, most users will choose persistence. Message queue persistence consists of three parts:

Exchange persistence, specify durable = > 1 when declaring

Queue persistence, specify durable = > 1 when declaring

Message persistence, specify delivery_mode = > 2 when delivering (1 is non-persistent)

If both Exchange and Queue are persistent, then the Binding between them is also persistent; while there is a persistence and a non-persistence between Exchange and Queue, binding is not allowed.

After Consumer retrieves a message from durable queue and concurrently returns the ack message, RabbitMQ marks it for subsequent garbage collection. If a persistent message is not taken away by consumer, the exchange and queue (and bingding relationship) are automatically rebuilt after RabbitMQ restart, and the message enters the corresponding queues,exchanges again through persistent log reconstruction.

9. Install rabbitmq

24 yum-y localinstall erlang-18.1-1.el6.x86_64.rpm rabbitmq-server-3.6.6-1.el6.noarch.rpm socat-1.7.3.2-2.el7.x86_64.rpm (install rabbitmq) 22 chkconfig-- add rabbitmq-server (add Boot) 25 chkconfig rabbitmq-server on (Boot on rabbitmq) 26 / etc/init.d/rabbitmq-server start (enable rabbitmq) check whether rebbitmq is running 27 ps-ef | grep rabbitmq

30 rabbitmq-plugins enable rabbitmq_management

# enable the web management plug-in of rabbitmq, which can be accessed by users through the browser

31 rabbitmqctl add_user admin 123.com

# create login user admin password 123.com

32 rabbitmqctl set_user_tags admin administrator

# add admin users to the administrators group

View port netstat-anpt | grep 15672

Browser access IP address: 15672

10. Rabbitmq cluster

(operation of 1.40-1.60)

Cluster mode

Rabbitmq clusters can be divided into two ways:

Normal mode: the default cluster mode. The entity of the message has only one mirrored mode on one node: the desired queue is made into a mirrored queue and exists on multiple nodes.

Ha-mode:all line up on all nodes

Exatly: randomly mirrored to other nodes

Nodes: mirroring to a specified node

Cluster node mode:

1. Memory node: working on memory

2. Disk node: working on disk

Exception: memory node and disk node co-exist, improving access speed and increasing persistence

Compared to the memory node, although it does not write to disk, it performs better than the disk node. In a cluster, only one disk node is needed to save state. If there are only memory nodes in the cluster, they cannot be stopped, otherwise all states, messages, and so on will be lost.

Install rabbitmq

24 yum-y localinstall erlang-18.1-1.el6.x86_64.rpm rabbitmq-server-3.6.6-1.el6.noarch.rpm socat-1.7.3.2-2.el7.x86_64.rpm (install rabbitmq) 22 chkconfig-- add abbitmq-server (add Boot) 25 chkconfig rabbitmq-server on (Boot on rabbitmq) 26 / etc/init.d/rabbitmq-server start (enable rabbitmq) check whether rebbitmq is running 27 ps-ef | grep rabbitmq

[root@localhost ~] # vim / etc/hosts (all four hosts have to operate) 192.168.1.30 rabbitmq1192.168.1.40 rabbitmq2192.168.1.50 rabbitmq3192.168.1.60 rabbitmq4## do not want to handwrite each machine can use scp [root @ rabbitmq1 ~] # scp / etc/hosts root@192.168.1.40:/etc/hosts # # prompt for yes

View cookie node information on rabbitmq1 and copy

# # Node cookie information needs to be consistent when installing cluster

Change the file cookie information and rabbitmq1 to the same on rabbitmq2,3 and 4

[root@rabbitmq2 ~] # echo MIIPQWTLIDGVGKMDWQFX > / var/lib/rabbitmq/.erlang.cookie [root@rabbitmq3 ~] # echo MIIPQWTLIDGVGKMDWQFX > / var/lib/rabbitmq/.erlang.cookie [root@rabbitmq4 ~] # echo MIIPQWTLIDGVGKMDWQFX > / var/lib/rabbitmq/.erlang.cookie

Reboot restarts the virtual machine after configuration

# # Note: 2 and 3 need to be rebooted manually. After the restart is completed, the hostname will change to rabbitmq1, 2, 3. The boot interface that cannot be started will stay in this place for 5-6 minutes to recover. This will not happen if you restart manually again.

[root@rabbitmq1 ~] # ps-ef | grep rabbit # # check whether to start after the restart is completed

Operate on rabbitmq1

45 rabbitmqctl stop_app

46 rabbitmqctl reset

47 rabbitmqctl start_app

# # Node name will be prompted and copied after the setting is completed

Add a node to rabbitmq2 and 3Jing 4 (2Jing 3 and 4 have the same operation)

[root@rabbitmq2 ~] # rabbitmqctl stop_appStopping node rabbit@rabbitmq2... [root@rabbitmq2 ~] # rabbitmqctl resetResetting node rabbit@rabbitmq2... [root@rabbitmq2 ~] # rabbitmqctl join_cluster-- ram rabbit@rabbitmq1Clustering node rabbit@rabbitmq2 with rabbit@rabbitmq1... # #-join_cluter join the cluster-- ram adds the node name rabbit@rabbitmq1 [root@rabbitmq2 ~] # rabbitmqctl start_appStarting node rabbit@rabbitmq2. [root@rabbitmq2 ~] # in the form of a memory node Rabbitmq-plugins enable rabbitmq_managementThe following plugins have been enabled: mochiweb webmachine rabbitmq_web_dispatch amqp_client rabbitmq_management_agent rabbitmq_managementApplying plugin configuration to rabbit@rabbitmq2... Started 6 plugins.## enables the rabbitmq plug-in. If you do not open it, you cannot use a browser to access the rabbit page for management.

Go back to rabbitmq1 to create administrative users and view cluster status

[root@rabbitmq1 ~] # rabbitmqctl add_user admin redhatCreating user "admin"... [root@rabbitmq1 ~] # rabbitmqctl set_user_tags admin administratorSetting tags for user "admin" to [administrator]. # # create a user admin password as redhat, and add it to the administrator group [root@rabbitmq1 ~] # rabbitmq-plugins enable rabbitmq_managementPlugin configuration unchanged.Applying plugin configuration to rabbit@rabbitmq1... Nothing to do. [root@rabbitmq1 ~] # rabbitmqctl cluster_status # # View node status # # rabbit1 working mode is disk node rabbit2, 3meme 4 is ram memory node mode # # running_nodes: running node # # cluster_name: node name # # alarms: when there is a problem, rabbit1, 2 and 3 will give an alarm

Browser access for management:

Http://192.168.1.30:15672 user name is admin password redhat

Click vxgp and find the premissions option below set to admin user access

Check again after the setup is complete

Set matching policy

Post the message:

Set the content of the publish message

You can see that there has been a notice.

Add rabbitmq node: rabbitmq4 192.168.83.4

Rabbitmq1, 2, 3 add hosts hosts all add 192.168.83.4 rabbitmq4

Rabbitmq4 installation is the same as above installation [root@localhost ~] # chkconfig rabbitmq-server on [root@localhost ~] # / etc/init.d/rabbitmq-server start [root@localhost ~] # echo MIIPQWTLIDGVGKMDWQFX > / var/lib/rabbitmq/.erlang.cookie [root@localhost ~] # reboot [root@rabbitmq4 ~] # rabbitmqctl stop_ app [root @ rabbitmq4 ~] # rabbitmqctl reset [root@rabbitmq4 ~] # rabbitmqctl join_cluster-- ram rabbit@rabbitmq1

Delete a node:

# # stop the node on rabbitmq4 first

[root@rabbitmq1 ~] # rabbitmqctl stop_app

Go back to the primary node, rabbitmq1

[root@rabbitmq1] # rabbitmqctl-n rabbit@rabbitmq1 forget_cluster_node rabbit@rabbitmq4

Removing node rabbit@rabbitmq4 from cluster...

# #-n specify node name

# = forget_cluster_node followed by the name of the node to be deleted

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.