Shulou(Shulou.com)06/01 Report--

The editor in this issue will bring you an introduction to the message queue in redis, analyze and describe it from a professional point of view. I hope you can get something after reading this article.

First, know the message queue

1.1 message queuing concept

A message is a unit of data that is transmitted between two computers. Messages can be very simple, such as containing only text strings, or they can be more complex and may contain embedded objects.

The message is sent to the queue. Message queuing is a container that saves messages during transmission. The message queue manager acts as a middleman when relaying messages from its source to its destination. The main purpose of the queue is to provide routing and ensure the delivery of the message; if the recipient is not available when the message is sent, message queuing retains the message until it can be successfully delivered.

1.2 Core structure

A business system joins the queue, inserts messages into the message queue one by one, and returns the successful result directly after the insertion is successful. later, there will be a message processing system. this system will take out and process the records in the message system one by one, and complete a process of leaving the team.

1.3 Application scenario

Data redundancy: for example, the order system requires strict data conversion and recording. Message queues can store these data persistently in the queue, and then there is an order, which can be acquired by subsequent processors. After subsequent processing, delete the record to ensure that each record can be processed.

System decoupling: after using the message system, the in-queue system and out-of-team system are separated, that is, as long as one day crashes, it will not affect the normal operation of another system.

Traffic peaking: such as second kill and panic buying, we can use message queues with cache, which can effectively withstand instant traffic and prevent the server from crashing.

Asynchronous communication: the message itself can be returned directly after joining the queue.

Extensibility: order queues, for example, can not only process orders, but also be used by other businesses.

Sort guarantee: some scenarios need to be processed in the order of the product, such as single in and out to ensure that the data is processed in a certain order, it is possible to use message queues.

The above are common usage scenarios of message queuing. Of course, message queuing is just a middleware that can be used with other products.

1.4 advantages and disadvantages of common queue implementation

Queue medium

1. Database, such as mysql (high reliability, easy to implement, slow speed)

2. Cache, such as redis (high speed, low efficiency when a single message packet is too large)

3. Messaging systems, such as rabbitMq (highly professional, reliable, high learning cost)

Message processing trigger mechanism

1. Read in an endless loop: easy to implement, unable to recover in time in case of failure; (more suitable for second kill, more centralized, centralized operation and maintenance)

2. Scheduled tasks: the pressure is evenly divided, and there is a processing upper limit; at present, the popular processing trigger mechanism. (the only drawback is that intervals and data need to be paid attention to, so don't wait for the last task to be completed and the next task to start.)

3. Daemon: similar to php-fpm and php-cg, basic shell is required.

1.5 process communication

Message queuing (also known as message queuing) can overcome some of the shortcomings of earlier unix communication mechanisms. As one of the early unix communication mechanisms, the amount of information that the signal can transmit is limited. Later, although POSIX 1003.1b expanded the real-time performance of the signal, the amount of information transmitted by the signal has been improved to a considerable extent, but the signal communication mode is more like an "instant" communication mode, which requires the process of receiving the signal to respond to the signal within a certain time range. Therefore, the signal is meaningful at most during the life cycle of the receiving process, and the information conveyed by the signal is close to the concept of duration with the process (process-persistent). Pipes and named pipes are typical continuous IPC with the process, and only sending unformatted byte streams will undoubtedly bring inconvenience to application development, in addition, its buffer size is also limited.

A message queue is a linked list of messages. Think of a message as a record with a specific format and a specific priority. Processes with write access to message queues can add new messages to message queues according to certain rules; processes with read access to message queues can read messages from message queues. Message queuing persists with the kernel.

At present, there are mainly two types of message queues: POSIX message queue and system V message queue, which is widely used at present. Considering the portability of the program, newly developed applications should use POSIX message queues as much as possible.

System V message queuing persists with the kernel, and only when the kernel restarts or explicitly deletes a message queue will the message queue be deleted. Therefore, the data structure (struct ipc_ids msg_ids) for recording message queues in the system is located in the kernel, and all message queues in the system can find access entries in the structure msg_ids. A message queue is a linked list of messages. Each message queue has a queue header, which is described by the structure struct msg_queue. The queue header contains a lot of information about the message queue, including the message queue key value, the user ID, the group ID, the number of messages in the message queue, and so on, and even records the recent ID of the message queue read and write process. Readers can access this information or set up some of it.

Second, decoupling cases: queue processing "order system" and "distribution system"

For the order process, we can design two systems, one is the "order system" and the other is the "delivery system". We should all have seen it when we shop online, when I submit an order. I can see my goods are being delivered backstage. At this time, we should participate in a "distribution system".

If we design the "order system" and "delivery system" together when we do the architecture, there will be some problems, first of all, for the order system, because the pressure of the system will be greater, but the "distribution system" does not need to make some immediate response to these pressures.

Second, we do not want to cause the failure of the distribution system after the failure of the order system, which will affect the normal operation of the two systems at the same time. So we hope to decouple the two systems. After the two systems are separated, we can communicate between the two systems through an intermediate "team list".

2.1 Architectural design

1. First, the order system will receive the order from the user, and then process the order.

2. The order information is then written into the team list, which is the key to communication between the two systems.

3. A program regularly executed by the distribution system to read the team list for processing.

4. After being processed by the distribution system, the processed records will be marked.

2.2 Program flow

Third, the case of traffic peaking: the list type of Redis achieves second kill.

Redis is based on memory, its speed will be very fast, redis has a very good supplement to the database because it can be persisted, redis will periodically write the data to the hard disk, so it does not have to worry about power outage, in this respect it has more advantages than another cache memcache, and redis provides five data types (string, two-way linked list, hash, collection, ordered collection).

In general, do second kill cases, snap-up, instantly higher than you, need to queue up in the case of redis is a good choice.

3.1 list types in redis data types

Redis's list is a two-way linked list that can append data from the header or tail.

* LPUSH/LPUSHX: insert values into the (/ existing) list header

* RPUSH/RPUSHX: inserts a value at the end of the (/ existing) list

* LPOP: remove and get the first element of the list

* RPOP: remove and get the last element of the list

* LTRIM: keep the elements within the specified range

* LLEN: get the list length

* LSET: set the value of the list element through the index

* LINDEX: get the elements in the list through the index

* LRANGE: gets the elements within the specified range of the list

3.2 Architectural design

A simple structure second kill program design.

1. First of all, record which user participated in the second kill and record his time.

2. Save the user's id to the redis list and let it queue up. If it is stipulated that only the top 10 users can participate in success, if the number in the list is enough, it will not be allowed to continue to append data. So the length of the list of redis will only be 10.

3. Finally, the data in redis is slowly written to the database to reduce the pressure on the data.

3.3 Code-level design

1. When the user starts the second kill, the request of the second kill program is written into Redis (uid, time_stamp).

2. If it is specified that only 10 people can be killed in seconds, check the length of the data that has been stored in Redis. If the data is discarded directly beyond the upper limit, the kill is completed.

3. Finally, the 10 pieces of data stored in Redis are processed in an endless loop, and then the data is slowly fetched and stored in the mysql database.

In the second kill this part of the pressure on the database is particularly great, if we do not have such a design, it will cause the writing bottleneck of mysql. We pass an alignment list of Redis, and then put the second kill request into Redis, and finally write the data to the database slowly through the storage program, so that the traffic can be balanced and will not cause too much pressure on mysql.

IV. RabbitMQ

First of all, when we talked about the second kill case, we mentioned the lock mechanism to prevent other programs from processing the same record. If our system architecture is very complex, multiple programs read a queue in real time, or I have multiple sending programs to operate one or more queues at the same time. I even want these programs to be distributed on different machines. It is somewhat inadequate to use redis queues in this case. What to do at this time, we need to introduce some more professional message queuing systems, these systems can better solve the problem.

4.1Architecture and principle of RabbitMQ

Features: complete implementation of AMQP, cluster simplification, persistence, cross-platform

RabbitMQS usage

1. RabbitMQ installation (rabbitmq-server, php-amqplib)

2. The producer sends messages to the message channel

3. Consumers process messages

Work queue

Idea: sent by the producer to the messaging system, which encapsulates the task into a message queue, and then allows multiple consumers to use the same queue

This not only solves the decoupling between producers and consumers, but also realizes the sharing of consumers and tasks, and reduces the pressure on the server.

The above is the message queue in redis shared by the editor. If you have similar doubts, it will not hinder your understanding with reference to the above analysis. If you want to know more about it, please follow the industry information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.