Shulou(Shulou.com)05/31 Report--

This article introduces how python uses redis to do queue service. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

The introduction of message queue mechanism into the system is a great improvement to the system. For example, in a web system, a user needs to send an email notification to the user's mailbox after doing something. You can use synchronization to ask the user to wait for the email to be sent back to the user, but this may cause the user to wait for a long time and affect the user experience because of the uncertainty of the network.

In some scenarios, it is impossible to use synchronization to wait for completion, those operations that take a lot of time in the background. For example, an extreme example, an online compilation system task, background compilation takes 30 minutes to complete. It is impossible for the design of this scenario to give back after synchronous waiting. It must first feedback the user and then complete the asynchronous processing, and then wait for the completion of the processing and then give feedback to the user or not according to the situation.

In addition, when the processing capacity of the system is limited, the queue mechanism is used to store the tasks temporarily, and then the system takes turns to deal with the queued tasks one by one. In this way, the tasks with high concurrency can be handled stably in the case of insufficient system throughput.

Message queuing can be used as a queuing mechanism, and message queuing can be used as long as the system needs to use the queuing mechanism.

Priority approach of rabbitmq

At present, there are many mature message queuing products, such as rabbitmq. It is relatively simple to use, relatively rich in functions, and is fully adequate in general situations. But the annoying thing is that it doesn't support priorities. For example, for an e-mail task, some privileged users want it to be sent in a more timely manner, or at least have priority over ordinary users. By default, rabbitmq cannot be disposed of, and all tasks thrown to rabbitmq are FIFO first-in, first-out. But we can use some flexible techniques to support these priorities. Create multiple queues and set the appropriate routing rules for rabbitmq consumers.

For example, there is such a queue by default. We use list to simulate [task1, task2, task3]. Consumers take turns to take out task one by one according to the principle of FIFO to dispose of it. If a high priority task comes in, it can only be processed at the end [task1, task2, task3, higitask1]. But if you use two queues, a high priority queue and a normal priority queue. Normal priority [task1, task2, task3], high priority [hightask1] and then we set up consumer routing to allow consumers to randomly fetch data from any queue.

And we can define a consumer who specializes in dealing with high-priority queues and does not process data from low-priority queues when it is idle. This is similar to the VIP counter of a bank, where ordinary customers line up for numbers at the bank, and a VIP arrives. Although he did not take out a ticket in front of ordinary members from the number-taking machine, he can still take the VIP channel more quickly.

If you use rabbitmq to support priority message queues, just like the VIP members of the bank described above, take different channels. However, this approach is only a relative priority, and there is no absolute priority control. For example, I hope that a high-priority task will be given priority over other ordinary tasks in an absolute sense, so that the above scheme is not feasible. Because rabbitmq consumers only know how to "randomly" fetch the first data from a queue they care about when they are free, it has no control over which queue to find first. Or finer-grained priority control. Or there are more than 10 priorities set in your system. It is also difficult to use rabbitmq in this way.

However, all of the above requirements can be implemented if redis is used as a queue.

How to do message queuing with redis

First of all, redis is designed to be cached, but it can be used as a message queue because of its own characteristics. It has several blocking API available, and it is these blocking API that give him the ability to do message queues.

Imagine that under the idea of "the database solves all problems", you can do what you need without using message queues. We store all the tasks in the database and then retrieve the task processing through continuous polling. This practice can accomplish your task, but it is very crude. But if your database interface provides a blocking method, you can avoid polling, and your database can also be used as a message queue, but the current database does not have such an interface. In addition, other features of message queuing, such as FIFO, are also easy to implement, with only a List object fetching data from the header and stuffing the data from the tail.

Redis can do message queuing thanks to his list object blpop brpop interface and some interfaces of Pub/Sub (publish / subscribe). They are all blocking versions, so they can be used as message queues.

Implementation of redis message queue priority

Explanation of some basic knowledge of redis

Redis > blpop tasklist 0 "im task 01"

This example uses the blpop command to block the first data from the tasklist list, and the last argument is the time to wait for the timeout. If set to 0, it means waiting indefinitely. In addition, the data stored in redis can only be of string type, so it can only be passed string when the task is passed. We just need to simply serialize the responsible data into a string in json format, and then convert it to the consumer.

Here our sample language uses python, and the library that links to redis uses redis-py. If you have some programming foundation, it should be no problem to switch it to your favorite language.

1 simple FIFO queue

Import redis, time

Def handle (info):

Print info

Time.sleep (20)

Def main ():

Pool = redis.ConnectionPool (host='localhost', port=6379, db=0)

R = redis.Redis (connection_pool=pool)

While 1:

Result = r.brpop ('task', 0)

Handle (result [1])

If _ name__ = = "_ _ main__":

Main ()

In the previous example, even for the simplest consumer, we constantly fetched data from the redis queue through an infinite loop. If there is no data in the queue, there is no timeout blocking there, and if there is data, it is taken out and executed.

In general, it is a complex string, and we may need to format it and pass it back to the handler, but for simplicity our example is a normal string. In addition, the handler in the example does not do any processing, only sleep is used to simulate time-consuming operations.

We open another redis client to simulate the producer, and our own client is fine. Stuff more data into the tasklist queue.

Redis 127.0.0.1 fuckin1 6379 > LPUSH task "fuckin1"

(integer) 1

Redis 127.0.0.1 fuckin2 6379 > LPUSH task "fuckin2"

(integer) 1

Redis 127.0.0.1 fuckin3 6379 > LPUSH task "fuckin3"

(integer) 2

Redis 127.0.0.1 fuckin4 6379 > LPUSH task "fuckin4"

(integer) 3

Redis 127.0.0.1 fuckin5 6379 > LPUSH task "fuckin5"

(integer) 4

Redis 127.0.0.1 fuckin6 6379 > LPUSH task "fuckin6"

(integer) 5

Redis 127.0.0.1 fuckin7 6379 > LPUSH task "fuckin7"

(integer) 6

Redis 127.0.0.1 fuckin8 6379 > LPUSH task "fuckin8"

(integer) 7

Redis 127.0.0.1 fuckin10 6379 > LPUSH task "fuckin10"

(integer) 8

Redis 127.0.0.1 6379 > lrange task 0-1

1) "fuckin10"

2) "fuckin8"

3) "fuckin7"

4) "fuckin6"

5) "fuckin5"

6) "fuckin4"

7) "fuckin3"

You can see

[root@host-192-168-1-56 soft] # python duilie.py

('task',' fuckin1')

Fuckin1

('task',' fuckin2')-the interval between each task is 20 seconds, and 20 seconds is the simulation task execution time

Fuckin2

('task',' fuckin3')

Fuckin3

('task',' fuckin4')

Fuckin4

('task',' fuckin5')

. .

. . .

. . .

('task',' fuckin10')

Fuckin10

. . . Waiting status, waiting for a new task

two。 Queues with simple priority

Suppose a simple requirement requires only high-priority tasks to be dealt with first than low-priority tasks. Regardless of the order between other tasks, we just need to shove it to the front of the queue when we encounter a high-priority task, rather than push it to the end.

Because our queue uses redis's list, it's easy to implement. Encounter high priority using rpush encounter low priority using lpush

Redis > lpush tasklist'im task 01'redis > lpush tasklist'im task 02'redis > rpush tasklist'im high task 01'redis > rpush tasklist'im high task 01'redis > lpush tasklist'im task 03'redis > rpush tasklist'im high task 03'

As you will see later, the high priority is always executed first than the lower priority. But the disadvantage of this scheme is that the order of execution between high-priority tasks is first-in-first-out.

3. A relatively perfect queue

In example 2, we simply put the high-priority tasks at the front of the queue and the low-priority tasks at the bottom. This does not guarantee the order of high-priority tasks.

Suppose that when all tasks are high priority, then they will be executed in reverse order. This clearly violates the FIFO principle of the queue.

But with a little improvement, we can improve our queue.

As with rabbitmq, we set up two queues, one with high priority and one with low priority. The high priority tasks are placed in the high queue and the low ones in the low priority queue. Redis differs from rabbitmq in that it can ask queue consumers to read first from which queue.

Def main (): pool= redis.ConnectionPool (host='localhost', port=6379, db=0) r = redis.Redis (connection_pool=pool) while 1: result = r.brpop (['high_task_queue',' low_task_queue'], 0) handle (result [1])

The above code will block data from the two queues of 'high_task_queue',' low_task_queue', if the first one is not fetched from the second one. Therefore, the goal can be achieved only by making such improvements to the queue consumers.

Redis > lpush low_task_queue low001redis > lpush low_task_queue low002redis > lpush low_task_queue low003redis > lpush low_task_queue low004redis > lpush high_task_queue low001redis > lpush high_task_queue low002redis > lpush high_task_queue low003redis > lpush high_task_queue low004

Through the above tests, we can see that the high priority will be the first to be executed, and the high priority also ensures the principle of FIFO.

In this scheme, we can support priority queues at different stages, such as three or more levels in high, middle and low levels.

4. A situation in which there are many priority levels.

Suppose there is a requirement that the priority is not a simple high, middle or low level or a fixed level of 0-10. It's a level similar to 0-99999. Then our third option will not be appropriate. Although redis has sortable data types like sorted set, it's a pity that it doesn't have a blocking version of the interface. So we still have to use the list type to achieve our goal in other ways.

There is a simple way to set up only one queue and make sure it is sorted according to priority. Then use the binary search method to find the appropriate location of a task, and insert it into the appropriate location through the lset command. For example, the queue contains tasks that write priority [1, 3, 6, 8, 9, 14]. When a task with a priority of 7 comes, we use our own dichotomy algorithm to take data out of the queue one by one and compare it with the target data. calculate the corresponding location and insert it into the specified location.

Because the binary search is relatively fast, and the redis itself is in memory, the speed can be guaranteed in theory. But if the amount of data is really large, we can also tune it in some ways.

Looking back on our third option, combining the third option will greatly reduce the cost. For example, for queues with 100, 000 data, their priority is also a random range of 0-100, 000. We can set up 10 or 100 different queues, putting 0-10,000 priority tasks in queue 1 and 10,000-20,000 tasks in queue 2. In this way, after splitting a queue according to different levels, the data of a single queue will be much less, so that the efficiency of binary search matching will be a little higher. However, the resources occupied by the data are basically the same, and how much memory should be occupied by 100,000 data. It's just that there are more queues in the system.

So much for sharing about how python uses redis to do queuing service. I hope the above content can be of some help and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.