Shulou(Shulou.com)05/31 Report--

This article shows you how to implement the real-time subscription and push function in Redis. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Delayed delivery of solution 1:MQ. Although MQ supports delayed delivery of messages, the scale is too large 1s 5s 10s 30s 1m, which can not be used for accurate time delivery! And if the user cancels the subscription after performing the subscription, the operation of delete the MQ message sent out is a bit big, and it is difficult to land in a short time! And users can cancel and then subscribe, which involves the problem of de-duplication. So MQ's plan will not be dropped.

Scheme 2: traditional scheduled tasks. This is relatively simple. The scheduled task is to go to the subscription reminder records of load users in db to select the records that can be pushed currently. But there is a good saying that any design that is divorced from the actual business is a hooligan. Let's analyze whether the traditional scheduled tasks are suitable for our business.

To sum up, we know that the general traditional scheduled tasks have the following shortcomings:

1. Performance bottleneck. There is only one machine in the processing, unable to do so in front of a large amount of data!

2. The actual effect is poor. The frequency of scheduled tasks can not be too high, too high will cause a lot of pressure on the business database!

3. Single point of failure. If the running machine dies, the whole business will be unavailable. -it's a terrible thing!

So traditional timed tasks are not suitable for this business.

Is there nothing we can do? Actually, it's not! We just need to make a simple transformation to the traditional scheduled task! It can be turned into a scheduled task cluster that can run multiple machines at the same time, and the effectiveness can be accurate to seconds, and reject a single point of failure! It's going to take advantage of our powerful redis.

Solution 3: timed task cluster

First of all, we need to define three problems to be solved by the timed task cluster!

1. High effectiveness

2. The throughput should be large

3. The service should be stable and there should be no single point of failure.

The following is an architecture diagram of the entire timed task cluster.

The architecture is simple: we store users' subscription and push records in the sortedSet queue of the redis cluster, and take the reminder timestamp as the score value, and then start a timer in our each business server with a timer frequency of seconds, which I set to 1s. After load balancing, we obtain the user records to be pushed from a queue for push. Let's analyze the following architecture

1. Performance: excluding bandwidth and other factors, it is basically linearly related to the number of machines. The greater the number of machines, the greater the throughput, and the relative throughput decreases when the number of machines is small.

2. Effectiveness: improved to seconds, the effect is acceptable.

3. Single point of failure? It doesn't exist! Unless the redis cluster or all server is dead.

Why do you use redis here?

The first redis can be used as a high-performance storage db with much better performance than MySQL and supports persistence and good stability.

The second redis SortedSet queue naturally supports sorting with time as a condition, which perfectly satisfies the records we have selected to push.

Ok~, now that the plan is in place, how can it be implemented in one day? Yes, I designed this plan until the basic coding was completed, and the time was one day. Because time is too fast.

First we use user_id as the key, and then the number of mod queues hash to the redis SortedSet queue. Why is this so, because if users subscribe to two coupons at the same time and the push time is very close, the two pushes can be merged into one, and the hash is relatively uniform. Here is a screenshot of some of the code:

Then to determine the number of queues, generally speaking, we define as many queues as many servers we have to process. Because there are too few queues, it will cause queue competition, and too many records may not be processed in a timely manner.

However, the best practice is that the number of queues should be dynamically configurable, because the number of cluster machines online changes frequently. When we make a big push, we will add machines, and when the business volume increases, the number of machines will also increase, right? So I borrowed the diamond of Taobao to dynamically configure the number of queues.

How many records we take from the queue at a time can also be dynamically configured.

In this way, the throughput of the whole cluster can be adjusted according to the actual production situation at any time. Therefore, our scheduled task cluster still has a feature that supports dynamic adjustment.

The last key component is load balancing. This is very important! Because if this is not done well, it may lead to multiple computers competing to deal with a queue at the same time, affecting the efficiency of the whole cluster! In the case of tight time, I used a simple and practical algorithm to use a self-increasing key and then mod queue number using redis. This largely ensures that there will not be two machines competing for a queue at the same time.

Finally, let's calculate the throughput of the whole cluster.

10 (number of machines) * 2000 (number of pull at a time) = 20000. Then the message is pushed to the message center in the form of MQ, and the MQ is asynchronous, counting the other processing 0.5s.

In fact, it takes more than 10 seconds to send a 20W push.

When ok~ arrives here, our entire scheduled task cluster is almost on the ground. If you ask me if there is anything else I can improve, it is:

1. Add monitoring. How can there be no monitoring in the cluster? what if something goes wrong and tasks pile up?

2. Add the visual interface.

3. It is better to have intelligent scheduling and increase the priority of tasks. Let's run the high priority task first.

4. Resource scheduling, in case there are not enough machines, priority should be given to ensuring the execution of important tasks.

The above is how to implement the real-time subscription and push function in Redis. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.