Shulou(Shulou.com)05/31 Report--

This article mainly introduces the solution to the hot key problem in redis, which has a certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article.

Text

Hot Key problem

As mentioned above, the so-called hot key problem is that there are suddenly hundreds of thousands of requests to access a particular key on redis. Then, this will cause the traffic to be too concentrated, reaching the upper limit of the physical network card, which will lead to the downtime of this redis server.

Then the next key request will be directed against your database, causing your service to be unavailable.

How to find hot key

Method 1: estimate which hot key is based on business experience

In fact, this method is quite feasible. For example, if a product is doing a second kill, then the key of this product can be judged to be hot key. The drawback is that not all businesses can predict which key is hot key.

Method 2: collect on the client side

This way is to add a line of code for data statistics before operating redis. Well, there are many ways to count this data, and it can also be to send a notification message to an external communication system. The disadvantage is that it invades the client code.

Method 3: do a collection at the Proxy layer

Some cluster architectures are as follows. Proxy can be Twemproxy, which is a unified entry. Collection and reporting can be done at the Proxy layer, but the disadvantage is obvious that not all redis cluster architectures have proxy.

Method 4: use redis to bring your own commands

(1) monitor command, which can capture the commands received by the redis server in real time, and then write code to calculate what the hot key is. Of course, there are ready-made analysis tools for you to use, such as redis-faina. However, under the condition of high concurrency, this command has the hidden danger of memory explosion, and will also degrade the performance of redis.

(2) hotkeys parameter. Redis 4.0.3 provides the hotspot key discovery function of redis-cli. Add the-hotkeys option when executing redis-cli. However, when this parameter is executed, if there are more key, it is slower to execute.

Method 5: grab the bag and evaluate it yourself.

The Redis client uses the TCP protocol to interact with the server, and the communication protocol uses RESP. Write your own program to listen to the port, parse and analyze the data according to the rules of RESP protocol. The disadvantages are high development cost, difficult maintenance and the possibility of packet loss.

Each of the above five schemes has its own advantages and disadvantages. You can make a choice according to your business scenario. So how to solve the problem after the discovery of hot key?

How to solve

At present, there are two schemes in the industry.

(1) using second-level cache

For example, using ehcache, or a HashMap is fine. After you find the hot key, load the hot key into the JVM of the system.

For this hot key request, it is taken directly from the jvm rather than going to the redis layer.

Suppose there are 100, 000 requests for the same key at this time. If there is no local cache, the 100, 000 requests will be directly directed to the same redis.

Now suppose you have 50 machines in your application layer, OK, and you also have jvm cache. The 100, 000 requests are evenly distributed, with 2000 requests per machine, and the value is taken from the JVM and the data is returned. It avoids the situation in which 100, 000 requests go to the same redis.

(2) backup hot key

The plan is also very simple. Just don't let key go to the same redis. Let's just deposit a copy of this key on multiple redis. Next, when a hot key request comes in, we randomly select one on the redis with backup, access the value, and return the data.

Assume that the number of clusters in redis is N, and the steps are shown in the following figure

Note: it doesn't have to be 2N, you can take 3N and 4N if you want, depending on the requirements.

The pseudo code is as follows

Const M = N * 2ramp / generate random number random = GenRandom (0, M) / / construct backup new keybakHotKey = hotKey + "_" + randomdata = redis.GET (bakHotKey) if data = = NULL {data = GetFromDB () redis.SET (bakHotKey, expireTime + GenRandom (0L5))}

Industry plan

OK, in fact, after reading the above content, you may have a question.

Brother Yan, is there a way to automatically find hot key during the operation of the project, and then the program automatically handles it?

Well, that's a good question. Let's talk about how the industry does it. Actually, there are only two steps.

(1) Monitoring hot key

(2) the notification system does the processing.

Coincidentally, a few days ago, there was an article entitled "like transparent Multi-level caching solution (TMC)", which also mentioned hot key issues, which we happened to take advantage of.

(1) Monitoring hot key

In terms of monitoring hot key, the second way is to collect it on the client side.

There is a sentence in "like transparent Multi-level caching solution (TMC)" that mentions

TMC has modified the JedisPool and Jedis classes of the native jedis package to integrate the initialization logic of the TMC "hotspot discovery" + "local cache" function Hermes-SDK package in the JedisPool initialization process.

In other words, the native jar package of jedis has been rewritten and the Hermes-SDK package has been added.

Then what's the Hermes-SDK bag for?

OK is to do hotspot discovery and local caching.

From the point of view of monitoring, for each key value access request of Jedis-Client, Hermes-SDK asynchronously reports the key access event to the Hermes server cluster through its communication module, so that it can carry out "hot spot detection" according to the reported data.

Of course, this is just one of the ways, and some companies use the fifth way in terms of monitoring: grab their own bags and evaluate them.

To do this, we first use flink to build a set of streaming computing system. Then write a packet grab program to capture the data of the redis listening port, catch the data and throw it into the kafka.

Next, the streaming computing system consumes the data in kafka, carries on the data statistics, and can also achieve the purpose of monitoring hot key.

(2) the notification system does the processing.

From this point of view, what is appreciated is the above solution 1: using a secondary cache for processing.

After Youzan monitors the hot key, the Hermes server cluster will notify the Hermes-SDK in each business system through various means, telling them: "Brother, this key is a hot key, remember to do the local cache."

Hermes-SDK then caches the key locally for subsequent requests. Hermes-SDK found that this was a hot key that was taken directly from the local rather than visiting the cluster.

In addition to this form of notification. We can also do this, for example, your streaming computing system monitors the hot key and writes to a node in the zookeeper. Then your business system listens to the node and finds that the node data has changed, which means finding hot key. Finally, it is possible to write to the local cache.

Thank you for reading this article carefully. I hope the article "the solution to the hot key problem in redis" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.