Shulou(Shulou.com)06/01 Report--

This article will explain in detail which redis instructions should not be used in production. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Cause of the problem

Brother Black is responsible for the application is a management background application, rights management uses the Shiro framework, because there are multiple nodes, need to use distributed Session, so here use Redis to store Session information.

Voiceover: if you don't know distributed Session, you can take a look at what Brother Black wrote before and said in one breath that there are four ways to implement distributed consistent Session, the leverage of the interview.

Since Shiro does not provide Redis storage Session components directly, Xiao Hei has to use shiro-redis, an open source component of Github.

Because the Shiro framework needs to periodically verify that the Session is valid, the underlying Shiro will call SessionDAO#getActiveSessions to get all the Session information.

Shiro-redis happens to inherit the interface SessionDAO, and the underlying layer uses the keys command to find all the stored Session key of Redis.

Public Set keys (byte [] pattern) {checkAndInit (); Set keys = null; Jedis jedis = jedisPool.getResource (); try {keys = jedis.keys (pattern);} finally {jedis.close ();} return keys;}

Find the cause of the problem and the solution is relatively simple. Find the solution on github and upgrade shiro-redis to the latest version.

In this version, shiro-redis fixes this problem by using the scan command instead of keys.

Public Set keys (byte [] pattern) {Set keys = null; Jedis jedis = jedisPool.getResource (); try {keys = new HashSet (); ScanParams params = new ScanParams (); params.count (count); params.match (pattern); byte [] cursor = ScanParams.SCAN_POINTER_START_BINARY; ScanResult scanResult; do {scanResult = jedis.scan (cursor,params) Keys.addAll (scanResult.getResult ()); cursor = scanResult.getCursorAsBytes ();} while (scanResult.getStringCursor (). CompareTo (ScanParams.SCAN_POINTER_START) > 0);} finally {jedis.close ();} return keys;}

Although the problem was solved successfully, Brother Black was still a little confused.

Why does the keys instruction cause other commands to slow down?

Why is the Keys instruction query so slow?

Why is there nothing wrong with the Scan instruction?

How Redis executes commands

First of all, let's look at the first question, why does the keys instruction cause other commands to slow down?

To answer this question, let's first take a look at how the Redis client executes a command:

From the client's point of view, there are three steps to executing a command:

Send a command

Execute a command

Return the result

But this is only what the client thinks it is, but in fact, at the same time, there may be a lot of clients sending commands to Redis, and Redis we all know that it uses a single-threaded model.

In order to process the request commands of all clients at the same time, the queue is used inside the Redis to execute.

So it actually takes four steps for the client to execute a command:

Send a command

Command queue

Execute a command

Return the result

Because Redis executes commands in a single thread, tasks can only be sequentially fetched from the queue.

As long as the execution of the command in this process is too slow, other tasks in the queue have to wait, which seems to the external client that Redis is blocked and does not get a response.

Therefore, do not use Redis procedures to execute instructions that need to run for a long time, which may cause Redis blocking and affect the execution of other instructions.

KEYS principle

Next, let's answer the second question, why is the Keys instruction query so slow?

Before you answer this question, please recall the underlying storage structure of Redis.

It doesn't matter if you don't know your friends. You can take a look back at Brother Black's previous article "Ali interviewer: HashMap is familiar, right?" OK, let's talk about Redis dictionary. ".

Here, Brother Black copies the content of the previous article, and the bottom layer of Redis uses the structure of dictionary, which is similar to that of Java HashMap.

The keys command needs to return all Redis keys that match the given pattern pattern. To do this, Redis has to traverse the underlying array of the ht [0] hash table in the dictionary, which has a time complexity of O (N) (N is the number of key in Redis).

If the number of key in Redis is small, then the execution speed will still be very fast. When the number of Redis key increases gradually, rising to the level of millions, tens of millions, or even hundreds of millions, then the execution speed will be very slow.

Here is an experiment done locally by Brother Black, using the lua script to add 10W key to Redis, and then using keys to query all the keys, which will probably block for more than ten seconds.

Eval "for iTunes 1100000 do redis.call ('set',i,i+1) end" 0

Here, Brother Black uses Docker to deploy Redis, and the performance may be slightly worse.

SCAN principle

Finally, let's take a look at the third question, why is there nothing wrong with the scan instruction?

This is because the scan command uses a cool techs, a cursor-based iterator.

Each time the scan command is called, Redis returns a new cursor and a certain number of key to the user. The next time you want to continue to get the remaining key, you need to pass this cursor into the scan command to continue the previous iteration.

In a nutshell, the scan command uses paging to query redis.

The following is an example of an iterative process for the scan command:

The scan command uses cursors to cleverly split a full query into multiple times and reduce the complexity of the query.

Although the time complexity of the scan command is the same as that of keys, it is O (N), but because the scan command only needs to return a small amount of key, the execution speed is very fast.

Finally, although the scan command addresses the keys deficiency, it also introduces some other defects:

The same element may be returned multiple times, which requires our application to add the ability to deal with repeating elements.

If an element is added to redis during the iteration, or is deleted during the iteration, the element will be returned or may not be.

These defects need to be taken into account in our development.

In addition to scan, redis has several other incremental iteration commands:

Sscan: used to iterate over the database keys in the current database to solve possible blocking problems caused by smembers

The hscan command is used to iterate over the key-value pairs in the hash key to solve the blocking problem that may occur in hgetall.

Zscan: the command is used to iterate over elements in an ordered collection (including element members and element scores) and to generate zrange that may cause blocking problems.

This is the end of this article on "which redis instructions should not be used in production". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.