Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to use scan commands in Redis". In daily operation, I believe many people have doubts about how to use scan commands in Redis. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to use scan commands in Redis"! Next, please follow the editor to study!

So to record the process of stepping on the pit, the background is as follows:

The company needs to remove some useless key that does not set the expiration time because the memory of the redis server is tight. There are probably more than 500w key. Although the number of key sounds scary. But I've been playing redis for years, isn't this kind of thing easy to catch?

I thought about it at that time, and the specific plan was to filter out 500w key through lua script. Then delete the action. The lua script is executed on redis server and is fast, and it only needs to establish a connection with redis server to execute a batch. Filter out the key and delete it one week at a time. Then you can delete all of them through the shell script 500 times. Similar batch updates have been done through lua scripts in the past, and 3w at a time is also in seconds. Basically will not cause redis blocking. In this way, the 500W key can be done in 10 minutes.

Then I started writing lua scripts directly. The first is screening.

People who have used redis must know that redis works in a single thread and cannot be filtered with the keys command, because the keys command will conduct a comprehensive search at one time, which will block redis, thus affecting the command execution of normal business.

The key of 500W data can only be carried out by incremental iteration. Redis provides the scan command, which is used for incremental iterations. This command can return a small number of elements at a time, so this command is ideal for dealing with iterations of large datasets and can be used in production environments.

The scan command returns an array, with the first item being the position of the cursor and the second item being a list of key. If the cursor reaches the end, the first item returns 0.

So my first version of the lua script is as follows:

Local c = 0local resp = redis.call ('SCAN',c,'MATCH','authToken*','COUNT',10000) c = tonumber (resp [1]) local dataList = resp [2] for iTunes 1 redis.call dolocal d = dataList [I] local ttl = redis.call (' TTL',d) if ttl =-1 thenredis.call ('DEL',d) endendif cantilever 0 then return' all finished'else return 'end'end

In the local test redis environment, mock 20w of test data by executing the following command:

Eval "for I = 1, 200000 do redis.call ('SET','authToken_'.. I) end" 0

Then execute the script load command to upload the lua script to get the SHA value, and then execute evalsha to execute the resulting SHA value to run. The specific process is as follows:

It is found that although the cursor does not reach the end, the list of key is empty.

This result confused me for a while. I checked the lua script carefully and there was no problem. Is there a bug in redis's scan command? Is there something wrong with my understanding?

I'll go back to redis's command document for an explanation of the count option:

It suddenly dawned on me after reading it. It turns out that the number followed by the count option does not mean the number of elements returned each time, but the number of dictionary slots each time the scan command traverses

Every time I execute scan, I start traversing from the position of cursor 0, and not every dictionary slot stores the data I need to filter, which leads to my last phenomenon: although my count is followed by 10000, the actual redis traverses 10000 dictionary slots from the beginning to down, and finds that there are no data slots to store the data I need. So my final number of dbsize stays at 124204 forever.

So when using the scan command, if you need iterative traversal, you need to use the cursor returned by the last call as the cursor parameter of the call each time, so as to continue the previous iterative process.

At this point, the doubts in my mind were solved, and a version of lua was changed:

Local c = tonumber (ARGV [1]) local resp = redis.call ('SCAN',c,'MATCH','authToken*','COUNT',10000) c = tonumber (resp [1]) local dataList = resp [2] for item1 dolocal d = dataList dolocal d = dataList [I] local ttl = redis.call (' TTL',d) if ttl =-1 thenredis.call ('DEL',d) endendreturn c

Execute after uploading locally:

As you can see, the scan command does not fully guarantee that the number of filters per filter is exactly the same as the given count, but the whole iteration continues well. In the end, the cursor returns 0, which is at the end. At this point, the test data of 20w has been deleted.

This section of lua can be run directly in production as long as it is looped on shell. It is estimated that 500w of data can be deleted in about 12 minutes.

At this point, the study on "how to use the scan command in Redis" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.