Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to delete the expired key in redis. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Recently we found an interesting problem in the Redis cluster. After spending a lot of time debugging and testing, we can reduce the key memory usage in some clusters by 25% by changing the Redis expiration.

Multiple caching services are running within Twitter. One of them is implemented by Redis. Our Redis cluster stores some Twitter important use case data, such as presentation and participation data, ad spending counts, and direct messages.

Problem background

As early as early 2016, Twitter's Cache team made a large number of updates to the architecture of the Redis cluster. Some changes have taken place in Redis, including updates from Redis version 2.4 to version 3.2.

After this update, there are several problems, such as users starting to see that memory usage is not consistent with their expected or intended memory usage, increased latency, and key cleanup issues. Key cleanup is a big problem, which can cause data that should have been persisted to be deleted or requests sent to the original data storage.

Preliminary investigation

The affected team and the cache team began to conduct a preliminary investigation. We found that the increase in delay is related to the key cleanup that is now taking place. When Redis receives a write request but has no memory to save the write, it stops what is being done, clears the key, and then saves the new key.

However, we still need to find out the reasons for the increase in memory usage for these new purges.

We suspect that memory is full of expired but not yet deleted key. It is suggested to use scanning, which reads all key and causes expired key to be deleted.

In Redis, key has two ways to expire, active expiration and passive expiration. The scan will trigger the passive expiration of the key, and when reading the key, the TTL will be checked, and if the TTL has expired, the TTL will be deleted and nothing will be returned.

The active expiration of key in version 3.2 is described in the Redis documentation. The active expiration of key begins with a function called activeExpireCycle. It runs on an internal timer called cron at a frequency of several times per second.

The activeExpireCycle function traverses each key space, checks for random kry with TTL sets, and repeats the process until the time limit is met if the percentage threshold for expired kry is met.

This method of scanning all key is effective, and when the scan is complete, memory usage is reduced. It seems that Redis no longer effectively expires key.

However, the solution at the time was to increase the size of the cluster and more hardware, so that key would be more distributed and more memory would be available. This is disappointing because the previously mentioned Redis upgrade project reduces the size and cost of running these clusters by improving the efficiency of the clusters.

This is the end of the plan to delete the expired key in redis. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.