Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of "Redis memory is full and then go to optimize". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this "Redis memory is full and then go to optimize" article can help you solve the problem.

What if the Redis memory is full? How to optimize memory? There are 2000w data in MySQL and only 20w data in redis. How to ensure that the data in redis are hot data?

When the redis in-memory dataset size rises to a certain size, the data elimination strategy is implemented.

What are the main physical resources consumed by Redis?

Memory.

What happens when Redis runs out of memory?

If the upper limit is reached, Redis's write command returns an error message (but the read command returns normally. Or you can configure the memory elimination mechanism so that the old content will be washed out when the Redis reaches the memory upper limit.

Talking about the elimination mechanism of cached data

What are the elimination strategies for Redis caching?

Noeviction is the only strategy that does not eliminate data.

The seven strategies that will be phased out can be further divided into two categories according to the scope of the phase-out candidate dataset:

Elimination is carried out in the data with the expiration time set, including volatile-random, volatile-ttl, volatile-lru and volatile-lfu.

Phase-out is carried out within all data ranges, including allkeys-lru, allkeys-random and allkeys-lfu.

When filtering the policy rule volatile-ttl, it will delete the key-value pairs with the expiration time set according to the sequence of the expiration time. The earlier the expiration time, the earlier it will be deleted. Volatile-random deletes randomly in the key-value pair with the expiration time set. Volatile-lru uses LRU algorithm to filter keys with expiration time set to volatile-lfu using LFU algorithm to select key-value pairs policy rules allkeys-random randomly selects and deletes data from all key-value pairs; allkeys-lru uses LRU algorithm to filter all data vallkeys-lfu uses LFU algorithm to filter all data talk about LRU algorithm

Data is filtered according to the least recently used principle, where the least commonly used data is filtered out, while recently frequently used data is left in the cache.

How exactly is it screened? LRU organizes all the data into a linked list, with the head and tail of the linked list representing the MRU end and the LRU side, respectively, representing the most commonly used data and the least commonly used data, respectively.

The idea behind the LRU algorithm is very simple: it thinks that the data that has just been accessed will definitely be accessed again, so it puts it on the MRU side; the data that has not been accessed for a long time will certainly not be accessed again, so let it gradually move back to the LRU side, and when the cache is full, it will be deleted first.

Problem: in the actual implementation of the LRU algorithm, it needs to use a linked list to manage all the cached data, which will bring additional space overhead. Moreover, when data is accessed, the data needs to be moved to the MRU side on the linked list. If a large amount of data is accessed, it will bring a lot of linked list moving operations, which will be very time-consuming, which will reduce the performance of Redis cache.

Resolve:

In Redis, the LRU algorithm is simplified to reduce the impact of data elimination on cache performance. Specifically, Redis records by default the timestamp of the last access to each data (recorded by the key-value pair in the lru field in the data structure RedisObject). Then, when deciding on the data to be eliminated, Redis randomly selects N pieces of data for the first time as a candidate set. Next, Redis compares the lru fields of the N data to eliminate the data with the lowest lru field value from the cache.

When the data needs to be phased out again, Redis needs to pick the data to enter the candidate set created when the first phase-out is made. The selection criteria here is that the lru field value of the data that can enter the candidate set must be less than the minimum lru value in the candidate set. When new data enters the candidate dataset, if the number of data in the candidate dataset reaches maxmemory-samples,Redis, the data with the lowest lru field value in the candidate dataset will be eliminated.

Suggestions for use:

Priority is given to allkeys-lru policy. In this way, we can make full use of the advantages of LRU, a classical caching algorithm, to keep the most frequently accessed data in the cache and improve the access performance of the application. If there is a clear distinction between hot and cold data in your business data, I suggest you use the allkeys-lru strategy.

If there is little difference in data access frequency in business applications, and there is no obvious distinction between hot and cold data, it is recommended to use allkeys-random strategy and randomly select eliminated data.

If you have top-set requirements in your business, such as set-top news or set-top video, you can use the volatile-lru strategy without setting the expiration time for the set-top data. In this way, the data that needs to be topped will never be deleted, while other data will be filtered according to LRU rules when it expires.

How to deal with the eliminated data?

Once the obsolete data is selected, if the data is clean, then we will delete it directly; if the data is dirty, we need to write it back to the database.

So how do you tell whether a data is clean or dirty?

The difference between clean data and dirty data is whether it has been modified compared to the value originally read from the back-end database. Clean data has not been modified, so the data in the back-end database is also up-to-date. When replacing, it can be deleted directly.

The dirty data has been modified and is inconsistent with the data stored in the back-end database. At this point, if the dirty data is not written back to the database, the latest value of the data will be lost, which will affect the normal use of the application.

Even if the obsolete data is dirty, Redis will not write it back to the database. So, when we use Redis caching, if the data is modified, we need to write it back to the database when it is modified. Otherwise, when the dirty data is eliminated, it will be deleted by Redis, and there is no up-to-date data in the database.

How does Redis optimize memory?

1. Control the number of key: when using Redis to store a large amount of data, there are usually a large number of keys, and too many keys will also consume a lot of memory. Redis is essentially a data structure server, which provides us with a variety of data structures, such as hash,list,set,zset and other structures. Don't get into a misunderstanding when using Redis, use a lot of API like get/set and use Redis as a Memcached. For storing the same data content, using Redis's data structure to reduce the number of outer keys can also save a lot of memory.

2, reduce the key object, reduce the use of Redis memory the most direct way is to reduce the length of the key (key) and value (value).

Key length: when designing keys, in the case of a complete description of the business, the shorter the key value, the better.

Value length: the reduction of value objects is complex, and the common requirement is to serialize business objects into binary arrays and put them into Redis. First of all, business objects should be streamlined and unnecessary attributes should be removed to avoid storing invalid data. Secondly, in the choice of serialization tools, we should choose a more efficient serialization tool to reduce the size of the byte array.

3. Coding optimization. Redis provides external types such as string,list,hash,set,zet, but Redis internally has the concept of coding for different types, and the so-called coding is which underlying data structure is used to implement it. Different coding will directly affect the memory consumption and reading and writing efficiency of the data.

1. RedisObject object

Type field:

Take advantage of collection type data, because usually many small Key-Value can be stored together in a more compact way. Use a hash table (hashes) as much as possible. A hash table (that is, a small number stored in a hash table) uses very little memory, so you should abstract your data model into a hash table as much as possible. For example, if you have a user object in your web system, don't set a separate key for the user's name, last name, mailbox, and password. Instead, store all the user's information in a hash table.

Encoding field:

There are obvious differences in memory footprint by using different codes.

Lru field:

Development Tip: you can use the scan + object idletime command to query which keys have not been accessed for a long time, and find out the keys that have not been accessed for a long time to clean up to reduce memory footprint.

Refcount field:

When objects are integers and range is [0-9999], Redis can save memory by sharing objects.

Ptr field:

Development Tip: in high concurrent write scenarios, it is recommended that the string length be limited to 39 bytes if conditions permit, so as to reduce the number of redisObject memory allocations created to improve performance.

2. Reduce key value object

The most direct way to reduce Redis memory usage is to reduce the length of keys (key) and values (value).

You can use a general compression algorithm to compress json,xml and then save it to Redis, thereby reducing memory footprint

3. Shared object pool

Object sharing pool refers to the pool of integer objects maintained internally by Redis [0-9999]. Creating a large number of integer types redisObject has memory overhead, each redisObject internal structure takes at least 16 bytes, even more than the integer itself space consumption. So Redis memory maintains a [0-9999] pool of integer objects to save memory. In addition to integer-valued objects, other types such as list,hash,set,zset internal elements can also use integer object pooling. Therefore, in the development to meet the needs of the premise, try to use integer objects to save memory.

When maxmemory is set and LRU-related phase-out policies such as: volatile-lru,allkeys-lru are enabled, Redis prohibits the use of shared object pools.

Why is the object pool invalid when maxmemory and LRU phase-out policies are enabled?

The LRU algorithm needs to obtain the last access time of the object in order to eliminate the longest unaccessed data, and the last access time of each object is stored in the lru field of the redisObject object. Object sharing means that multiple references share the same redisObject, and the lru field is also shared, making it impossible to get the last access time for each object. If maxmemory is not set, Redis will not trigger memory collection until memory is exhausted, so shared object pooling works properly.

To sum up, the shared object pool conflicts with the maxmemory+LRU policy and should be used with care.

Why is there only a pool of integer objects?

First of all, the probability of reuse of integer object pool is the greatest, and then one of the key operations of object sharing is to judge equality. The reason why Redis only has integer object pool is that the time complexity of integer comparison algorithm is O (1), and only ten thousand integers are retained in order to prevent object pool waste. If the equality is judged by a string, the time complexity becomes O (n), especially when long strings consume more performance (floating-point numbers are stored using strings within Redis). For more complex data structures such as hash,list, equality judgment needs O (N2). For single-threaded Redis, this overhead is obviously unreasonable, so Redis only retains a pool of shared objects that are integers.

4. String optimization

Instead of using the string type of native C language, Redis implements its own string structure, internal simple dynamic string, referred to as SDS.

String structure:

Features:

O (1) time complexity acquisition: string length, used length, unused length.

Can be used to save byte arrays and support secure binary data storage.

Internal implementation of space pre-allocation mechanism to reduce the number of memory reallocation.

Lazy deletion mechanism, the space after string reduction is not released, and is reserved as pre-allocated space.

Pre-allocation mechanism:

Development Tip: minimize frequent string modification operations such as append,setrange, and use set to modify strings directly to reduce memory waste and memory fragmentation caused by pre-allocation.

String reconstruction: a second-level encoding based on hash type.

How to use the second-level code?

The ID length used in the two-level coding method is particular.

A problem is involved-the Hash type uses a compressed list when the underlying structure is less than the set value and a hash table when it is greater than the set value.

Once you move from a compressed list to a hash table, the Hash type is always saved in the hash table instead of going back to the compressed list.

Hash tables are not as efficient as compressed lists in terms of saving memory space. In order to make full use of the compact memory layout of compressed lists, it is generally necessary to control the number of elements saved in Hash.

5. Coding optimization

Hash types encoded with compressed list ziplist still save a lot of memory than collections encoded with hashtable.

6. Control the number of key

Development Tip: after using ziplist+hash to optimize keys, if you want to use the timeout delete function, developers can store the writing time of each object, and then scan the data through scheduled tasks using the hscan command to find out the timeout data items in the hash to delete.

When Redis is out of memory, the first thing to consider is not to add a machine to scale horizontally, but to try to optimize memory first. When you encounter a bottleneck, consider horizontal expansion. Even for clustering schemes, vertical optimization is also important to avoid unnecessary waste of resources and management costs after clustering.

This is the end of the content about "Redis memory is full and then go to optimize". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.