Shulou(Shulou.com)06/01 Report--

This article Xiaobian for you to introduce in detail "Redis hot and cold data identification and exchange how to achieve", the content is detailed, the steps are clear, the details are handled properly, I hope that this article "Redis hot and cold data identification and exchange how to achieve" article can help you solve doubts, the following follow the editor's ideas slowly in-depth, together to learn new knowledge.

Background

Redis hybrid storage product is a hybrid storage product independently developed by Aliyun that is fully compatible with Redis protocols and features.

By storing part of the cold data to disk, on the basis of ensuring that most of the access performance does not decline, the user cost is greatly reduced and the memory limit on the amount of data of a single instance of Redis is broken.

Among them, the identification and exchange of hot and cold data is the key factor of hybrid storage product performance.

Hot and cold data definition

In Redis hybrid storage, the ratio of memory to disk is free for users to choose:

The Redis hybrid storage instance regards all Key as hot data, ensuring that the performance of all Key access requests is efficient and consistent at the cost of a small amount of memory. For the Value part, in the case of insufficient memory, the instance itself will select part of the value as cold data to be asynchronously stored on disk according to the recent access time, access frequency, Value size and other dimensions until the memory is less than the specified threshold.

In the Redis hybrid storage instance, we consider all Key to be hot data in memory because of the following two considerations:

Key is much more frequently accessed than Value.

As a KV database, the usual access requests need to look for Key to confirm the existence of Key, and to confirm that a key does not exist, you need to check the collection of all Key in some form. Keeping all the Key in memory ensures that the search speed of the key is exactly the same as that of the pure memory version.

The proportion of Key size is very low.

Even if it is a normal string type, Value is several times larger than Key in the usual business model. For collection objects such as Set,List,Hash, the total Value of all members is several orders of magnitude larger than Key.

Therefore, there are two main applicable scenarios for Redis hybrid storage instances:

Uneven data access and hot data

There is not enough memory to hold all the data, and the Value is larger (compared to Key)

Hot and cold data identification

When memory is insufficient, the instance calculates the weight of value based on the dimensions such as recent access time, access frequency and value size, and stores the value with the lowest weight on disk and deletes it from memory.

The pseudo code is as follows:

Ideally, we certainly want to be able to accurately calculate the current coldest value. However, the hot and cold degree of the value varies dynamically according to the access situation, and the time consumption of recalculating the hot and cold weights of all value each time is totally unacceptable.

When the memory is full, Redis itself will phase out the data according to the elimination policy set by the user, and writing hot data from memory to disk can also be regarded as a "elimination" process. Considering the performance, accuracy and user understanding, we adopt the approximate calculation method similar to Redis in hot and cold data recognition, support a variety of strategies, reduce CPU and memory consumption by randomly sampling a small part of data, and use sampling history information through eviction pool to help improve accuracy.

The picture above is a schematic diagram of the hit rate of the Redis approximate elimination algorithm under different versions and different sample size configurations. The light gray point is the eliminated data, the gray point is the uneliminated data, and the green point is the newly added data during the test.

Hot and cold data exchange

Redis mixed storage in hot and cold data exchange process is completed in the background IO thread.

Hot data-> cold data

Asynchronous mode:

When the memory is close to the maximum, the main thread generates a series of data swapping tasks

The backstage thread performs these data swapping tasks and notifies the main thread when the execution is complete.

The main thread update releases the value in memory, and updates the value in the data dictionary in memory to a simple meta-information.

Synchronization mode:

If the write traffic is too large, there is no time to swap out data asynchronously, resulting in memory exceeding the maximum memory specification. The main thread will directly perform the data swap task to achieve the purpose of current restriction in disguised phase.

Cold data-> hot data

Asynchronous mode:

Before executing the command, the main thread determines whether the value involved in the command is in memory.

If not, generate a data load task, suspend the client, and the main thread continues to process other client requests

The backstage thread executes the data loading task and notifies the main thread when the execution is complete.

The main thread updates the value in the data dictionary in memory, waking up the previously suspended client to process its request.

Synchronization mode:

In the Lua script, specific command execution phase, if value is found to be stored on disk, the main thread will directly execute the data loading task to ensure that the semantics of Lua scripts and commands remain unchanged.

After reading this, the article "how to identify and exchange hot and cold Redis data" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it yourself. If you want to know more about related articles, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.