Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the asynchronous mechanism of Redis". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the asynchronous mechanism of Redis".

First, the blocking point of Redis

The objects that interact with the Redis instance, and the actions that occur during the interaction:

Client: network IO, key-value pair add, delete, change and search operation, database operation

Disk: generate RDB snapshot, record AOF log, AOF log rewrite

Master-slave node: the master library generates and transfers RDB files, receives RDB files from the library, empties the database, and loads RDB files

Slicing cluster instances: transfer hash slot information to other instances, data migration.

The relationship between four types of interactive objects and specific operations:

Blocking points when interacting with the client:

Network IO is sometimes slow, but Redis uses IO multiplexing mechanism to prevent the main thread from waiting for a network connection or request to arrive, so network IO is not a factor causing Redis blocking.

The operation of adding, deleting, modifying and querying key-value pairs is not only the main part of the interaction between Redis and the client, but also the main task executed by the main thread of Redis. Complex add, delete, modify and search operations will definitely block Redis.

The standard to judge the complexity of the operation: to see whether the complexity of the operation is O (N).

The first blocking point of Redis: collection full query and aggregate operation:

The complexity of operation involving sets in Redis is usually O (N), which should be paid attention to when using it.

For example, the collection element full query operations HGETALL, SMEMBERS, and aggregate statistical operations of the collection, such as intersection, union, and difference sets.

The second blocking point of Redis: bigkey delete operation

The deletion of the collection itself also has a potential blocking risk. The essence of the delete operation is to release the memory space occupied by the key-value pair. Freeing memory is only the first step. In order to manage memory space more efficiently, when an application frees memory, the operating system needs to insert the freed memory block into a linked list of free memory blocks for later management and reallocation.

This process itself takes a certain amount of time and blocks the application that currently releases memory. If a large amount of memory is released at once, the operation time of the free memory block linked list will increase, which in turn will cause blocking of the Redis main thread.

Time to free up a lot of memory: when you delete a large number of key-value pairs of data, delete a collection that contains a large number of elements, also known as bigkey deletion.

The time spent by a collection with a different number of elements in a delete operation:

Three conclusions are drawn:

When the number of elements increases from 100000 to 1 million, the deletion time of the four collection types increases from 5 times to nearly 20 times.

The larger the collection element, the longer it takes to delete.

When deleting a collection of 1 million elements, the maximum absolute value of deletion time has reached 1.98s (Hash class). The response time of Redis is generally at the level of microseconds, and an operation reaches nearly 2 seconds, which inevitably blocks the main thread.

The third blocking point of Redis: clearing the database

Since frequent deletion of key-value pairs is a potential blocking point, emptying databases (such as FLUSHDB and FLUSHALL operations) is also a potential blocking risk in Redis database-level operations, as it involves deleting and releasing all key-value pairs.

The fourth blocking point of Redis: synchronous writing of AOF logs

Disk IO is generally time-consuming and needs to be paid more attention. Redis developers have long realized that disk IO can cause blocking, so Redis is designed to use child processes to generate RDB snapshot files and perform AOF log rewriting operations. With the child process responsible for execution, the slow disk IO will not block the main thread.

When Redis records AOF logs directly, the data will be saved on disk according to different write-back policies. A synchronous write to disk operation takes about 1~2ms, and if a large number of writes need to be recorded in the AOF log and written back synchronously, the main thread will be blocked.

The fifth blocking point of Redis: loading RDB files from the library

In the master-slave cluster, the master library needs to generate a RDB file and transfer it to the slave library.

In the process of copying the main library, the creation and transfer of RDB files are done by child processes and will not block the main thread.

However, after receiving the RDB file from the slave library, you need to use the FLUSHDB command to empty the current database, just in time to hit the third blocking point.

After emptying the current database, the library needs to load the RDB file into memory. The speed of this process is closely related to the size of the RDB file. The larger the RDB file, the slower the loading process.

Blocking points when slicing cluster instances interact

When deploying a Redis slicing cluster, the hash slot information allocated on each Redis instance needs to be transferred between different instances. When load balancing is needed or instances are added or deleted, the data will be migrated between different instances. However, the amount of information in the hash slot is small, and the data migration is performed gradually, and these two types of operations do not pose a great risk of blocking the main thread of Redis.

If you use the Redis Cluster scheme and happen to migrate bigkey at the same time, it will block the main thread because Redis Cluster uses synchronous migration.

Five blocking points:

Collection full query and aggregation operation

Bigkey deletion

Empty the database

AOF log synchronous write

Load the RDB file from the library.

Blocking points that can be executed asynchronously

To avoid blocking operations, Redis provides an asynchronous threading mechanism:

Redis starts some child threads and then leaves some tasks to them to complete in the background instead of being performed by the main thread. Blocking the main thread can be avoided.

Requirements for operations to be executed asynchronously:

An operation that can be performed asynchronously is not an operation on the critical path of the Redis main thread (the client sends the request to the Redis and waits for the Redis to return the data result).

After the main thread receives operation 1, operation 1 does not need to return specific data to the client, the main thread can give it to the backstage child thread to complete, and just return a "OK" result to the client.

When the child thread executes operation 1, the client sends operation 2 to the Redis instance. The client needs to use the data result returned by operation 2. If operation 2 does not return the result, the client will be waiting all the time.

Operation 1 is not an operation on the critical path because it does not have to return specific data to the client, so it can be performed asynchronously by the backstage child thread.

Operation 2 needs to return the result to the client, which is the operation on the critical path, so the main thread must finish the operation immediately.

The Redis read operation is a typical critical path operation because after the client sends the read operation, it waits for the read data to return for subsequent data processing. On the other hand, the first blocking point of Redis, "set full query and aggregation operation", involves read operations and cannot be operated asynchronously.

The delete operation does not need to return specific data results to the client, so it is not a critical path operation. "bigkey delete" and "empty database" both delete data and are not on the critical path. You can use backstage subthreads to perform delete operations asynchronously.

"AOF log synchronous write". In order to ensure data reliability, Redis instances need to ensure that the operation records in the AOF log have been set down. Although this operation requires the instance to wait, it will not return specific data results to the instance. So you can start a child thread to perform synchronous writes of the AOF log.

Load RDB file from library if you want to provide data access service to the client, you must finish loading the RDB file. This operation is also an operation on the critical path and must be performed by the master thread of the slave library.

Except for "set full query and aggregate operation" and "load RDB text from library", the operations involved in the other three blocking points are not on the critical path, and Redis's asynchronous subthreading mechanism can be used to delete bigkey, empty the database, and write AOF logs synchronously.

Third, asynchronous child thread mechanism

After the Redis main thread starts, it uses the pthread_create function provided by the operating system to create three child threads, which are responsible for the asynchronous execution of AOF log write, key-value deletion and file closure.

The main thread interacts with the child thread through a linked list of task queues.

When a key-value operation is received to delete and empty the database, the main thread encapsulates the operation into a task, puts it in the task queue, and then returns a completion message to the client indicating that the deletion has been completed.

But in fact, the deletion has not been performed at this time, and it is not until the backstage child thread reads the task from the task queue that the key-value pair is actually deleted and the corresponding memory space is freed. This asynchronous deletion is also called lazy deletion (lazy free).

When the AOF log is configured with the everysec option, the main thread encapsulates the AOF write log operation into a task, which is also placed in the task queue. After the backstage child thread reads the task, it starts to write the AOF log on its own, and the main thread does not have to wait for the AOF log to finish writing.

Asynchronous child thread execution mechanism in Redis:

Asynchronous key-value pair deletion and database emptying operations are provided after Redis 4.0. Redis also provides new commands to perform these two operations:

Key-value pair deletion: when there are a large number of elements in the collection type (for example, millions or tens of millions of elements) that need to be deleted, it is recommended to use the UNLINK command

Empty the database: you can add the ASYNC option after the FLUSHDB and FLUSHALL commands to let the backstage child thread empty the database asynchronously.

FLUSHDB ASYNC FLUSHALL AYSNC thank you for reading, the above is the content of "what is the asynchronous mechanism of Redis". After the study of this article, I believe you have a deeper understanding of what the asynchronous mechanism of Redis is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.