Shulou(Shulou.com)06/01 Report--

Introduction

Redis is a high performance distributed key-value database. It supports a variety of data structures and can be applied to buffers, queues and other scenarios. Friends who have used Redis may already be familiar with these, so I'd like to talk about what Redis may not know by everyone.

Redis persistence mechanism

When you see the title, you might say, I know, it's just RDB and AOF. These are platitudes. So let's talk in depth today about the logic and principles of these two persistence methods.

The principle of RDB

There are two kinds of triggers for RDB persistence in Redis: manual triggering and Redis timing triggering.

For persistence in RDB mode, manual triggers can be used:

(1) save: the current Redis server will be blocked until the persistence is complete, so it should be disabled online.

(2) bgsave: this trigger will fork a child process, and the child process will be responsible for the persistence process, so blocking will only occur when the token child process.

The scenarios that are automatically triggered are as follows:

Trigger automatically according to our save m n configuration rules

When the slave node replicates in full, the master node sends the rdb file to the slave node to complete the copy operation, and the master node will trigger bgsave

Penalty when performing debug reload

When shutdown is executed, it will also be triggered if aof is not enabled.

Since save is rarely used, let's take a look at how the command bgsave does the persistence of RDB.

RDB file saving process

(1) redis calls fork and now has a child process and a parent process.

(2) the parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because os's write-time copy mechanism (copy on write) parent and child processes share the same physical page, when the parent process processes the write request, os creates a copy of the page to be modified by the parent process instead of writing to the shared page. Therefore, the data in the address space of the child process is a snapshot of the entire database at the time of fork.

(3) when the child process has finished writing the snapshot to the temporary file, replace the original snapshot file with the temporary file, and then the child process exits.

PS:fork operations can block, causing Redis read and write performance to degrade. We can control the maximum memory of a single Redis instance to minimize the time consumption of Redis in fork, or control the frequency of automatic trigger to reduce the number of fork.

The principle of AOF

Generally speaking, the whole process of AOF can be divided into two steps, one is the real-time writing of commands (in the case of appendfsync everysec configuration, there will be 1s loss), and the second step is to rewrite the aof file.

The main process for incrementally appending to a file is:

(1) Command write

(2) append to aof_buf

(3) synchronize to aof disk

So why write buf first and then synchronize to disk? If you write to disk in real time, it will result in very high disk IO, which will affect the overall performance.

AOF rewriting

You can think, every write command generates a log, so will the AOF file be very large? The answer is yes, the AOF file will get bigger and bigger, so Redis provides another feature called AOF rewrite. Its function is to regenerate an AOF file, and the operation of a record in the new AOF file will only occur once, unlike an old file, which may record multiple operations on the same value.

Manual trigger: bgrewriteaof

Automatic trigger is triggered according to configuration rules. Of course, the overall time of automatic trigger is also related to the frequency of scheduled tasks in Redis.

Let's take a look at the flowchart of the rewrite:

(1) redis calls fork, and now there are two processes: father and son.

(2) the child process writes the command to rebuild the database state to the temporary file according to the database snapshot in memory.

(3) the parent process continues to process the client request, except for writing the write command to the original aof file. At the same time, write commands received are cached. This ensures that there will be no problems if the child process rewrite fails.

(4) when the child process writes the snapshot contents to a temporary file, the child process signals the parent process. The parent process then writes the cached write command to the temporary file.

(5) now the parent process can replace the old aof file with a temporary file and rename it, and the write commands received later begin to append to the new aof file.

PS: note that the operation of rewriting the aof file does not read the old aof file, but rewrites the entire database contents in memory to a new aof file as a command, which is a bit similar to a snapshot.

Why is Redis so fast?

Redis uses a memory-based single-process single-threaded model of KV database, written by C language, the official data is up to 1000000 + QPS (number of queries per second). This data is no worse than the same memory-based KV database Memcached using single-process multithreading! The reasons are as follows:

1. Based entirely on memory, most of the requests are purely memory operations, which is very fast.

2. The data structure is simple and the data operation is simple. The data structure in Redis is specially designed.

3. Single thread avoids unnecessary context switching and competition conditions, and there is no CPU consumption caused by multi-process or multi-thread switching, there is no need to consider all kinds of locks, there is no lock release operation, and there can be no performance consumption caused by deadlocks.

4. Using the multi-channel Istroke O multiplexing model, non-blocking IO

5. The underlying model used is different, the underlying implementation and the application protocol for communicating with the client are different, so Redis directly constructs its own VM mechanism.

Multi-channel I / O multiplexing model

The multiplexing model of multiple streams uses the ability of select, poll, and epoll to monitor multiple streams' Icano events at the same time, blocking the current thread when idle, and waking up from the blocked state when one or more streams have Icano events, so the program polls all streams once (epoll polls only those streams that actually emit events) and processes only those that are ready in sequence. This practice avoids a large number of useless operations. Here, "multiplex" refers to multiple network connections, and "multiplexing" refers to the reuse of the same thread. The use of multi-channel IO multiplexing technology enables a single thread to process multiple connection requests efficiently (minimizing the time consumption of the network IGO).

Redis transaction

A transaction (transaction) in Redis is a collection of commands. Transactions, like commands, are the smallest unit of execution in Redis. Commands in a transaction are either executed or not executed. The realization of Redis transaction requires two commands: MULTI and EXEC. At the beginning of the transaction, the MULTI command is sent to the Redis server, then the commands that need to be processed in this transaction are sent in turn, and finally the EXEC command is sent to indicate the end of the transaction command.

For example, connect to redis using redis-cli, and then enter the following command in the command line tool:

As you can see from the output, when the MULTI command is entered, the server returns OK to indicate that the transaction is successful, and then enters all the commands that need to be executed in this transaction in turn. Each time the server enters a command, it does not execute immediately, but returns "QUEUED", which indicates that the command has been accepted by the server and temporarily saved. Finally, after entering the EXEC command, all commands in this transaction will be executed in turn. You can see that in the end, the server returned three OK at a time, and the results returned here are in order with the commands sent, which means that all the commands in this transaction have been executed successfully.

For another example, enter the following command in the command line tool:

As in the previous example, entering MULTI first and then EXEC indicates that the intermediate command belongs to a transaction, except that there is an error in the intermediate command (set is written as sett), so because there is an incorrect command that causes other commands in the transaction not to be executed, you can see that all commands in the transaction are consistent.

If the client is disconnected before sending the EXEC command, the server clears the transaction queue and all commands in the transaction are not executed. Once the client sends the EXEC command, all commands in the transaction are executed, even if the client is disconnected since then, because the server has saved all the commands in the transaction.

In addition to ensuring that all commands in a transaction are either executed or not executed, Redis transactions also ensure that commands in one transaction are executed in turn without being inserted by other commands. Imagine that a client A needs to execute several commands, and client B sends several commands at the same time. If you do not use transactions, the commands of client B may be inserted into several commands of client A. if you want to avoid this situation, you can also use transactions.

Redis transaction error handling

What does Redis do if a command in a transaction goes wrong? To answer this question, you must first find out what caused the command execution error:

1. Syntax error: as in the above example, a syntax error indicates that the command does not exist or the parameter is incorrect. In this case, it is necessary to distinguish the version of Redis. The version before 2.6.5 of Redis ignores the wrong command and executes other correct commands. The version after 2.6.5 ignores all commands in this transaction and does not execute them.

two。 A run error indicates that an error occurred during the execution of the command, such as using the GET command to get a key value of a hash table type. This error cannot be detected by Redis before the command is executed, so such a command is accepted and executed by Redis in a transaction. If one command in the food is executed incorrectly, other commands will still be executed (including those after the error).

Transactions in Redis do not have transaction rollback (rollback) capabilities in relational databases, so consumers have to clean up the rest of the mess themselves. However, because Redis does not support transaction rollback, it also makes Redis transactions simple and fast.

WATCH, UNWATCH, DISCARD commands

We can see from the above example that the results of each command cannot be obtained until all the commands in the transaction have been executed, but what if command B in a transaction depends on the result of his previous command A? For example, to implement a function similar to iTunes + in Java, you have to get the current value before you can add one to the current value. It is impossible to use only MULTI and EXEC described above, because the commands in MULTI and EXEC are executed together, and the execution result of one command cannot be used as the execution parameter of the other command, so it is necessary to introduce another member of the Redis transaction family: the WATCH command.

From another perspective, think about the above method of implementing iTunes +, which can be achieved as follows:

Monitor the value of I to ensure that the value of I will not be modified to obtain the original value of I. If the value of I has not been modified in the process, the current value of I will be + 1, otherwise it will not be executed.

The WATCH command can monitor one or more keys, and once one of the keys is modified (or deleted), the subsequent transaction is not executed, monitoring continues until the EXEC command (the command in the transaction is executed after the EXEC, and the monitored key is automatically UNWATCH after the EXEC command is executed). For example:

In the above example, first set the key value of mykey to 1, then use the WATCH command to monitor mykey, then change the value of mykey to 2, then enter the transaction, set the value of mykey to 3 in the transaction, then execute EXEC to run the command in the transaction, and finally use the get command to check the value of mykey, it is found that the value of mykey is still 2, that is, the command in the transaction is not executed at all (because mykey was modified when WATCH was monitoring mykey So the subsequent transaction will be cancelled.

The UNWATCH command can unmonitor a key after the WATCH command is executed and before the MULTI command is executed. For example:

In the above example, first set the key value of mykey to 1, then use the WATCH command to monitor mykey, then change the value of mykey to 2, then cancel the monitoring of mykey, enter the transaction, set the value of mykey to 3 in the transaction, then execute EXEC to run the command in the transaction, and finally use the get command to check the value of mykey. It is found that the value of mykey is still 3, that is, the command in the transaction runs successfully.

The DISCARD command can cancel the MULTI command and empty the transaction queue after the WATCH command is executed and before the EXEC command is executed, and then exit from the transaction state. For example:

In the above example, first set the key value of mykey to 1, then use the WATCH command to monitor mykey, then change the value of mykey to 2, then enter the transaction, set the value of mykey to 3 in the transaction, then execute the DISCARD command, and then execute the EXEC to run the command in the transaction, and find the error "ERR EXEC without MULTI", indicating the successful execution of the DISCARD command-- cancel the WATCH command and empty the transaction queue, and then exit from the transaction state.

Redis distributed lock

The WATCH, MULTI and EXEC commands of Redis described above will only notify the client executing these commands to undo the modification of the data if the data is preemptively modified by other clients, which does not prevent other clients from modifying the data, so it can only be called optimistic optimistic locking.

This optimistic lock is not extensible-when the client tries to complete a transaction, it may retry repeatedly because the transaction execution fails. It is important to ensure the accuracy of the data, but when the load increases, the use of optimistic locks is not perfect. At this point, you need to use Redis to implement distributed locks.

Distributed lock: a way to control synchronous access to shared resources between distributed systems. In distributed systems, it is often necessary to coordinate their actions. If one or a group of resources are shared between different systems or different hosts of the same system, mutual exclusion is often needed to prevent interference from each other to ensure consistency when accessing these resources. distributed locks are needed.

The Redis command describes:

The SETNX command (SET if Not eXists) is mainly used to realize the distributed lock in Redis.

Syntax: SETNX key value

Function: if and only if key does not exist, set the value of key to value and return 1; if the given key already exists, SETNX does not take any action and returns 0.

Use Redis to build locks:

Idea: set the "lock:" parameter name to the key of the lock, use the SETNX command to try to set a random uuid to the value of the lock, and set the expiration time for the lock. Setting the value of the lock using SETNX can prevent the lock from being acquired by other processes. If the attempt to acquire the lock fails, the program will retry until the lock is successfully acquired or the given time limit is exceeded.

Code:

Public String acquireLockWithTimeout (Jedis conn, String lockName, long acquireTimeout, long lockTimeout) {String identifier = UUID.randomUUID () .toString (); / the value of the lock String lockKey = "lock:" + lockName; / / the key of the lock int lockExpire = (int) (lockTimeout / 1000); / / the expiration time of the lock long end = System.currentTimeMillis () + acquireTimeout / / while (System.currentTimeMillis () < end) {/ / determine whether to exceed the lock acquisition time limit if (conn.setnx (lockKey, identifier) = = 1) {/ / determine whether the lock value is set successfully conn.expire (lockKey, lockExpire); / / set the lock expiration time return identifier; / / return the lock value} if (conn.ttl (lockKey) = =-1) {/ / determine whether the lock timeout conn.expire (lockKey, lockExpire) } try {Thread.sleep (1000); / / wait 1 second and then retry to set the value of the lock} catch (InterruptedException ie) {Thread.currentThread () .interrupt ();}} / / return null return null;} if the lock failed to acquire

Release of lock:

Idea: use the WATCH command to monitor the key that represents the lock, then check that the value of the key is the same as the value set when the lock was added, and delete the key after confirming that the value has not changed.

Code:

Public boolean releaseLock (Jedis conn, String lockName, String identifier) {String lockKey = "lock:" + lockName; / / key of the lock while (true) {conn.watch (lockKey); / / key of the monitoring lock if (identifier.equals (conn.get (lockKey) {/ / determine whether the value of the lock is consistent with that set when the lock is added, that is, check whether the process still holds the lock Transaction trans = conn.multi (); trans.del (lockKey) / / release the lock in the Redis transaction List results = trans.exec (); if (results = = null) {continue; / / retry after the transaction fails (the monitored key is modified to cause the transaction to fail, re-monitor and release the lock)} return true;} conn.unwatch (); / / unmonitor break;} return false;}

Summary

The above is the whole content of this article. I hope the content of this article has a certain reference and learning value for everyone's study or work. Thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.