Shulou(Shulou.com)06/01 Report--

Preface

Redis provides two ways of data storage, namely: cache-only & & persistence;cache-only is used with caching service, and the data will disappear after the server is terminated. In this mode, there will be no way of "data recovery". It is a mode with low security, high efficiency and easy expansion. Persistence means that the data in memory is permanently backed up to disk files, and the data can be recovered after the service is restarted. In this mode, the security of the data is greatly improved. There is nothing to say about cache-only, but here we focus on the persistent storage mode of Redis. For persistence persistent storage, Redis provides two persistence methods: "Redis DataBase" for "RDB" and "Append-Only Flie" for "AOF". Of course, in addition to these two methods, Redis also used virtual memory in earlier versions, but it has been abandoned since Redis 3.0.

RDB persistence mode

RDB is the abbreviation of "Redis DataBase". RDB persistence means that the dataset in memory is snapped and written to disk according to the specified policy. Is the default persistence method, which writes in-memory data to a binary file as a snapshot, and the default file name is dump.rdb. You can set the way to take snapshots automatically by modifying the configuration file. We can set redis to take snapshots automatically if the key is modified more than m times in n seconds. Here is the default snapshot save configuration. The matching priority is from bottom to top.

Save 9001 # 900s if the key is modified more than once, initiate a snapshot save

If the key is modified more than 10 times in save 30010 # 300 seconds, a snapshot save will be initiated.

If 10000 value modifications occur within save 60 10000 # 60s, a snapshot will be initiated.

1 RDB file stored procedure

1) when there is a related operation, the redis parent process calls fork () to create the child process.

2) the parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because os's write-time copy mechanism (copy on write) parent and child processes share the same physical page, when the parent process processes the write request, os creates a copy of the page to be modified by the parent process instead of writing to the shared page. So the number in the address space of the child process is a snapshot of the entire database at the time of fork ().

3) when the child process has finished writing the snapshot to the temporary file, replace the original snapshot file with the temporary file, and then the child process exits.

Redis's client can also use the "save" or "bgsave" commands to tell redis to do a snapshot persistence. The save operation saves the snapshot in the main thread, and since redis uses a main thread (that is, a single process) to handle all client requests, this method will block all client requests and is not recommended.

It is worth noting that each time RDB writes all the memory data to disk once, not incrementally just synchronizing the dirty data. If there is a large amount of data and a lot of write operations, it will inevitably cause a large number of disk IO operations, which may seriously affect performance.

2 advantages of RDB

1) once you use this approach, your entire Redis database will contain only one file, which is very convenient for backup. For example, you can back up your data every day without worrying about running out of disk space. And we can easily move a RDB file to other storage media.

2) RDB is faster than AOF in recovering large datasets.

3) RDB can maximize the performance of Redis: the only thing the parent process needs to do when saving the RDB file is to call fork () to generate a child process, and then the child process will handle all the rest of the save work, while the parent process does not need to participate in disk IO, greatly improving performance.

3 disadvantages of RDB

1) if the server fails at runtime, then RDB will lose data, although Redis allows you to set different save points (save point) to control the frequency of saving RDB files, but because RDB files need to save the state of the entire dataset, it is not an easy operation. So you may save the RDB file at least once in 5 minutes. In this case, in the event of a downtime, you may lose the data in the backup period.

2) each time the RDB is saved, the Redis needs to fork () a child process, and the child process carries out the actual persistence work. When the dataset is large, fork () can be very time-consuming, causing the server to stop processing the client within a millisecond; if the dataset is very large and CPU time is very tight, the stop time may even be as long as a full second. Although AOF rewriting also requires fork (), there is no loss of data durability no matter how long the execution interval of AOF rewriting is.

4 relevant configuration parameters of RDB

In addition to the items above that define RDB snapshot conditions, there are some other configuration parameters that are placed in the "SNAPSHOTTING" field in the configuration file "/ etc/redis.conf"; the details are as follows:

Whether rdbcompression yes # enables compression

Rdbchecksum yes # checksum of test data

Dbfilename dump.rdb # dataset filename

Dir / var/lib/redis # data storage directory

Stop-writes-on-bgsave-error yes # by default, if RDB snapshots are enabled (with at least one SavePoint) and the latest data fails to be saved, writes are stopped. This will make the user realize (in a difficult way) that the data is not being persisted correctly on disk, otherwise it is likely that no one will notice and some errors will occur, even if there are disk, permission problems, and so on. Still working as usual. If the background save process starts working again, Redis will automatically allow you to write again. However, if you have set up proper monitoring of the Redis server and persistence, you should disable this feature so that Redis will continue to work even if there are problems with disks, permissions, and so on.

AOF persistence mode

AOF is the abbreviation of "Append-Only File". It is a way to append the data in memory to a temporary file in the form of a command, and then rely on this file for data reproduction.

1 AOF file saving process

Redis appends every write command received to the file through the write function (default is appendonly.aof). When redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write command saved in the file. Of course, because os caches changes made by write in the kernel, it may not be written to disk immediately. In this way, it is still possible to lose some of the changes in AOF persistence. However, we can tell redis through the option that we want to force os to write to disk through the fsync function. There are three ways to do the following (default: fsync once per second)

Appendonly yes # enable aof persistence mode

Appendfsync always # is forced to write to disk every time it receives a write command. Although complete persistence is guaranteed, it seriously affects performance and is not recommended.

Appendfsync everysec # forces writing to disk once per second, which makes a good compromise between performance and persistence. It is recommended.

Appendfsync no # is completely dependent on os, that is, Redis is not involved. This performance is the best, but persistence is not guaranteed.

The AOF approach also brings another problem. Persistent files will get bigger and bigger. For example, if we call the "incr count" command 100 times, all 100 commands must be saved in the file, but the first 99 are superfluous. Because you only need to execute "set test 100" when you want to restore the state of the database. To compress the persistence file of aof. Redis provides the "bgrewriteaof" command. Upon receipt of this command, redis will save the data in memory to a temporary file (similar to the binary log in Mysql) in a command similar to a snapshot, and finally replace the original file. The specific process is as follows:

1) redis calls fork to generate child processes

2) according to the database snapshot in memory, the child process writes the command to rebuild the database state to the temporary file.

3) the parent process continues to process the client request, except for writing the write command to the original aof file. At the same time, write commands received are cached. This ensures that there will be no problems if the child process rewrite fails.

4) when the child process writes the contents of the snapshot to a temporary file in the form of a command, the child process signals the parent process. The parent process then appends the cached write command to the temporary file.

5) now the parent process can replace the old aof file with a temporary file and rename it, and the write commands received later begin to append to the new aof file.

It is worth noting that the operation of rewriting the aof file does not read the old aof file, but rewrites the entire in-memory database contents to a new aof file as a command, which is somewhat similar to a snapshot.

2 AOF persistence advantage

1) using AOF persistence makes Redis very durable (much more durable): you can set different fsync policies, such as no fsync, fsync per second, or fsync each time a write command is executed. AOF's default policy is to fsync once per second, under this configuration, Redis can still maintain good performance, and even if a failure occurs, it will only lose data for up to one second (fsync executes in the background thread, so the main thread can continue to work hard on command requests).

2) the AOF file is a log file (append only log) that only performs append operations, so writing to the AOF file does not require seek. Even if the log contains commands that are not fully written for some reason (such as the disk is full at the time of writing, write stops, etc.), the redis-check-aof tool can easily fix this problem.

Redis can automatically rewrite AOF in the background when the AOF file becomes too large: the new AOF file after rewriting contains the minimum set of commands needed to recover the current dataset. The entire rewrite operation is perfectly safe because Redis continues to append commands to existing AOF files during the creation of new AOF files, and existing AOF files will not be lost even if downtime occurs during rewriting. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

3) the AOF file saves all write operations to the database in an orderly manner, and these write operations are saved in the format of the Redis protocol, so the contents of the AOF file are very easy to read and parse. Exporting (export) the AOF file is also very simple: for example, if you accidentally execute the FLUSHALL command, but as long as the AOF file is not rewritten, simply stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, and you can restore the dataset to the state it was before the FLUSHALL execution.

3 disadvantages of AOF persistence

1) for the same dataset, the volume of the AOF file is usually larger than that of the RDB file.

2) depending on the fsync policy used, AOF may be slower than RDB. In general, the performance of fsync per second is still very high, and turning off fsync can make AOF as fast as RDB, even under heavy loads. However, RDB can provide a more guaranteed maximum latency (latency) when dealing with large write loads.

3) AOF has had such a bug in the past: due to individual commands, when the AOF file is reloaded, the dataset cannot be restored to its original state. (for example, the blocking command BRPOPLPUSH has caused such a bug.) the test suite adds tests to this situation: they automatically generate random, complex data sets and make sure everything is all right by reloading the data. Although this kind of bug is not common in AOF files, it is almost impossible for RDB to have this kind of bug by comparison.

4 relevant configuration parameters of AOF

With regard to "AOF", one of the persistence methods of Redis, the configuration items related to it are placed in the "APPEND ONLY MODE" field in the Redis configuration file "/ etc/redis.conf"; the related parameters are as follows:

# enable AOF persistence mode

Appendonly no

# The name of the appendonly file (default: "appendonly.aof")

Appendfilename "appendonly.aof"

# AOF persistence methods, as long as there are three types: fsync (affecting performance) with write operations, fync per second (eclectic mode, only losing 1s of data), and OS execution of brushing (persistence is not guaranteed)

# appendfsync always

Appendfsync everysec

# appendfsync no

# whether fsync is not called during aof rewriting in the background. The default is no, which means calling. When the fsync policy of AOF is set to always or everysec, and the background save process (background save or AOF log background rewrite) performs a large number of fsync O on the disk, Redis may block on the fsync () call for too long. There is currently no solution to this problem, because even executing fsync in other threads will prevent our synchronous write calls. To alleviate this problem, you can use the following options to prevent fsync () from being called in the main process in the BGSAVE or BGREWRITEAOF process. This means that Redis has the same persistence as "appendfsync none" when another child process is saving. This means that in the worst case (using the default setting), logs may be lost for up to 30 seconds. If there is a delay problem, please change it to "yes"

No-appendfsync-on-rewrite no

# defines the file rewriting frequency. There are two parameters, one is percentage no, that is, the first command of the series, and the other is by AOF log size, which defaults to 64MB. If the current increment of the log is greater than the predefined growth rate (percentage), an overwrite is triggered. In addition, you need to specify the minimum size of the AOF file to be rewritten, which is still very small even if the percentage increase is reached, which is useful to avoid rewriting AOF files. Specifies a percentage of zero, which means that automatic AOF rewriting is disabled.

Auto-aof-rewrite-percentage 100

Auto-aof-rewrite-min-size 64mb

During Redis startup, when AOF data is loaded back into memory, an AOF file may be found to be truncated. If AOF-load-truncated is set to yes, a truncated AOF file is loaded and the Redis server begins to issue logs to notify the user of the event. If this option is set to no, the server will abort with an error and refuse to start. When the option is set to no, the user needs to repair the AOF file using the "redis-checkaof" utility before restarting the server.

Aof-load-truncated yes

Summary:

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence features. If you are very concerned about your data, but can still withstand data loss within a few minutes, then you can only use RDB persistence.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.