Shulou(Shulou.com)06/02 Report--

This article focuses on "the way of redis persistence". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "the way of redis persistence"!

Introduction

First of all, we should be clear about the use of persisted data, the answer is for data recovery after restart.

Redis is an in-memory database, whether it is RDB or AOF, it is only a measure to ensure data recovery.

So when Redis uses RDB and AOF for recovery, it will read the RDB or AOF file and reload it into memory.

How Redis is persisted

RDB

AOF

RDB

RDB is the Snapshot snapshot storage, which is the default persistence method. It can be understood as semi-persistence mode, that is, data is saved to disk periodically according to a certain strategy. The corresponding data file is dump.rdb, and the period of the snapshot is defined by the save parameter in the configuration file.

The following are the default snapshot settings:

Save 9001 # when a piece of Keys data is changed, refresh to Disk once in 900s

Save 30010 # when 10 pieces of Keys data are changed, refresh to Disk once in 300 seconds

Save 60 1000 refresh to Disk once in 60 seconds when 10000 pieces of Keys data have been changed

The RDB file for Redis will not break because its write operation is done in a new process. When a new RDB file is generated, the child process generated by Redis first writes the data to a temporary file, and then renames the temporary file to the RDB file through atomic rename system calls. In this way, the RDB file for Redis is always available whenever there is a failure. At the same time, the RDB file of Redis is also a part of the internal implementation of Redis master-slave synchronization.

For the first time, the implementation of synchronization from Slave to Master is as follows: * Slave sends a synchronization request to Master. Master first dump out rdb files, and then transfers all rdb files to slave, and then Master forwards the cached commands to Slave, and the first synchronization is completed.

The second and subsequent synchronization implementation is: * Master sends snapshots of variables directly to each Slave in real time. However, no matter what causes Slave and Master to disconnect and reconnect, the process of the above two steps will be repeated.

The master-slave replication of Redis is based on the persistence of memory snapshots. As long as there is Slave, there must be memory snapshots.

It is obvious that RDB has its shortcomings, that is, once there is a problem with the database, then the data saved in our RDB file is not new.

All the data from the last time the RDB file was generated to the time when Redis was down have been lost.

AOF

AOF (Append-Only File) has better persistence than RDB.

Because when using AOF persistence, Redis appends every write command received to the file through the Write function, similar to MySQL's binlog.

When Redis restarts, it rebuilds the contents of the entire database in memory by re-executing the write command saved in the file.

The corresponding setting parameters are: $vim / opt/redis/etc/redis_6379.conf

Appendonly yes enables AOF persistence mode

Name of the appendfilename appendonly.aof # AOF file, default is appendonly.aof

Appendfsync always is forced to write to disk every time it receives a write command, which is the most guaranteed and complete persistence, but it is also the slowest and is generally not recommended.

Appendfsync everysec forces writing to disk every second, making a good tradeoff between performance and persistence, and is the recommended approach.

Appendfsync no # relies entirely on OS writes, usually about once in 30 seconds, with the best performance but the least guaranteed persistence, and is not recommended.

The full persistence of AOF also brings another problem: persistent files get bigger and bigger.

For example, if we call the INCR test command 100 times, all 100 commands must be saved in the file, but 99 are actually superfluous.

Because in order to restore the state of the database, it is enough to save a SET test 100in the file.

To compress the persistence file of AOF, Redis provides the bgrewriteaof command.

After receiving this command, Redis will save the data in memory to a temporary file in a command similar to a snapshot, and finally replace the original file, so as to control the growth of AOF files.

Because it is the process of simulating the snapshot, when rewriting the AOF file, it does not read the old AOF file, but rewrites a new AOF file with the entire database contents in memory as a command.

The corresponding setting parameters are: $vim / opt/redis/etc/redis_6379.conf

No-appendfsync-on-rewrite yes

When the log is rewritten, the command append operation is not carried out, but is put in the buffer to avoid conflict with the command append on the DISK IO. Auto-aof-rewrite-percentage 100

When the current AOF file size is twice the size of the AOF file obtained from the last log rewrite, a new log rewrite process is automatically started. Auto-aof-rewrite-min-size 64mb # the minimum value of the current AOF file to start a new log rewrite process to avoid frequent rewriting due to small file size when Reids is just started.

Choice

What exactly do you choose? Here are some official suggestions:

In general, if you want to provide high data security, it is recommended that you use both persistence methods. If you can accept a few minutes of data loss caused by a disaster, then you can only use RDB. Many users only use AOF, but we recommend that since RDB can take a complete snapshot of the data from time to time and provide a faster restart, it is best to use RDB as well. Therefore, we hope to unify AOF and RDB into a persistence model in the future (long-term planning).

In terms of data recovery

The startup time of RDB will be shorter for two reasons:

There is only one record for each piece of data in the RDB file, unlike the AOF log, which may have multiple operations of a piece of data. So each piece of data only needs to be written once. Another reason is that the storage format of RDB files is consistent with the encoding format of Redis data in memory, so there is no need for data coding, so the CPU consumption is much less than the loading of AOF logs.

Disaster recovery simulation since the role of persistent data is for data recovery after restart, it is very necessary for us to carry out such a disaster recovery simulation. It is said that if the data is to be persisted and stable, it is recommended to leave half of the physical memory empty. Because when taking a snapshot, the child process that fork comes out for dump operation will occupy the same memory as the parent process, and the real copy-on-write will have a greater impact on performance and memory consumption. At present, the usual design idea is to use Replication mechanism to make up for the lack of performance of aof and snapshot, so as to achieve data persistence.

That is, neither Snapshot nor AOF is done on Master to ensure the read and write performance of Master, while Snapshot and AOF are enabled on Slave at the same time for persistence to ensure data security.

First, modify the following configuration on Master:

Sudo vim / opt/redis/etc/redis_6379.conf

Save 9001 # disable Snapshot

Save 300 10

Save 60 10000

Appendonly no # disable AOF

Next, modify the following configuration on Slave:

Sudo vim / opt/redis/etc/redis_6379.conf

Save 9001 # enable Snapshot

Save 300 10

Save 60 10000

Appendonly yes # enable AOF

Name of the appendfilename appendonly.aof # AOF file

Appendfsync always

Appendfsync everysec # force writing to disk once per second

Appendfsync no

No-appendfsync-on-rewrite yes # does not perform command append operation when log rewrite

Auto-aof-rewrite-percentage 100 # automatically starts a new log rewrite process

Auto-aof-rewrite-min-size 64mb # minimum value to start a new log rewrite process

Start Master and Slave respectively

$/ etc/init.d/redis start

Confirm that the Snapshot parameter is not started in Master after startup is completed.

Redis 127.0.0.1 6379 > CONFIG GET save

"save"

"

Then generate 250000 pieces of data in Master with the following script:

Dongguo@redis:/opt/redis/data/6379$ cat redis-cli-generate.temp.sh

#! / bin/bashREDISCLI= "redis-cli-a slavepass-n 1 SET" ID=1while ($ID info

Redis_version:2.4.17 redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:64 multiplexing_api:epoll gcc_version:4.4.5 process_id:27623 run_id:e00757f7b2d6885fa9811540df9dfed39430b642 uptime_in_seconds:1541 uptime_in_days:0 lru_clock:650187 used_cpu_sys:69.28 used_cpu_user:7.67 used_cpu_sys_children:0.00 used_cpu_user_ Children:0.00 connected_clients:1 connected_slaves:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 used_memory:33055824 used_memory_human:31.52M used_memory_rss:34717696 used_memory_peak:33055800 used_memory_peak_human:31.52M mem_fragmentation_ratio:1.05 mem_allocator:jemalloc-3.0.0 loading:0 aof_enabled:0 Changes_since_last_save:250000 bgsave_in_progress:0 last_save_time:1348677645 bgrewriteaof_in_progress:0 total_connections_received:250007 total_commands_processed:750019 expired_keys:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:246 vm_enabled:0 role:master slave0:10.6.1.144,6379 Online db1:keys=250000,expires=0

The current data volume is 250000 key, occupying 31.52m of memory.

Then we directly Kill the Redis process of Master to simulate the disaster.

Dongguo@redis:/opt/redis/data/6379$ sudo killall-9 redis-server

Let's check the status in Slave:

Redis 127.0.0.1 6379 > info

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version:4.4.5process_id:13003run_id:9b8b398fc63a26d160bf58df90cf437acce1d364uptime_in_seconds:1627uptime_in_days:0lru_clock:654181used_cpu_sys:29.69used_cpu_user:1.21used_cpu_sys_children:1.70used_cpu_user_children:1.23connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf: 0blocked_clients:0used_memory:33047696used_memory_human:31.52Mused_memory_rss:34775040used_memory_peak:33064400used_memory_peak_human:31.53Mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_since_last_save:3308bgsave_in_progress:0last_save_time:1348718951bgrewriteaof_in_progress:0total_connections_received:4total_commands_processed:250308expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_channels: 0pubsub_patterns:0latest_fork_usec:694vm_enabled:0role:slaveaof_current_size:17908619aof_base_size:16787337aof_pending_rewrite:0aof_buffer_length:0aof_pending_bio_fsync:0master_host:10.6.1.143master_port:6379master_link_status:downmaster_last_io_seconds_ago:-1master_sync_in_progress:0master_link_down_since_seconds:25slave_priority:100db1:keys=250000 Expires=0

You can see that the status of master_link_status is already down, and Master is no longer accessible.

At this point, Slave is still running well, and the AOF and RDB files are retained.

Next we will restore the data on the Master through the saved AOF and RDB files on Slave.

First of all, cancel the synchronization status on the Slave to prevent the master database from restarting before the data recovery is completed, thus directly overwriting the data on the slave database, resulting in the loss of all data.

Redis 127.0.0.1 6379 > SLAVEOF NO ONE

OK

Make sure that there is no configuration information related to master:

Redis 127.0.0.1 6379 > INFO

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version:4.4.5process_id:13003run_id:9b8b398fc63a26d160bf58df90cf437acce1d364uptime_in_seconds:1961uptime_in_days:0lru_clock:654215used_cpu_sys:29.98used_cpu_user:1.22used_cpu_sys_children:1.76used_cpu_user_children:1.42connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf: 0blocked_clients:0used_memory:33047696used_memory_human:31.52Mused_memory_rss:34779136used_memory_peak:33064400used_memory_peak_human:31.53Mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_since_last_save:0bgsave_in_progress:0last_save_time:1348719252bgrewriteaof_in_progress:0total_connections_received:4total_commands_processed:250311expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_channels: 0pubsub_patterns:0latest_fork_usec:1119vm_enabled:0role:masteraof_current_size:17908619aof_base_size:16787337aof_pending_rewrite:0aof_buffer_length:0aof_pending_bio_fsync:0db1:keys=250000 Expires=0

Copy the data file on Slave:

Dongguo@redis-slave:/opt/redis/data/6379$ tar cvf / home/dongguo/data.tar *

Appendonly.aof

Dump.rdb

Upload data.tar to Master and try to recover the data:

You can see that there is a data file initializing Slave in the Master directory, which is very small. Delete it.

Dongguo@redis:/opt/redis/data/6379$ ls-l

Total 4

-rw-r--r-- 1 root root 10 Sep 27 00:40 dump.rdb

Dongguo@redis:/opt/redis/data/6379$ sudo rm-f dump.rdb

Then extract the data file:

Dongguo@redis:/opt/redis/data/6379$ sudo tar xf / home/dongguo/data.tar

Dongguo@redis:/opt/redis/data/6379$ ls-lh

Total 29M

-rw-r--r-- 1 root root 18m Sep 27 01:22 appendonly.aof

-rw-r--r-- 1 root root 12m Sep 27 01:22 dump.rdb

Start Redis on Master

Dongguo@redis:/opt/redis/data/6379$ sudo / etc/init.d/redis start

Starting Redis server...

Check to see if the data is restored:

Redis 127.0.0.1 6379 > INFO

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version:4.4.5process_id:16959run_id:6e5ba6c053583414e75353b283597ea404494926uptime_in_seconds:22uptime_in_days:0lru_clock:650292used_cpu_sys:0.18used_cpu_user:0.20used_cpu_sys_children:0.00used_cpu_user_children:0.00connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf: 0blocked_clients:0used_memory:33047216used_memory_human:31.52Mused_memory_rss:34623488used_memory_peak:33047192used_memory_peak_human:31.52Mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:0changes_since_last_save:0bgsave_in_progress:0last_save_time:1348680180bgrewriteaof_in_progress:0total_connections_received:1total_commands_processed:1expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_channels: 0pubsub_patterns:0latest_fork_usec:0vm_enabled:0role:masterdb1:keys=250000 Expires=0

You can see that 250000 pieces of data have been fully restored to the Master.

At this point, you can safely restore the synchronization settings of Slave.

Redis 127.0.0.1 6379 > SLAVEOF 10.6.1.143 6379

OK

View synchronization status:

Redis 127.0.0.1 6379 > INFO

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version:4.4.5process_id:13003run_id:9b8b398fc63a26d160bf58df90cf437acce1d364uptime_in_seconds:2652uptime_in_days:0lru_clock:654284used_cpu_sys:30.01used_cpu_user:2.12used_cpu_sys_children:1.76used_cpu_user_children:1.42connected_clients:2connected_slaves:0client_longest_output_list:0client_biggest_input_buf: 0blocked_clients:0used_memory:33056288used_memory_human:31.52Mused_memory_rss:34766848used_memory_peak:33064400used_memory_peak_human:31.53Mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_since_last_save:0bgsave_in_progress:0last_save_time:1348719252bgrewriteaof_in_progress:1total_connections_received:6total_commands_processed:250313expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_channels: 0pubsub_patterns:0latest_fork_usec:12217vm_enabled:0role:slaveaof_current_size:17908619aof_base_size:16787337aof_pending_rewrite:0aof_buffer_length:0aof_pending_bio_fsync:0master_host:10.6.1.143master_port:6379master_link_status:upmaster_last_io_seconds_ago:0master_sync_in_progress:0slave_priority:100db1:keys=250000 Expires=0

The master_link_status is displayed as up and the synchronization status is normal.

In the process of this recovery, we copied both AOF and RDB files, so which file completed the data recovery?

In fact, when the Redis server hangs, the data will be restored to memory at the following priority when rebooted:

If only AOF is configured, load the AOF file to recover the data when restarting

If both RDB and AOF are configured, only the AOF file is loaded to recover the data.

If only RDB is configured, the startup will load the dump file to recover the data.

In other words, AOF takes precedence over RDB, which is understandable because AOF itself guarantees data integrity higher than RDB.

In this case, we secured the data and restored Master by enabling AOF and RDB on Slave.

However, in our current online environment, because the data has an expiration time, it is not practical to use AOF. Too frequent write operations will make the AOF file grow to an unusually large size, which greatly exceeds our actual amount of data, which will also lead to a lot of time spent in data recovery. Therefore, you can only enable Snapshot on Slave for localization, and consider increasing the frequency in save or calling a scheduled task to store snapshots of bgsave on a regular basis to ensure the integrity of localized data as much as possible.

Under such a framework, if only Master is down, Slave is complete and data recovery can reach 100%. Data recovery can also reach an acceptable level if Master and Slave are hung up at the same time.

At this point, I believe you have a deeper understanding of the "way of redis persistence", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.