Shulou(Shulou.com)06/01 Report--

Editor to share with you how many ways Redis provides for persistence, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's learn about it!

Redis provides two ways for persistence:

RDB: snapshots of your data can be stored at specified intervals.

AOF: records each write to the server and reexecutes these commands to restore the original data when the server is restarted.

Persistent configuration

In order to use persistence, we need to know how to turn on persistence.

Persistence configuration of RDB # time policy save 900 1save 300 10save 60 1000 records file name dbfilename dump.rdb# file save path dir / home/work/app/redis/data/# if there is a persistence error, whether the main process stops writing stop-writes-on-bgsave-error yes#, whether the rdbcompression yes# is compressed, whether the rdbchecksum yes is checked during import

The configuration is actually very simple, and here's what the persistence time strategy means.

Save 9001 means that if there is a write command within 900s, a snapshot is triggered, which can be understood as a backup.

Save 30010 means that there are 10 writes within 300s, resulting in a snapshot

The following is similar, so why do you need to configure so many rules? Because the read and write requests for each period of time in Redis are definitely not balanced, in order to balance performance and data security, we are free to customize when the backup is triggered. So here is a reasonable configuration based on your own Redis writes.

The stop-writes-on-bgsave-error yes configuration is also a very important configuration, which is that when the backup process goes wrong, the main process stops accepting new writes in order to protect persistent data consistency. If your business has a perfect monitoring system, you can disable this configuration, otherwise, please turn it on.

It is not recommended to enable rdbcompression yes for compression. After all, Redis itself is a CPU-intensive server. Enabling compression will lead to more CPU consumption, and CPU is more valuable than hard disk cost.

Of course, if you want to disable RDB configuration, it's also very easy, just write: save "" on the last line of save.

Configuration of AOF # whether to enable aofappendonly yes# file name appendfilename "appendonly.aof" # whether to synchronize no-appendfsync-on-rewrite no# rewriting during appendfsync everysec# aof rewriting triggers configuration auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb# loading aof if there is an error, how to handle aof-load-truncated yes# file rewriting policy aof-rewrite-incremental-fsync yes

Or focus on explaining some key configurations:

Appendfsync everysec actually has three modes:

Always: synchronize every write command to aof immediately, which is slow but secure

Everysec: synchronizing once per second is a compromise

No:redis will not handle it and let OS handle it. It is very fast, but it is also the least secure.

In general, everysec configuration is used, which can take into account both speed and security, and lose up to 1 second of data.

Aof-load-truncated yes if the configuration is enabled, it is found that the aof tail is incorrect when loading. Yes, a log will be written to the client, but execution will continue. If set to no, the error will stop and must be repaired before it can be reloaded.

working principle

As for the principle part, we mainly look at how RDB and AOF achieve persistence, and how their process is.

Before introducing the principle, let's talk about the scheduled task mechanism within Redis. The frequency of scheduled task execution can be set through hz 10 in the configuration file (this configuration means 10 times in 1 s, that is, a scheduled task is triggered once per 100ms). The maximum value can be set to: 500, but it is not recommended to exceed: 100, because a higher value means more frequent execution, which results in more CPU consumption, which affects the read and write performance of the main process.

Scheduled tasks use TimeEvent implemented by Redis itself, which periodically invokes commands to complete scheduled tasks that may block the main process and degrade Redis performance. Therefore, when we configure Redis, we must consider some configurations that trigger scheduled tasks as a whole, and adjust them according to the actual situation.

The principle of RDB

There are two kinds of triggers for RDB persistence in Redis: manual triggering and Redis timing triggering.

For persistence in RDB mode, manual triggers can be used:

Save: will block the current Redis server until the persistence is complete, which should be banned online.

Bgsave: this trigger will fork a child process, which will be responsible for the persistence process, so blocking will only occur when the token child process.

The scenarios that are automatically triggered mainly include the following:

Trigger automatically according to our save m n configuration rules

When the slave node replicates in full, the master node sends the rdb file to the slave node to complete the copy operation, and the master node will trigger bgsave

When performing debug reload

When shutdown is executed, it will also be triggered if aof is not enabled.

Since save is rarely used, let's focus on how the bgsave command does the persistence of RDB.

It is noted here that fork operations can block, resulting in poor Redis read and write performance. We can control the maximum memory of a single Redis instance to minimize the event consumption of Redis during fork. And the frequency of automatic trigger mentioned above to reduce the number of fork, or use manual trigger, according to their own mechanism to complete the persistence.

The principle of AOF

Generally speaking, the whole process of AOF can be divided into two steps, one is the real-time writing of commands (in the case of appendfsync everysec configuration, there will be 1s loss), and the second step is to rewrite the aof file.

The main process for incrementally appending to a file is: command write = "append to aof_buf =" synchronize to the aof disk. So why write buf before synchronizing to disk first? If you write to disk in real time, it will result in very high disk IO, which will affect the overall performance.

Aof rewriting is to reduce the size of the aof file. It can be triggered manually or automatically. For the rules of automatic trigger, please see the configuration section above. The operation of the fork also occurs in the rewrite step, which also blocks the main process.

Manual trigger: bgrewriteaof. Automatic trigger is triggered according to configuration rules. Of course, the overall time of automatic trigger is also related to the frequency of scheduled tasks in Redis.

Let's take a look at a flowchart that is rewritten:

There are four key points to add to the above figure:

During the rewrite, because the main process is still responding to the command, in order to ensure the integrity of the final backup; so it will still be written to the old AOF file, if the rewrite fails, it can ensure that the data is not lost.

In order to write the write information of the response during the rewrite to the new file, a buf is also reserved for the child process to prevent the newly written file from losing data.

Rewriting is to generate the corresponding commands directly from the current memory data, and there is no need to read the old AOF files for analysis and command merging.

AOF files directly use the text protocol, mainly good compatibility, easy to add, high readability can be regarded as modification and repair.

Both RDB and AOF can write a temporary file first, and then replace the file through rename.

Recover data from persistence

After the backup and persistence of the data are done, how can we recover the data from these persistent files? If there are both RDB and AOF files on a server, who should load them?

To recover data from these files, all you need to do is restart Redis. Let's use the diagram to understand the process:

At startup, the AOF file is checked to see if it exists, and if not, an attempt is made to load RDB. So why load AOF first? Because the data saved by AOF is more complete, through the above analysis, we know that AOF basically loses up to 1 second of data.

Performance and practice

From the above analysis, we all know that snapshots of RDB and rewriting of AOF require fork, which is a heavyweight operation that can block Redis. Therefore, in order not to affect the response of the Redis main process, we need to reduce blocking as much as possible.

Reduce the frequency of fork, for example, you can manually trigger RDB to generate snapshots and rewrite with AOF

Control the maximum memory usage of Redis to prevent fork from taking too long

Use more powerful hardware

Reasonably configure the memory allocation policy of Linux to avoid fork failure due to insufficient physical memory.

What exactly are we supposed to do online? I provide some practical experience of my own.

If the data in Redis is not particularly sensitive or the generated data can be rewritten in other ways, persistence can be turned off, and if the data is lost, it can be recovered in other ways.

Make your own policy to check Redis periodically, and then you can manually trigger backups and rewrite data.

If multiple instances are deployed on a stand-alone machine, it is necessary to prevent multiple machines from running persistence and rewrite operations at the same time, prevent competition for memory, CPU and IO resources, and make persistence serial.

You can join the master-slave machine, use a slave machine for backup processing, and other machines normally respond to client commands.

RDB persistence and AOF persistence can exist together and can be used together.

These are all the contents of the article "how many ways does Redis provide for persistence". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.