Shulou(Shulou.com)06/01 Report--

I. the role of persistence

1. What is persistence?

All the data of redis is saved in memory, and updates to the data are saved asynchronously to the hard disk.

two。 The implementation of persistence

Snapshot: a completed backup of data at a certain time-mysql's Dump-redis RDB writes a log: any operation records the log, to recover the data, just go through the log again-mysql's Binlog-Hhase's HLog-Redis's AOF

II. RDB

1. What is RDB?

two。 Trigger mechanism-three main ways

The first kind: save (synchronization)

1 the client enters the save command-"redis server -" synchronously create RDB binaries.

2 will cause redis blocking (when the amount of data is very large)

3 File policy: if the old RDB exists, it will replace the old one.

4 complexity o (n)

The second kind: bgsave (Asynchronous, Backgroud saving started)

1 the client enters the save command-"redis server -" asynchronously create RDB binaries (the fork function generates a child process (fork blocks reids), executes createRDB, executes successfully, and returns the reids message)

2 if you access redis at this time, you will respond to the client normally.

3 File policy: same as save. If the old RDB exists, it will replace the old one.

4 complexity o (n)

Third: (commonly used) (*) automatically (through configuration files)

Configure seconds changes

Save 900 1save 300 10save 60 10000 if 1w pieces of data are changed in 60s, rdb is automatically generated

If 10 pieces of data are changed in 300s, rdb is automatically generated.

If a piece of data is changed in 900s, rdb is automatically generated.

If any of the above three items are met, rdb will be generated automatically and bgsave will be used internally.

# configuration:

Save 9001 # configure one

Save 30010 # configure one

Save 60 10000 # configure one

Name of the dbfilename dump.rdb # rdb file, default to dump.rdb

The dir. / # rdb file exists in the current directory

Stop-writes-on-bgsave-error yes # if there is an error in bgsave, whether to stop writing. Default is yes

Rdbcompression yes # is in compressed format

Does rdbchecksum yes # checksum the rdb file?

# Best configuration

Save 900 1

Save 300 10

Save 60 10000 dbfilename dump-$ {port} .rdb

# with the port number as the file name, there may be a lot of reids on a machine, so it won't be messy.

Dir / bigdiskpath # save path to a large hard disk location directory

Stop-writes-on-bgsave-error yes

# stop when an error occurs

Rdbcompression yes # Compression

Rdbchecksum yes # check

The RDB trigger mechanism generally uses the third way, but this approach also has its drawbacks. If the number of modified entries is not within the set range, then it will not be triggered, which will lead to a lot of cases where the data is not persisted. So we generally use the following approach: AOF.

RDB can be used to save unimportant data (such as caching data), and AOF can be used to save important data, but both methods can be used at the same time.

III. AOF

1.RDB problem

Time-consuming and performance-consuming. Out of control, data may be lost.

2.AOF introduction

Every time the client writes a command, it records a log and puts it in the log file. If there is an outage, the data can be fully restored.

Three Strategies of 3.AOF

The log is not written directly to the hard disk, but is first placed in the buffer, which is written to the hard disk according to some strategies.

# first: always:redis-- "write command flushed buffer -" each command fsync to the hard disk-"AOF file

# second: everysec (default): redis-- "buffer flushed by write command -" fsync the buffer to the hard disk per second-"AOF file

# third: no:redis-- "write command flushed buffer -" operating system decides, buffer fsync to hard disk-"AOF file

Command alwayseverysecno has the advantage of not losing data

Fsync once a second, never mind losing 1 second of data

Shortcoming

IO costs a lot of money, and the average Sata disk only has a few hundred TPS and can't control the data in one second.

4.AOF rewriting

With the gradual writing of the command and the increase of concurrency, the AOF file will become larger and larger. This problem can be solved by AOF rewriting.

Native AOFAOF rewriting

Set hello world

Set hello java

Set hello hehe

Incr counter

Ncr counter

Rpush mylist a

Rpush mylist b

Rpush mylist c

Expired data

Set hello hehe

Set counter 2

Rpush mylist a b c

The essence is to optimize out-of-date, useless, repetitive, optimizable commands, which can reduce disk usage and speed up recovery.

Mode of realization

Bgrewriteaof: the client sends a bgrewriteaof command to the server, and the server starts a fork process to complete the AOF rewrite

AOF rewrite configuration:

Rewrite the process

AOF profile (*)

Appendonly yes # set this option to yes, open appendfilename "appendonly-$ {port} .aof" # the saved name of the file appendfsync everysec # adopt the second strategy dir / bigdiskpath # storage path no-appendfsync-on-rewrite yes # when aof rewrite, whether to do aof append operation, because aof rewrite consumption performance, disk consumption, normal aof write disk has a certain conflict, during this period, data is allowed to be lost

IV. The choice of RDB and AOF

Comparison between 1.rdb and aof

Command rdbaof has a low startup priority

High (hang up and restart, the data of aof will be loaded)

Small in size

Big

Recovery speed

Fast and slow

Data security

Lose data

According to the strategy.

Light and heavy

Heavy

Light

The best strategy of 2.rdb

When rdb is turned off, when master-slave operation

Centralized management: backup data on a daily and hourly basis

Master-slave configuration, open from the node

The best strategy of 3.aof

On: caching and storage, open in most cases

Centralized management of aof rewriting

Everysec: through the strategy of refreshing per second

4. The best strategy

Small fragments: the maximum memory of each redis is 4G

Caching or storage: use an impassability policy depending on the nature

Monitor hard disk, memory, load network, etc.

Have enough memory

These are the details of how Redis implements persistence solutions (RDB and AOF usage). Please pay attention to other related articles for more!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.