Shulou(Shulou.com)06/01 Report--

This paper mainly introduces the principle, configuration and basic operation of Redis replication master-slave replication.

Mainly refer to the official documents:

Https://redis.io/topics/replication

Http://redisdoc.com/topic/replication.html

one。 Principle

This system works using three main mechanisms:

There are three main mechanisms of the system:

1. When a master and a slave instances are well-connected, the master keeps the slave updated by sending a stream of commands to the slave, in order to replicate the effects on the dataset happening in the master side due to: client writes, keys expired or evicted, any other action changing the master dataset.

When the master and slave instances are well connected, master updates the slave by sending a command stream from slave

2. When the link between the master and the slave breaks, for network issues or because a timeout is sensed in the master or the slave, the slave reconnects and attempts to proceed with a partial resynchronization: it means that it will try to just obtain the part of the stream of commands it missed during the disconnection.

When the master and slave connections are interrupted, due to a network failure or perceived master or slave timeout. Slave reconnects and attempts to resynchronize: it means getting the command stream lost when the connection is lost

3. When a partial resynchronization is not possible, the slave will ask for a full resynchronization. This will involve a more complex process in which the master needs to create a snapshot of all its data, send it to the slave, and then continue sending the stream of commands as the dataset changes.

When partial resynchronization (partial resynchronization) is not possible, slave will require a full resynchronization (full resynchronization). It will contain a complex process in which master needs to create a snapshot of all the data, send it to slave, and then continue to send the command stream to the changed data.

Data security concept

Persistent storage of Redis mainly provides two ways: RDB and AOF

Http://redisdoc.com/topic/persistence.html

1. RDB (Snapshot) mode

Is the default, that is, save the data at a certain time to the hard disk, if the server crash before the next snapshot is triggered, the data modified after the last snapshot will be lost.

Main configuration redis.conf parameters:

Save

Stop-writes-on-bgsave-error yes

Rdbcompression yes

Dbfilename dump.rdb

Dir. /

Parameter description:

Save: persistence is triggered if key changes at least X times in X seconds. For example, save 9001 triggers persistence if key changes at least once in 900s. If you want to turn off this feature, you can comment or delete all save lines or use save "".

Stop-writes-on-bgsave-error: whether to stop persistence when bgsave encounters error. The default is yes for yes and no for not.

Rdbcompression: whether to compress or not. The default is yes for yes and no for no. If you want to save CPU, set it to no, but the rdb file will be larger.

Dbfilename: the name of the persistent file. Default is dump.rdb.

Dir: persistent directory name. Default is the directory where redis.conf resides. /

2.AOF (append-only file) mode

It needs to be turned on manually. After it is turned on, the recorded modification operations will be saved to the hard disk by default every second.

Main configuration redis.conf parameters:

Appendonly yes

Appendfilename "appendonly.aof"

Appendfsync everysec

Dir. /

# the following rewrite parameters. AOF mode records all write commands to the hard disk in append mode, and the file will become larger and larger. To alleviate this problem, redis uses BGREWRITEAOF to delete redundant write commands, and deletes old AOF files similar to BGSAVE,rewrite.

# # #

No-appendfsync-on-rewrite no

Auto-aof-rewrite-percentage 100

Auto-aof-rewrite-min-size 64mb

Parameter description:

Appendonly: whether to enable aof. Default is no, which means that it is not enabled.

The file name of appendfilename:aof. Default is appendonly.aof.

Appendfsync: the interval between triggers. By default, everysec represents every second. You can also use always to represent all changes triggered. The performance is poor, but the data is the most secure. No represents let OS decide when to execute. The performance is the best, but the data is not secure.

Dir: the name of the persisted directory. Default is the directory where redis.conf resides. /

No-appendfsync-on-rewrite: whether appendfsync is paused when executing rewrite. Default no means no pause.

Auto-aof-rewrite-percentage:aof rewrite trigger time. Rewrite is triggered when the current aof file size exceeds the percentage of the aof file after the last rewrite. For example, the current aof file will not be rewrite again until it exceeds 2 times that of the aof file after the last rewrite.

Auto-aof-rewrite-min-size aof file overrides the minimum file size, that is, the initial aof file must reach this file before it is triggered, and each subsequent rewrite will not be based on this variable (based on the size after the last rewrite). This variable is valid only for initializing startup redis. If it is a redis restore, the lastSize is equal to the initial aof file size.

Data Security of Replication

In settings that use redis replication, it is strongly recommended to turn on persistence in the master and slave servers. If it cannot be turned on, for example, due to latency problems caused by very slow disk speed, you should avoid automatically restarting the service after restarting OS to avoid data loss. Officials have given examples of this:

To better understand why masters with persistence turned off configured to auto restart are dangerous, check the following failure mode where data is wiped from the master and all its slaves:

1. We have a setup with node An acting as master, with persistence turned down, and nodes B and C replicating from node A.

2. Node A crashes, however it has some auto-restart system, that restarts the process. However since persistence is turned off, the node restarts with an empty data set.

3. Nodes B and C will replicate from node A, which is empty, so they'll effectively destroy their copy of the data.

How the replication function works

Whether you connect for the first time or reconnect, when a slave server is established, the slave server sends a SYNC command to the master server.

The master server that receives the SYNC command will start executing BGSAVE and save all newly executed write commands to a buffer during the save operation.

When the BGSAVE is finished, the master server sends the .rdb file obtained from the save operation to the slave server, receives the .rdb file from the server, and loads the data in the file into memory.

The master server then sends all the accumulated contents in the write command buffer to the slave server in the format of the Redis command protocol.

Even if there are multiple slave servers sending SYNC to the master server at the same time, the master server only needs to execute the BGSAVE command once to process all of these slave server synchronization requests.

The slave server can reconnect automatically when the connection between the master and slave servers is disconnected. Before Redis 2.8, the reconnected slave server always performed a full resynchronization (full resynchronization) operation, but since Redis 2.8, the slave server can choose to perform a full resynchronization or partial resynchronization (partial resynchronization) according to the situation of the master server.

Partial resynchronization

Starting with Redis 2.8, after a temporary failure of the network connection, the master-slave server can try to continue the original replication process (process) without having to perform a full resynchronization operation.

This feature requires the master server to create a memory buffer (in-memory backlog) for the replication stream being sent, and a replication offset (replication offset) and a master server ID (master run id) are recorded between the master server and all slave servers. When the network connection is disconnected, the slave server reconnects and requests the master server to continue with the original replication process:

If the master server ID recorded from the slave server is the same as the ID of the current master server to be connected, and the data specified by the offset recorded from the slave server is still stored in the replication stream buffer of the master server, then the master server will send the missing data to the slave server when it is disconnected, and the replication can continue.

Otherwise, a full resynchronization operation is performed from the server.

This partial resynchronization feature of Redis 2.8 uses a new PSYNC master_run_id offset internal command, whereas previous versions of Redis 2.8 only had the SYNC command, but as long as the slave server is Redis 2.8 or later, it will decide whether to use PSYNC master_run_id offset or SYNC based on the version of the master server:

If the master server is Redis 2.8or later, the slave server uses the PSYNC master_run_id offset command to synchronize.

If the master server is a version prior to Redis 2.8, the slave server uses the SYNC command to synchronize.

two。 Specific configuration of replication

Lab: configure two servers on one machine, master 6379 slave 6380

Version: redis_version:3.0.6

Master: use default values except appendonly to change to yes

/ usr/local/redis/etc/redis.conf

Appendonly yes

Because to do slave testing on the same server, you need to copy a configuration file and change the port number, pid file name, dump.rdb name, and aof file name respectively.

Slave: / usr/local/redis/etc/redis.conf.6380

Pidfile / var/run/redis6380.pid

Port 6380

Dbfilename dump6380.rdb

Appendfilename "appendonly6380.aof"

Appendonly yes

Slaveof 127.0.0.1 6379

Note: starting with Redis 2.6, slave supports read-only mode and is the default mode for slave.

# Since Redis 2.6 by default slaves are read-only.

Slave-read-only yes

Turn on slave

# redis-server / usr/local/redis/etc/redis.conf.6380

View statu

Execute in master:

[root@cloud ~] # redis-cli

127.0.0.1 purl 6379 > info replication

# Replication

Role:master

Connected_slaves:1

Slave0:ip=127.0.0.1,port=6380,state=online,offset=57,lag=0

Master_repl_offset:57

Repl_backlog_active:1

Repl_backlog_size:1048576

Repl_backlog_first_byte_offset:2

Repl_backlog_histlen:56

127.0.0.1 purl 6379 > role

1) "master"-current server role

2) (integer) 71-current server replication offset

3) 1) 1) "127.0.0.1"-subordinate slave IP

2) "6380"-subordinate slave port

3) "71"-subordinate slave replication offset

Execute in slave:

127.0.0.1 6380 > info replication

# Replication

Role:slave

Master_host:127.0.0.1

Master_port:6379

Master_link_status:up

Master_last_io_seconds_ago:5

Master_sync_in_progress:0

Slave_repl_offset:83567

Slave_priority:100

Slave_read_only:1

Connected_slaves:0

Master_repl_offset:0

Repl_backlog_active:0

Repl_backlog_size:1048576

Repl_backlog_first_byte_offset:0

Repl_backlog_histlen:0

127.0.0.1 6380 > role

1) "slave"-current server role

2) "127.0.0.1"-upper master IP

3) (integer) 6379-Upper master port

4) "connected"

5) (integer) 83581-replica offset currently received from master

Above, master-slave configuration is complete.

Note: because Redis uses asynchronous replication, write data sent by master is not necessarily received by slave. Therefore, the possibility of data loss still exists.

To further ensure data security, the master server can be set to perform write operations only if there are at least N slave servers

Here is how this feature works:

The slave server PING the master server once per second and reports on the processing of the replication stream.

The master server records the last time each slave server sent PING to it.

Users can specify the maximum network delay min-slaves-max-lag through configuration

And at least the number of slave servers required to perform the write operation min-slaves-to-write.

Slave is enabled as master

Executing the command SLAVEOF NO ONE on the slave server will cause the secondary server to turn off the replication function and change from the slave server to the master server so that the original synchronized dataset will not be discarded.

Taking advantage of the feature that "SLAVEOF NO ONE will not discard the synchronized dataset", the secondary server can be used as the new master server when the primary server fails, thus achieving uninterrupted operation.

Above, the principle and the simplest configuration of Redis replication master-slave replication are introduced, and Sentinel,Redis-Cluster and other schemes will be sorted out later.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.