Shulou(Shulou.com)06/01 Report--

This article mainly shows you the "sample Analysis of Redis Master-Slave Technology", which is easy to understand and clear. I hope it can help you solve your doubts. Let me lead you to study and learn this article "sample Analysis of Redis Master-Slave Technology".

Redis replication

In a production environment, Redis ensures no data loss (or a small amount of data loss) even if the server is restarted through persistence capabilities (RDB and AOF technologies). However, because the data is stored on a server, if this server has problems such as hard disk failure (encountered many times in the production environment), it will also lead to data loss, in order to avoid a single point of failure, the common practice is to copy multiple copies of the database to deploy on different servers, so that even if one server fails, other servers can still provide services as quickly as possible. To this end, Redis provides a replication function, which can automatically synchronize the updated data to other databases when the data in one database is updated.

In the concept of replication, databases are divided into two categories, one is the master database (master), the other is the slave database (slave). The master database can read and write, and automatically synchronize the data to the slave database when the write operation causes data changes. The slave database is generally read-only and accepts data synchronized from the master database. A master database can have multiple slave databases.

Redis replication is easy to use and is configured to allow replicas of slave Redis Servers or Master Servers. Here are a few very important features about redis replication:

A Master can have more than one Slaves.

Slaves can accept links from other slave, not only from slaves under the same master, but also from other slaves in the same structure diagram.

Redis replication is non-blocking in master segments, which means that master can also accept queries when performing synchronization on the same or more slave sides.

Replication is also non-blocking on the server. Suppose you configure redis in redis.conf. When slave is performing a new synchronization, it can still use the old data information to provide queries. Otherwise, you can configure that when redis slaves goes to master and loses contact, slave will send a client error.

In order to have multiple slaves to do read-only queries, replication can be repeated 2 or even more times, with scalability (for example, slaves conversations and repeated sorting operations, it is relatively simple to have multiple data redundancy).

By copying, you can avoid the consumption of master's full write hard disk: just configure master's configuration file redis.conf to "avoid saving" (comment out all "save" commands), and then connect to a slave to persist the data. But this ensures that masters does not restart automatically (read the next paragraph for more)

Redis replication configuration

Using the replication feature in Redis is very easy, as long as you add "slaveof master data replication master database port" to the configuration file of the slave database.

No configuration is required for the primary database. Let's take a look at the simplest replication system, where we start two redis examples on one server, listening on different ports, one as the master database and the other as the slave database.

First, let's start a redis instance as the primary database without any parameters:

$redis-server-port 6379 &

This instance listens to port 6379 by default, and then starts another redis instance as a slave database with the parameter slaveof, and lets it listen on port 6380:

$redis-server-- port 6380-- slaveof 127.0.0.1 6379 &

Check the startup status of the instance:

$ps aux | grep redisroot 2886 0.0 38652 4448 pts/0 Sl 16:57 0:00 redis-server *: 6379root 2889 0.0 36604 4368 pts/0 Sl 16:57 0:00 redis-server *: 6380

At this point, any data changes in the master database are automatically synchronized to the slave database, and we open redis-cli instance An and connect to the database:

$redis-cli-p 6379

Then open redis-cli instance B and connect to the slave database:

$redis-cli-p 6380

At this point, we use the INFO command to get information about replication in instance An and instance B, respectively.

127.0.0.1 purl 6379 > INfo replication# Replicationrole:masterconnected_slaves:1slave0:ip=127.0.0.1,port=6380,state=online,offset=266,lag=1master_repl_offset:266repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:2repl_backlog_histlen:265

As you can see, the role of instance A (role) is master, that is, the master database, and the number of connected slave databases (connectd_slaves) is 1.

The response information also obtained in instance B is as follows:

127.0.0.1 6380 > INfo replication# Replicationrole:slavemaster_host:127.0.0.1master_port:6379master_link_status:upmaster_last_io_seconds_ago:9master_sync_in_progress:0slave_repl_offset:378slave_priority:100slave_read_only:1connected_slaves:0master_repl_offset:0repl_backlog_active:0repl_backlog_size:1048576repl_backlog_first_byte_offset:0repl_backlog_histlen:0

You can see that the role of instance B is slave, the slave database, and the address of its master database is 127.0.0.1 and the port is 6379.

Then we use the SET command in instance A to set the value of a key:

127.0.0.1 purl 6379 > set foo barOK

At this point, you can get this value in instance B.

127.0.0.1 6380 > get foo "bar"

Prove that the replication capabilities of the two Redis instances are already available. By default, the slave database is read-only, and an error will occur if you modify the data from the database directly, as follows:

127.0.0.1 READONLY You can't write against a read only slave 6380 > set foo hey (error).

But you can also make the slave database writable by setting the slave-read-only=no in the configuration file of the slave database, but because any changes to the slave database will not be synchronized to any other database, and once the winning data in the master database is updated, the changes in the slave database will be overwritten, so usually the slave database should not be set writable to avoid potential application logic errors that are easy to be ignored.

The same is true for configuring multiple slave databases by adding the salveof parameter to the configuration file of all slave databases to point to the same master database. In addition to setting the slaveof parameter through the configuration file or command line parameters, you can also use the slaveof command to modify it at run time. Let's add another instance C (6381):

$redis-server-- port 6381 & $redis-cli-p 6381127.0.0.1 slaveof 6381 > get foo "bar"

If the database is already a slave to another master database, the slaveof command will stop synchronizing with the original database and synchronizing with the new database. In addition, for the slave database, you can also use the slaveof no one command to make the current database stop receiving the synchronization of other databases and convert to the master database. The following test shows that it is not allowed to write data from library instance C, and then use slaveof no one to convert this database to the primary database, and then write the data. It is generally used to switch the slave node to provide data operation service when the master node dies.

127.0.0.1 Connection with master lost.5493:M 6381 > set foo bar (error) READONLY You can't write against a read only slave.127.0.0.1:6381 > SLAVEOF no one5493:M 07 Aug 17 Aug 23 MASTER MODE enabled 06.792 # Connection with master lost.5493:M 07 Aug 17 MASTER MODE enabled 06.792 * Caching the disconnected master state.5493:M 07 Aug 17 23 MASTER MODE enabled (user request from 'id=2 addr=127.0. 0.1 Connection with slave 40825 fd=6 name= age=348 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=slaveof') 2886 Connection with slave 40825 lost.OK127.0.0.1:6381) 2886 Connection with slave 277 Aug 1715 23V 06.792 # Connection with slave 127.0.1 lost.OK127.0.0.1:6381 > set foo barOK

Copy common parameters

Slaveof

Use the current server as the slave and specify the master information for it.

Masterauth

Connect to master by authentication. If password Protection is used in master, slave must deliver the correct authorization password to connect successfully. The "requirepas" configuration item specifies the password for the current server. The value in this configuration item needs to be consistent with the "requirepas" of the master machine. This parameter is configured on the server.

Slave-serve-stale-data yes

If the current server is slave, whether to continue to provide services to the client when slave loses communication with master. "yes" means to continue, and "no" means to terminate. In the case of "yes", slave continues to provide read-only services to clients, and it is possible that the data has expired at this time. In the case of "no", any data request service sent to this server (including the client and the slave of this server) will be told "error".

Whether slave-read-only yesslave is read-only, it is strongly recommended that it be "yes". The interval (in seconds) at which repl-ping-slave-period 10slave sends ping messages to the specified master, default is 10. In the communication between repl-timeout 60slave and master, the maximum idle time is 60 seconds by default. A timeout will cause the connection to be closed. The connection between repl-disable-tcp-nodelay noslave and master, whether to disable the TCP nodelay option. " Yes "means disabled, then the data in socket communication will be sent in packet mode (the size of packet is limited by socket buffer), which can improve the efficiency of socket communication (the number of tcp interactions), but small data will be buffer, will not be sent immediately, and there may be a delay for the recipient." "no" means that if the tcp nodelay option is turned on, any data will be sent immediately, with better timeliness but less efficiency. Recommended as "no". Slave-priority 100 is suitable for the Sentinel module (unstable,M-S cluster management and monitoring) and requires additional profile support. The weight value of slave. Default is 100. When the master fails, Sentinel will find the slave with the lowest weight (> 0) from the slave list and promote it to master. If the weight value is 0, the slave is the Observer and does not participate in the master election. Understand the replication principle of Redis

Understanding the principle of Redis replication is of great help to the operation and maintenance of Redis process, including how to plan nodes, how to deal with node failures and so on. The following is the project for Redis to implement replication.

When a slave database is started, a SYNC command will be sent to the master database, and after receiving the SYNC command, the master database will start to save the snapshot in the background (that is, the process of RDB persistence), and cache the commands received during the snapshot. When the snapshot is completed, Redis Master will send the snapshot file to the slave database, and after receiving it from the database, it will load the snapshot file. Redis Master then sends all the accumulated content in the write command buffer to the slave server in the format of the Redis command protocol. The above process is called replication initialization. After the replication initialization ends, the master database will synchronize the command to the slave database whenever it receives a write command, thus ensuring the consistency of the master and slave database data.

You can verify the synchronization process yourself with the telnet command: first connect to a Redis server that is processing command requests, then send SYNC commands to it, and after a while, you will see that the telnet session receives large chunks of data (.rdb files) from the server, and then you will see that all write commands executed by the server are re-sent to the telnet session.

When the connection between the master and slave databases is disconnected and reconnected, Redis 2.6 and previous versions will reinitialize the replication (that is, the master database will re-save the snapshot and transfer it to the slave database). Even if only a few commands have not been received from the slave database, the master database must retransmit all the data in the database to the slave database. This makes the data recovery process after the disconnection of the master and slave database is very inefficient, especially when the network environment is not good. An important improvement of Redis version 2.8is that the disconnection reconnection can support conditional incremental data transmission. When the master database is reconnected to the master database, the master database only needs to transmit the commands executed during the disconnection to the slave database. Thus the practicability of Redis replication is greatly improved.

The implementation of incremental replication is based on the following three points:

1) the slave database will store the running ID (run id) of the master database, and each Redis running instance will have a unique running ID. Every time the instance is restarted, a new running ID will be automatically generated.

2) in the replication synchronization phase, every time the master database sends a command to the slave database, it will store the command in a backlog queue (backlog), and record the offset range of the commands stored in the current backlog queue.

3) at the same time, when a command from the master database is received from the database, the offset of the command is recorded.

These three points are the basis for incremental replication. When the master-slave connection is ready, the slave database sends a SYNC command to tell the master database that all data can be synchronized. After version 2.8, instead of sending SYNC commands, send PSYNC in the format of "the latest command offset before the running ID of the PSYNC master database was disconnected". After the primary database receives the PSYNC command, the following judgment is performed to determine whether incremental replication can be performed for this reconnection.

1) first of all, the master database will determine whether the running ID transmitted from the database is the same as its own running ID. The significance of this step is to ensure that the slave database is indeed synchronized with itself before, so as to avoid getting the wrong data from the database (for example, if the master database is restarted during disconnection, it will cause data inconsistency).

2) then determine whether the command offset of the last synchronization from the database is in the backlog queue, if so, you can perform an incremental copy and send the corresponding command in the backlog queue to the slave database.

If the reconnection does not meet the conditions of incremental replication, the master data will be synchronized once (that is, the process is the same as that of Redis 2.6. in most cases, the process of incremental replication is completely transparent to the developer, the developer does not need to care about the details of incremental replication, and the 2.8master database can be synchronized with the old slave database normally (by receiving SYNC commands). Similarly, the 2.8Slave database can also be synchronized with the old master database (by sending the SYNC command). The only thing the developer needs to set is the size of the backlog queue.

The backlog queue is essentially a fixed-length circular queue, and the size of the backlog queue is 1MB by default, which can be adjusted through the repl-backlog-size option of the profile. It is easy to understand that the larger the backlog queue, the longer it takes to disconnect the master-slave database. According to the network status between master and slave databases, it is important to set up a reasonable backlog queue. Because the content of the backlog queue is the command itself, such as SET FOO BAR, to estimate the size of the backlog queue only needs to estimate the size of the command that may be executed by the master-slave database in the time when the master-slave database is disconnected. Another configuration option related to the backlog queue is repl-backlog-ttl, which is how long it takes to free up the memory space of the backlog queue after all master and slave databases are disconnected from the master data. The default time is 1 hour.

Persistence from the database

Another relatively time-consuming operation is persistence. In order to improve performance, you can establish one or more slave databases through replication, enable persistence in the slave database, and disable persistence in the master database. When the slave database crashes and restarts, the master database will automatically synchronize the time, so there is no need to worry about data loss.

Then when the primary database crashes, the situation becomes a little more complicated. When recovering master database data manually from database data, you need to strictly follow the following two steps:

1) use the SLAVEOF NO ONE command in the slave database to promote the slave database to continue service for the master database.

2) start the previously crashed master database, and then use the SLAVEOF command to set it as the slave database of the new master database to synchronize the data back.

Note that when replication is turned on and persistence of the primary database is turned off, do not use supervisor and similar process management tools to automatically restart the primary database after it crashes. Similarly, when the server where the primary database is located is shut down for some reason, avoid a direct restart. This is because when the master database is restarted, all data in the database is emptied because persistence is not enabled, and the slave database will still receive data from the master database, so that all slave databases will also be emptied, causing the persistence of the slave database to be meaningless.

In either case, manual maintenance of slave database or master data restart and data recovery are relatively troublesome. Fortunately, Redis provides an automation scheme to implement this process, avoiding the trouble of manual maintenance and error-prone problems.

No hard disk replication

When introducing the working principle of Redis replication above, it is introduced that replication is based on the persistence of RDB mode, that is, the master database side saves the RDB snapshot in the background, and the snapshot file is received and loaded from the database side. This implementation can significantly simplify the logic and reuse the existing code, but the shortcomings are also obvious.

1) when the RDB snapshot is disabled in the master database (that is, the save statements in all configuration files are deleted), Redis will still generate a RDB snapshot if the replication initialization operation is performed, so the master database will restore data with this snapshot after the next startup. Because the timing of replication cannot be determined, the recovered data may be at any point in time.

2) because the RDB snapshot file needs to be created on the hard disk during replication initialization, this process can have an impact on performance if the hard disk performance is slow. For example, when using Redis as a caching system, the server's hard disk read and write speed may be poor because there is no need for persistence. However, when the cache system uses an one-master and multi-slave cluster architecture, every time it synchronizes with the slave database, Redis will execute a snapshot and read and write to the hard disk, resulting in performance degradation.

Therefore, starting from version 2.8.18, Redis introduced the option of no hard disk replication. When this option is turned on, Redis will not store the snapshot content on the hard disk when it is initialized with replication from the database, but will directly send it to the slave database through the network, avoiding the performance bottleneck of the hard disk. You can use the following configuration in the configuration file to enable this feature:

Repl-diskless-sync yes

PS: you can use the "SLAVEOF ON ONE" directive when you need to convert Slave to Master.

The above is all the contents of the article "sample Analysis of Redis Master-Slave Technology". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.