Shulou(Shulou.com)06/01 Report--

This article shows you how to achieve master-slave fault switching in Redis Sentinel mode, which is concise and easy to understand, and can definitely brighten your eyes. I hope you can gain something through the detailed introduction of this article.

Introduction

Redis Sentinel is a distributed system where you can run multiple Sentinel processes (progress) in a single architecture, which use the rumor protocol (gossip protocols) to receive information about whether the master server is offline, and use the voting protocol (agreement protocols) to decide whether to perform automatic failover and which slave server to choose as the new master server.

Although Redis Sentinel is released as a separate executable redis-sentinel, it is really just a Redis server running in a special mode, and you can start Redis Sentinel with the-- sentinel option when you start a normal Redis server.

The Sentinel system is used to manage multiple Redis servers (instance) and performs the following three tasks:

Monitoring (Monitoring): Sentinel will constantly check whether your master server and slave server are working properly.

Reminder (Notification): when there is a problem with a monitored Redis server, Sentinel can send notifications to administrators or other applications through API.

Automatic failover (Automatic failover): when a master server does not work properly, Sentinel starts an automatic failover operation, which upgrades one of the slaves of the failed master server to the new master server and allows the other slave servers of the failed master server to replicate the new master server When the client tries to connect to the failed primary server, the cluster also returns the address of the new primary server to the client, so that the cluster can use the new primary server instead of the failed server.

Redis version: 3.0.7

Master: 6379, sentinel:26379

From: 6380, sentinel:26380

Configuration

This chapter mainly introduces how to build an automatic failover reids cluster. When the master downtime takes over from the master to the new master, the master of the outage automatically becomes the slave after startup. In fact, it is the same as the dual master mode of Mysql; the redis cluster needs to use redis-sentinel programs and sentinel.conf configuration files.

Master configuration

Vim redis.conf

Daemonize yespidfile / usr/local/redis-6379/run/redis.pidport 6379tcp-backlog 128timeout 0tcp-keepalive 0loglevel noticelogfile "" databases 16save 900 1 # savesave 300 10save 60 10000stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename dump.rdb # dbfiledir "/ usr/local/redis-6379" masterauth "123456" requirepass "123456" slave-serve-stale-data yesslave-read-only yesrepl-diskless-sync norepl-diskless-sync-delay 5repl-disable-tcp-nodelay noslave-priority 100appendonly yesappendfilename "appendonly.aof" appendfsync Everysecno-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yeslua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency-monitor-threshold 0notify-keyspace-events "" hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer- Limit normal 0 0 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

Vim sentinel.conf

Cluster file matching

Port 26379dir "/ usr/local/redis-6379" # daemon mode daemonize yes# specifies the log file name logfile ". / sentinel.log" sentinel monitor mymaster 192.168.137.40 6379 1sentinel down-after-milliseconds mymaster 5000sentinel failover-timeout mymaster 18000sentinel auth-pass mymaster 123456

From configuration

Vim redis.conf

Daemonize yespidfile "/ usr/local/redis-6380/run/redis.pid" port 6380tcp-backlog 128timeout 0tcp-keepalive 0loglevel noticelogfile "databases 16save 900 1save 300 10save 60 10000stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename" dump.rdb "dir" / usr/local/redis-6380 "masterauth" 123456 "requirepass" 123456 "slave-serve-stale-data yesslave-read-only yesrepl-diskless-sync norepl-diskless-sync-delay 5repl-disable-tcp-nodelay noslave-priority 100appendonly yesappendfilename" appendonly.aof "appendfsync everysecno-appendfsync-on -rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yeslua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency-monitor-threshold 0notify-keyspace-events "" hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer-limit normal 00 0client -output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

Vim sentinel.conf

# sentinel port port 2638 working path, be careful not to repeat the master dir "/ usr/local/redis-6380" # daemon mode daemonize yes# specifies the log file name logfile ". / sentinel.log" # Sentinel monitored master, the master / slave configuration is the same: how long sentinel monitor mymaster 192.168.137.40 6379 "master or slave cannot be used (default 30 seconds) and then marked as s_down status. Sentinel down-after-milliseconds mymaster 500 if the sentinel fails to complete the failover operation within the configuration value (that is, the master/slave automatically switches over in case of failure), the failover is considered to have failed. Sentinel failover-timeout mymaster 1800 sets master and slaves authentication passwords sentinel auth-pass mymaster 123456

Start redis

Both the master and the subordinate should be activated.

Src/redis-server redis.conf

Start cluster monitoring

Both the master and the subordinate should be activated.

Src/redis-sentinel sentinel.conf-sentinel

Start error processing

Error 1:

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory=1' to / etc/sysctl.conf and then reboot or run the command' sysctl vm.overcommit_memory=1' for this to take effect.

Two solutions (overcommit_memory) 1. Echo "vm.overcommit_memory=1" > / etc/sysctl.conf or vi / etcsysctl.conf, and then reboot restarts the machine 2. Echo 1 > / proc/sys/vm/overcommit_memory takes effect without starting the machine. Overcommit_memory parameters description: set memory allocation policy (optional, set according to the actual situation of the server) / proc/sys/vm/overcommit_memory optional values: 0, 1, 2. 0, which means that the kernel will check whether there is enough memory available for the application process; if there is enough memory available, the memory request is allowed; otherwise, the memory request fails and the error is returned to the application process. 1, which means that the kernel allows all physical memory to be allocated, regardless of the current memory state. 2, indicates that the kernel allows the allocation of memory that exceeds the sum of all physical memory and swap space. Redis will fork a child process when dump data. Theoretically, the memory occupied by the child process is the same as that of parent. For example, if the memory occupied by parent is 8 GB, 8 GB of memory should also be allocated to child. If the memory cannot afford it, the downmachine or IO load of the redis server is often too high and the efficiency is reduced. So the optimized memory allocation policy here should be set to 1 (meaning that the kernel allows all physical memory to be allocated, regardless of the current memory state). Overcommit and OOM are also involved here. What is Overcommit and OOM in Unix, when a user process requests memory using the malloc () function, if the return value is NULL, the process knows that there is no available memory space and will process it accordingly. Many processes print error messages and exit. Linux uses a different approach, replying "yes" to most requests for memory so that it can run more and larger programs. Because memory will not be used immediately after applying for memory. This technique is called Overcommit. OOM killer (OOM=out-of-memory) occurs when there is insufficient memory. It chooses to kill some processes (user-mode processes, not kernel threads) to free memory. The strategy of Overcommit overcommit has three strategies (Documentation/vm/overcommit-accounting) under Linux: 0. Heuristic strategy. Reasonable overcommit will be accepted, unreasonable overcommit will be rejected. 1. Any overcommit will be accepted. two。 Commit is rejected when the system allocates more memory than the swap+N%* physical RAM (N% is determined by vm.overcommit_ratio). The policy for overcommit is set through vm.overcommit_memory. The percentage of overcommit is set by vm.overcommit_ratio. # echo 2 > / proc/sys/vm/overcommit_memory# echo 80 > / proc/sys/vm/overcommit_ratio when oom-killer occurs, linux chooses which processes to kill. The function that selects the process is the oom_badness function (in mm/oom_kill.c), which calculates the number of points for each process (0,1000). The higher the points, the more likely the process is to be killed. The number of points per process is related to oom_score_adj, and oom_score_adj can be set (- 1000 lowest, 1000 highest).

Error 2:

WARNING: The TCP backlog setting of 511 cannot be enforced because / proc/sys/net/core/somaxconn is set to the lower value of 128.

Echo 511 > / proc/sys/net/core/somaxconn

Error 3:

16433 Jun X 12 it was originally set to 14 52 it was originally set to 37.734 * Increased maximum number of open files to 10032.

The default value of the newly installed linux is only 1024. When the load is heavy, error: too many open filesulimit-a: use all the limit values vim / etc/security/limits.conf that can view the current system to add * soft nofile 65535 * hard nofile 65535 to the end of the file to execute su or re-close the connection and then execute ulimit-a to view the modified results.

Failover mechanism

1. After starting the cluster, the cluster program defaults to adding the configuration of the connection master in the redis file of the slave library.

# Generated by CONFIG REWRITEslaveof 192.168.137.40 6379

two。 After starting the cluster, the cluster program defaults to adding the cluster information to the master-slave sentinel.conf file

Lord:

Port 26379dir "/ usr/local/redis-6379" # daemon mode daemonize yes# specifies the log file name logfile ". / sentinel.log" sentinel monitor mymaster 192.168.137.40 6379 1sentinel down-after-milliseconds mymaster 5000sentinel failover-timeout mymaster 18000sentinel auth-pass mymaster 12345 Generated by CONFIG REWRITEsentinel config-epoch mymaster 0sentinel leader-epoch mymaster 1sentinel known-slave mymaster 192.168.137.40 6380sentinel known-sentinel mymaster 192.168.137.40 26380 c77c5f64aaad0137a228875e531c7127ceeb5c3fsentinel current-epoch 1

From:

# sentinel port port 2638 "working path dir" / usr/local/redis-6380 "# daemon mode daemonize yes# specifies the log file name logfile". / sentinel.log "# Sentinel monitored master, like the master-slave configuration, 6379 will become the current master port when switching between master and slave, and how long sentinel monitor mymaster 192.168.137.40 6379" master or slave cannot be used (default 30 seconds) is marked as s_down. Sentinel down-after-milliseconds mymaster 500 if the sentinel fails to complete the failover operation within the configuration value (that is, the master/slave automatically switches over in case of failure), the failover is considered to have failed. Sentinel failover-timeout mymaster 1800 sets master and slaves authentication password sentinel auth-pass mymaster 12345 Sentinel program automatically adds # Generated by CONFIG REWRITEsentinel config-epoch mymaster 0sentinel leader-epoch mymaster 1 slave # indicates the ip and port of the slave library of the current cluster, this value will change the sentinel known-slave mymaster 192.168.137.40 6380 Sentinel sentinel known-sentinel mymaster 192.168.137.40 26379 7a88891a6147e202a53601ca16a3d438e9d55c9dsentinel current-epoch 1 in addition to the current sentinel

Simulate the main fault

[root@monitor redis-6380] # ps-ef | grep redisroot 4171 10 14:20? 00:00:15 / usr/local/redis-6379/src/redis-server *: 6379 root 4175 10 14:20? 00:00:15 / usr/local/redis-6380/src/redis-server *: 6380 root 4305 10 15:28? 00:00:05 / usr/local/redis-6379/src/redis-sentinel *: 26379 [sentinel] root 4306 1 0 15:28? 00:00:05 / usr/local/redis-6380/src/redis-sentinel *: 26380 [sentinel] root 4337 4144 15:56 pts/1 00:00:00 grep redis [root@monitor redis-6380] # kill-9 4171 [root@monitor redis-6380] # ps-ef | grep redisroot 4175 10 14:20? 00:00:15 / usr/local/redis-6380/src/redis-server *: 6380 root 4305 10 15:28? 00:00:05 / usr/local/redis-6379/src/redis-sentinel *: 26379 [sentinel] root 4306 10 15:28? 00:00:05 / usr/local/redis-6380/src/redis-sentinel *: 26380 [sentinel] root 4339 4144 0 15:56 pts/1 00:00:00 grep redis [root@monitor redis-6380] #

From the Sentinel configuration file, you can see that the current main library has changed.

From the log file, you can also see that the current master has been converted from 6379 to 6380.

Official description of redis configuration file: h t tp svv / / r aw.g ithubusercon tent.com/antirez/redis/3.0/redis.conf

Sentinel ports 26379 and 26380 of redis cannot be connected by client software, but can be connected by programs. Client software can only connect ports 6379 and 6380 directly. The use of Sentinel Monitoring automatically switches over the slave when the master fails, and becomes the slave when the master starts. Seeing that others only configure single Sentinel 26379, this situation does not guarantee the high availability of the Sentinel program itself.

The above is how to achieve master-slave fault switching in Redis Sentinel mode. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.