Shulou(Shulou.com)06/01 Report--

How does Redis Sentinel mode switch between master and slave failures? I believe that many novice rookies have not yet learned this skill. Through the summary of this article, I hope you can learn this skill. The following materials are the steps for implementation.

Redis Sentinel is a distributed system where you can run multiple Sentinel processes (progress) in a single architecture, which use the rumor protocol (gossip protocols) to receive information about whether the master server is offline, and use the voting protocol (agreement protocols) to decide whether to perform automatic failover and which slave server to choose as the new master server.

Although Redis Sentinel is released as a separate executable redis-sentinel, it is really just a Redis server running in a special mode, and you can start Redis Sentinel with the-- sentinel option when you start a normal Redis server.

The Sentinel system is used to manage multiple Redis servers (instance) and performs the following three tasks:

1. Monitoring: Sentinel will constantly check whether your master server and slave server are working properly.

2. Notification: when there is a problem with a monitored Redis server, Sentinel can send a notification to the administrator or other applications through API.

3. Automatic failover (Automatic failover): when a master server does not work properly, Sentinel starts an automatic failover operation, which upgrades one of the slaves of the failed master server to the new master server and allows the other slave servers of the failed master server to replicate the new master server. When the client tries to connect to the failed primary server, the cluster also returns the address of the new primary server to the client, so that the cluster can use the new primary server instead of the failed server.

Configuration

When the master downtime replaces the master as the new master, the master automatically becomes the slave after startup. In fact, it is the same as the dual master mode of Mysql; redis Sentinel needs to use redis-sentinel program and sentinel.conf configuration file.

Mkdir-p / usr/local/redismkdir-p / usr/local/redis/6379mkdir-p / usr/local/redis/6380mkdir-p / usr/local/redis/redis_cluster

Master configuration

Vim redis_6379.conf

Daemonize yespidfile / usr/local/redis/6379/redis_6379.pidport 6379tcp-backlog 128timeout 0tcp-keepalive 0loglevel noticelogfile "" databases 16save 900 1 # savesave 300 10save 60 10000stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename dump.rdb # dbfiledir "/ usr/local/redis/6379" masterauth "123456" requirepass "123456" slave-serve-stale-data yesslave-read-only yesrepl-diskless-sync norepl-diskless-sync-delay 5repl-disable-tcp-nodelay noslave-priority 100appendonly yesappendfilename "appendonly.aof" appendfsync Everysecno-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yeslua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency-monitor-threshold 0notify-keyspace-events "" hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer- Limit normal 0 0 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

Vim sentinel_1.conf

Sentinel file configuration

Port 6000dir "/ usr/local/redis/sentinel" # daemon mode daemonize yesprotected-mode nologfile "/ usr/local/sentinel/sentinel.log"

From configuration

Vim redis_6380.conf

Daemonize yespidfile "/ usr/local/redis/6380/redis_6380.pid" port 6380tcp-backlog 128timeout 0tcp-keepalive 0loglevel noticelogfile "databases 16save 900 1save 300 10save 60 10000stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename" dump.rdb "dir" / usr/local/redis/6380 "masterauth" 123456 "requirepass" 123456 "slave-serve-stale-data yesslave-read-only yesrepl-diskless-sync norepl-diskless-sync-delay 5repl-disable-tcp-nodelay noslave-priority 100appendonly yesappendfilename" appendonly.aof "appendfsync everysecno-appendfsync-on -rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yeslua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency-monitor-threshold 0notify-keyspace-events "" hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer-limit normal 00 0client -output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

Vim sentinel_2.conf

# sentinel port port 600 working path, be careful not to repeat the dir "/ usr/local/sentinel" # daemon mode daemonize yesprotected-mode no# indicates the log file name logfile "/ usr/local/sentinel/sentinel.log"

Note:

1. The application connects to the Sentinel port and connects to the specific master copy by specifying a different master name.

two。 In the Sentinel configuration file, you only need to configure the master copy ip and port in the master-slave replica. When the master-slave switch occurs, the Sentinel will automatically modify the master copy ip in the Sentinel configuration file to the new master copy ip.

3. A Sentinel profile can be configured to monitor multiple master-slave replication at the same time.

4. A single sentry can be used for master / slave fault monitoring, but if there is only one sentinel process, if there is an error or network congestion, then the master / slave handover of the redis cluster will not be realized (single point problem). This 2 represents the number of votes. When 2 sentinel thinks that a master is no longer available, failover will be triggered to truly believe that the master is no longer available. (the sentinel in the sentinel cluster also communicate with each other through the gossip protocol); so a reasonable configuration should be to start multiple sentinel processes at the same time, preferably on different servers.

5. Note that the need for mymaster is unique throughout the network environment, and the Sentinels will automatically establish relationships through mastername as long as the network environment is connected.

Start redis

1. Both the master and the subordinate should be activated.

Src/redis-server redis.conf

two。 Log in to 6380 to establish a master-slave relationship

Redis-cli-p 6380slaveof 192.168.137.40 6379

Deploy sentinels

Both master and slave sentinels need to be activated, and they can also be started by redis-server, such as "redis-server sentinel.conf-- sentinel".

1. Activate the sentinel.

Src/redis-sentinel sentinel.conf

two。 Log in to the sentry (both sentinels need to log in for execution) and add master-slave monitoring information

Redis-cli-p 6000

Sentinel monitor mymaster 192.168.137.40 6379 2sentinel set mymaster down-after-milliseconds 5000sentinel set mymaster failover-timeout 15000sentinel set mymaster auth-pass 123456

Start error processing

Error 1:

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory=1' to / etc/sysctl.conf and then reboot or run the command' sysctl vm.overcommit_memory=1' for this to take effect.

Two solutions (overcommit_memory)

1. Echo "vm.overcommit_memory=1" > / etc/sysctl.conf or vi / etcsysctl.conf, and then restart the machine by reboot

2. Echo 1 > / proc/sys/vm/overcommit_memory takes effect without starting the machine.

Overcommit_memory parameter description:

Set memory allocation policy (optional, according to the actual situation of the server)

/ proc/sys/vm/overcommit_memory

Optional values: 0, 1, 2.

0, which means that the kernel will check whether there is enough memory available for the application process; if there is enough memory available, the memory request is allowed; otherwise, the memory request fails and the error is returned to the application process.

1, which means that the kernel allows all physical memory to be allocated, regardless of the current memory state.

2, indicating that the kernel allows more than the sum of all physical memory and swap space to be allocated

Note: when redis uses dump data, it will fork a child process. Theoretically, the memory occupied by the child process is the same as that of parent. For example, the memory occupied by parent is 8 GB. At this time, 8 GB of memory should also be allocated to child. If the memory is not affordable, it will often cause the downmachine or IO load of the redis server to be too high and reduce efficiency. So the optimized memory allocation policy here should be set to 1 (meaning that the kernel allows all physical memory to be allocated, regardless of the current memory state).

Overcommit and OOM are also involved here.

What are Overcommit and OOM?

In Unix, when a user process requests memory using the malloc () function, if the return value is NULL, the process knows that there is no available memory space and will process it accordingly. Many processes print error messages and exit.

Linux uses a different approach, replying "yes" to most requests for memory so that it can run more and larger programs. Because memory will not be used immediately after applying for memory. This technique is called Overcommit.

OOM killer (OOM=out-of-memory) occurs when there is insufficient memory. It chooses to kill some processes (user-mode processes, not kernel threads) to free memory.

Strategy of Overcommit

There are three strategies for overcommit under Linux (Documentation/vm/overcommit-accounting):

0. Heuristic strategy. Reasonable overcommit will be accepted, unreasonable overcommit will be rejected.

1. Any overcommit will be accepted.

two。 Commit is rejected when the system allocates more memory than the swap+N%* physical RAM (N% is determined by vm.overcommit_ratio).

The policy for overcommit is set through vm.overcommit_memory.

The percentage of overcommit is set by vm.overcommit_ratio.

# echo 2 > / proc/sys/vm/overcommit_memory

# echo 80 > / proc/sys/vm/overcommit_ratio

When oom-killer occurs, linux chooses which processes to kill

The function for selecting a process is the oom_badness function (in mm/oom_kill.c), which calculates the number of points for each process (0,1000).

The higher the points, the more likely the process is to be killed.

The number of points per process is related to oom_score_adj, and oom_score_adj can be set (- 1000 lowest, 1000 highest).

Error 2:

WARNING: The TCP backlog setting of 511 cannot be enforced because / proc/sys/net/core/somaxconn is set to the lower value of 128.

Echo 511 > / proc/sys/net/core/somaxconn

Error 3:

16433 Jun X 12 it was originally set to 14 52 it was originally set to 37.734 * Increased maximum number of open files to 10032.

The default value of the newly installed linux is only 1024. When the load is heavy, error: too many open files often appears.

Ulimit-a: use all the limit values that you can view the current system

Vim / etc/security/limits.conf

Add at the end of the file

* soft nofile 65535 * hard nofile 65535

Execute su or re-close the connection and then execute ulimit-a to view the modified results.

Failover mechanism

1. After starting the cluster, the cluster program defaults to adding the configuration of the connection master in the redis file of the slave library.

# Generated by CONFIG REWRITEslaveof 192.168.137.40 6379

two。 After starting the cluster, the cluster program defaults to adding the cluster information to the master-slave sentinel.conf file

Lord:

Port 26379dir "/ usr/local/redis-6379" # daemon mode daemonize yes# specifies the log file name logfile ". / sentinel.log" sentinel monitor mymaster 192.168.137.40 6379 1sentinel down-after-milliseconds mymaster 5000sentinel failover-timeout mymaster 18000sentinel auth-pass mymaster 12345 Generated by CONFIG REWRITEsentinel config-epoch mymaster 0sentinel leader-epoch mymaster 1sentinel known-slave mymaster 192.168.137.40 6380sentinel known-sentinel mymaster 192.168.137.40 26380 c77c5f64aaad0137a228875e531c7127ceeb5c3fsentinel current-epoch 1

From:

# sentinel port port 2638 "working path dir" / usr/local/redis-6380 "# daemon mode daemonize yes# specifies the log file name logfile". / sentinel.log "# Sentinel monitored master, like the master-slave configuration, 6379 will become the current master port when switching between master and slave, and how long sentinel monitor mymaster 192.168.137.40 6379" master or slave cannot be used (default 30 seconds) is marked as s_down. Sentinel down-after-milliseconds mymaster 500 if the sentinel fails to complete the failover operation within the configuration value (that is, the master/slave automatically switches over in case of failure), the failover is considered to have failed. Sentinel failover-timeout mymaster 1800 sets master and slaves authentication password sentinel auth-pass mymaster 12345 Sentinel program automatically adds # Generated by CONFIG REWRITEsentinel config-epoch mymaster 0sentinel leader-epoch mymaster 1 slave # indicates the ip and port of the slave library of the current cluster, this value will change the sentinel known-slave mymaster 192.168.137.40 6380 Sentinel sentinel known-sentinel mymaster 192.168.137.40 26379 7a88891a6147e202a53601ca16a3d438e9d55c9dsentinel current-epoch 1 in addition to the current sentinel

Simulate the main fault

[root@monitor redis-6380] # ps-ef | grep redisroot 4171 10 14:20? 00:00:15 / usr/local/redis-6379/src/redis-server *: 6379 root 4175 10 14:20? 00:00:15 / usr/local/redis-6380/src/redis-server *: 6380 root 4305 10 15:28? 00:00:05 / usr/local/redis-6379/src/redis-sentinel *: 26379 [sentinel] root 4306 1 0 15:28? 00:00:05 / usr/local/redis-6380/src/redis-sentinel *: 26380 [sentinel] root 4337 4144 15:56 pts/1 00:00:00 grep redis [root@monitor redis-6380] # kill-9 4171 [root@monitor redis-6380] # ps-ef | grep redisroot 4175 10 14:20? 00:00:15 / usr/local/redis-6380/src/redis-server *: 6380 root 4305 10 15:28? 00:00:05 / usr/local/redis-6379/src/redis-sentinel *: 26379 [sentinel] root 4306 10 15:28? 00:00:05 / usr/local/redis-6380/src/redis-sentinel *: 26380 [sentinel] root 4339 4144 0 15:56 pts/1 00:00:00 grep redis [root@monitor redis-6380] #

From the Sentinel configuration file, you can see that the current main library has changed.

Summary

Sentinel ports 26379 and 26380 of redis cannot be connected by client software, but can be connected by programs. Client software can only connect ports 6379 and 6380 directly. The use of Sentinel Monitoring automatically switches over the slave when the master fails, and becomes the slave when the master starts. Seeing that others only configure single Sentinel 26379, this situation does not guarantee the high availability of the Sentinel program itself.

After reading this article, have you learned the method of master-slave fault switching in Redis Sentinel mode? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel. Thank you for reading.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.