Shulou(Shulou.com)06/02 Report--

3 servers Redis highly available Sentinel mode

Learning redis is highly available

3 servers Redis highly available Sentinel mode

3.1Master redis configuration

3.2Configuring from redis

1. Introduction

2. Redis program installation

3. Sentinel mode configuration

3.3.activate redis and Sentinel

4. Summary

1. Introduction

Redis's Sentinel system, which manages multiple Redis servers (instance), performs the following three tasks:

Monitoring (Monitoring): Sentinel will constantly check whether your master server and slave server are working properly.

Reminder (Notification): when there is a problem with a monitored Redis server, Sentinel can send notifications to administrators or other applications through API.

Automatic failover (Automatic failover): when a master server does not work properly, Sentinel starts an automatic failover operation, which upgrades one of the slaves of the failed master server to the new master server and allows the other slave servers of the failed master server to replicate the new master server When the client tries to connect to the failed primary server, the cluster also returns the address of the new primary server to the client, so that the cluster can use the new primary server instead of the failed server.

Redis Sentinel is a distributed system where you can run multiple Sentinel processes (progress) in a single architecture, which use the rumor protocol (gossip protocols) to receive information about whether the master server is offline, and use the voting protocol (agreement protocols) to decide whether to perform automatic failover and which slave server to choose as the new master server.

Although Redis Sentinel is released as a separate executable redis-sentinel, it is really just a Redis server running in a special mode, and you can start Redis Sentinel with the-sentinel option when starting a normal Redis server.

Environment

CentOS7.2

Redis3.2.8

Server IPredis port Sentinel port server role 10.1.0.160637926379 Master 10.1.0.1637926379 from 110.1.0.71637926379 from 22. Redis program installation

The following is a single redis installation script that can be used with a single redis.

Cat install_redis.sh

#! / usr/bin/env bash

# It's Used to be install redis.

# Created on 2016-10-19 11:18.

# @ author: Chinge_Yang.

# Version: 1.0

Function install_redis () {

#

Sourcepackage_dir= "/ tmp"

Redis_install_dir= "/ usr/local/redis"

Cd ${sourcepackage_dir}

If [!-f "redis-stable.tar.gz"]; then

Wget http://download.redis.io/releases/redis-stable.tar.gz

Fi

Cd ${makework_dir}

Tar-zxvf ${sourcepackage_dir} / redis-stable.tar.gz

Cd redis-stable

Make PREFIX=/usr/local/redis install

Return_echo "make"

Mkdir-p / usr/local/redis/ {etc,var}

Rsync-avz redis.conf / usr/local/redis/etc/

Sed-I 's@pidfile.*@pidfile / var/run/redis-server.pid@' $redis_install_dir/etc/redis.conf

Sed-I "s@logfile.*@logfile $redis_install_dir/var/redis.log@" $redis_install_dir/etc/redis.conf

Sed-I "s @ ^ dir. * @ dir $redis_install_dir/var@" $redis_install_dir/etc/redis.conf

Sed-I 's/daemonize no/daemonize yes/g' / usr/local/redis/etc/redis.conf

Sed-I's / ^ # bind 127.0.0.1/bind 127.0.0.1 G'/ usr/local/redis/etc/redis.conf

Rsync-avz ${sourcepackage_dir} / init.d/redis-server / etc/init.d/

/ etc/init.d/redis-server start

Chkconfig-add redis-server

Chkconfig redis-server on

#

}

Install_redis

Example of redis start and stop script:

Cat redis-server

#! / bin/bash

#

# redis- this script starts and stops the redis-server daemon

#

# chkconfig:-85 15

# description: Redis is a persistent key-value database

# processname: redis-server

# config: / usr/local/redis/etc/redis.conf

# config: / etc/sysconfig/redis

# pidfile: / usr/local/redis/var/redis-server.pid

# Source function library.

. / etc/rc.d/init.d/functions

# Source networking configuration.

. / etc/sysconfig/network

# Check that networking is up.

["$NETWORKING" = "no"] & & exit 0

Redis= "/ usr/local/redis/bin/redis-server"

Prog=$ (basename $redis)

REDIS_CONF_FILE= "/ usr/local/redis/etc/redis.conf"

[- f / etc/sysconfig/redis] & &. / etc/sysconfig/redis

Lockfile=/var/lock/subsys/redis-server

Start () {

[- x $redis] | | exit 5

[- f $REDIS_CONF_FILE] | | exit 6

Echo-n $"Starting $prog:"

Daemon $redis $REDIS_CONF_FILE

Retval=$?

Echo

[$retval-eq 0] & & touch $lockfile

Return $retval

}

Stop () {

Echo-n $"Stopping $prog:"

Killproc $prog

Retval=$?

Echo

[$retval-eq 0] & & rm-f $lockfile

Return $retval

}

Restart () {

Stop

Start

}

Reload () {

Echo-n $"Reloading $prog:"

Killproc $redis-HUP

RETVAL=$?

Echo

}

Force_reload () {

Restart

}

Rh_status () {

Status $prog

}

Rh_status_q () {

Rh_status > / dev/null 2 > & 1

}

Case "$1" in

Start)

Rh_status_q & & exit 0

, 1

Stop)

Rh_status_q | | exit 0

, 1

Restart)

, 1

Reload)

Rh_status_q | | exit 7

, 1

Force-reload)

Force_reload

Status)

Rh_status

Condrestart | try-restart)

Rh_status_q | | exit 0

*)

Echo $"Usage: $0 {start | stop | status | restart | condrestart | try-restart | reload | force-reload}"

Exit 2

Esac

Example of redis-sentinel start and stop script:

#! / bin/bash

#

# redis-sentinel-this script starts and stops the redis-server sentinel daemon

#

# chkconfig:-85 15

# description: Redis sentinel

# processname: redis-server

# config: / usr/local/redis/etc/sentinel.conf

# config: / etc/sysconfig/redis

# pidfile: / usr/local/redis/var/redis-sentinel.pid

# Source function library.

. / etc/rc.d/init.d/functions

# Source networking configuration.

. / etc/sysconfig/network

# Check that networking is up.

["$NETWORKING" = "no"] & & exit 0

Redis= "/ usr/local/redis/bin/redis-sentinel"

Prog=$ (basename $redis)

REDIS_CONF_FILE= "/ usr/local/redis/etc/sentinel.conf"

[- f / etc/sysconfig/redis] & &. / etc/sysconfig/redis

Lockfile=/var/lock/subsys/redis-sentinel

Start () {

[- x $redis] | | exit 5

[- f $REDIS_CONF_FILE] | | exit 6

Echo-n $"Starting $prog:"

Daemon $redis $REDIS_CONF_FILE-- sentinel

Retval=$?

Echo

[$retval-eq 0] & & touch $lockfile

Return $retval

}

Stop () {

Echo-n $"Stopping $prog:"

Killproc $prog

Retval=$?

Echo

[$retval-eq 0] & & rm-f $lockfile

Return $retval

}

Restart () {

Stop

Start

}

Reload () {

Echo-n $"Reloading $prog:"

Killproc $redis-HUP

RETVAL=$?

Echo

}

Force_reload () {

Restart

}

Rh_status () {

Status $prog

}

Rh_status_q () {

Rh_status > / dev/null 2 > & 1

}

Case "$1" in

Start)

Rh_status_q & & exit 0

, 1

Stop)

Rh_status_q | | exit 0

, 1

Restart)

, 1

Reload)

Rh_status_q | | exit 7

, 1

Force-reload)

Force_reload

Status)

Rh_status

Condrestart | try-restart)

Rh_status_q | | exit 0

*)

Echo $"Usage: $0 {start | stop | status | restart | condrestart | try-restart | reload | force-reload}"

Exit 2

Esac

3. Sentinel mode configuration

3 hosts with the same settings:

Follow the previous single redis installation method to install the program

Create the corresponding data directory

Mkdir-p / usr/local/redis/data/redis

Mkdir-p / usr/local/redis/data/sentinel

Mkdir-p / usr/local/redis/sbin

Vim / usr/local/redis/sbin/redis-server # uses the sample script above

Vim / usr/local/redis/sbin/redis-sentinel # uses the sample script above

3.1Master redis configuration

Vim redis.conf

Daemonize yes

Pidfile "/ usr/local/redis/var/redis-server.pid"

Port 6379

Tcp-backlog 128

Timeout 0

Tcp-keepalive 0

Loglevel notice

Logfile "/ usr/local/redis/var/redis-server.log"

Databases 16

Save 900 1

Save 300 10

Save 60 10000

Stop-writes-on-bgsave-error yes

Rdbcompression yes

Rdbchecksum yes

Dbfilename dump.rdb

Dir "/ usr/local/redis/data/redis"

Masterauth "20170310"

Requirepass "20170310"

Slave-serve-stale-data yes

Slave-read-only yes

Repl-diskless-sync no

Repl-diskless-sync-delay 5

Repl-disable-tcp-nodelay no

Slave-priority 100

Appendonly yes

Appendfilename "appendonly.aof"

Appendfsync everysec

No-appendfsync-on-rewrite no

Auto-aof-rewrite-percentage 100

Auto-aof-rewrite-min-size 64mb

Aof-load-truncated yes

Lua-time-limit 5000

Slowlog-log-slower-than 10000

Slowlog-max-len 128

Latency-monitor-threshold 0

Notify-keyspace-events ""

Hash-max-ziplist-entries 512

Hash-max-ziplist-value 64

List-max-ziplist-entries 512

List-max-ziplist-value 64

Set-max-intset-entries 512

Zset-max-ziplist-entries 128

Zset-max-ziplist-value 64

Hll-sparse-max-bytes 3000

Activerehashing yes

Client-output-buffer-limit normal 0 0 0

Client-output-buffer-limit slave 256mb 64mb 60

Client-output-buffer-limit pubsub 32mb 8mb 60

Hz 10

Aof-rewrite-incremental-fsync yes

Cluster file configuration

Vim sentinel.conf

Port 26379

Pidfile "/ usr/local/redis/var/redis-sentinel.pid"

Dir "/ usr/local/redis/data/sentinel"

Daemonize yes

Logfile "/ usr/local/redis/var/redis-sentinel.log"

Sentinel monitor mymaster 10.1.0.160 6379 2

Sentinel parallel-syncs mymaster 2

Sentinel auth-pass mymaster 20170310

3.2Configuring from redis

Compared with the main redis configuration, the following lines have been added:

Slaveof 10.1.0.160 6379

Vim redis.conf

Daemonize yes

Pidfile "/ usr/local/redis/var/redis-server.pid"

Port 6379

Tcp-backlog 128

Timeout 0

Tcp-keepalive 0

Loglevel notice

Logfile "/ usr/local/redis/var/redis-server.log"

Databases 16

Save 900 1

Save 300 10

Save 60 10000

Stop-writes-on-bgsave-error yes

Rdbcompression yes

Rdbchecksum yes

Dbfilename dump.rdb

Dir "/ usr/local/redis/data/redis"

Masterauth "20170310"

Requirepass "20170310"

Slaveof 10.1.0.160 6379

Slave-serve-stale-data yes

Slave-read-only yes

Repl-diskless-sync no

Repl-diskless-sync-delay 5

Repl-disable-tcp-nodelay no

Slave-priority 90

Appendonly yes

Appendfilename "appendonly.aof"

Appendfsync everysec

No-appendfsync-on-rewrite no

Auto-aof-rewrite-percentage 100

Auto-aof-rewrite-min-size 64mb

Aof-load-truncated yes

Lua-time-limit 5000

Slowlog-log-slower-than 10000

Slowlog-max-len 128

Latency-monitor-threshold 0

Notify-keyspace-events ""

Hash-max-ziplist-entries 512

Hash-max-ziplist-value 64

List-max-ziplist-entries 512

List-max-ziplist-value 64

Set-max-intset-entries 512

Zset-max-ziplist-entries 128

Zset-max-ziplist-value 64

Hll-sparse-max-bytes 3000

Activerehashing yes

Client-output-buffer-limit normal 0 0 0

Client-output-buffer-limit slave 256mb 64mb 60

Client-output-buffer-limit pubsub 32mb 8mb 60

Hz 10

Aof-rewrite-incremental-fsync yes

Vim sentinel.conf

Port 26379

Pidfile "/ usr/local/redis/var/redis-sentinel.pid"

Dir "/ usr/local/redis/data/sentinel"

Daemonize yes

Logfile "/ usr/local/redis/var/redis-sentinel.log"

Sentinel monitor mymaster 10.1.0.160 6379 2

Sentinel config-epoch mymaster 0

3.3.activate redis and Sentinel

Start redis, both master and slave must be started.

/ usr/local/redis/sbin/redis-server start

Start cluster monitoring, master and slave must be started

/ usr/local/redis/sbin/redis-sentinel start

Start error processing

Error 1:

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory=1' to / etc/sysctl.conf and then reboot or run the command' sysctl vm.overcommit_memory=1' for this to take effect.

Solution (overcommit_memory)

1. Add the following settings for `vim / etc/ sysctl.conf`, and then `sysctl-p`

"vm.overcommit_memory = 1"

Optional values: 0, 1, 2.

0, which means that the kernel will check whether there is enough memory available for the application process; if there is enough memory available, the memory request is allowed; otherwise, the memory request fails and the error is returned to the application process.

1, which means that the kernel allows all physical memory to be allocated, regardless of the current memory state.

2, indicating that the kernel allows more than the sum of all physical memory and swap space to be allocated

Note: when redis uses dump data, it will fork a child process. Theoretically, the memory occupied by the child process is the same as that of parent. For example, the memory occupied by parent is 8 GB. At this time, 8 GB of memory should also be allocated to child. If the memory is not affordable, it will often cause the downmachine or IO load of the redis server to be too high and reduce efficiency. So the optimized memory allocation policy here should be set to 1 (meaning that the kernel allows all physical memory to be allocated, regardless of the current memory state).

Overcommit and OOM are also involved here.

What are Overcommit and OOM?

In Unix, when a user process requests memory using the malloc () function, if the return value is NULL, the process knows that there is no available memory space and will process it accordingly. Many processes print error messages and exit.

Linux uses a different approach, replying "yes" to most requests for memory so that it can run more and larger programs. Because memory will not be used immediately after applying for memory. This technique is called Overcommit.

OOM killer (OOM=out-of-memory) occurs when there is insufficient memory. It chooses to kill some processes (user-mode processes, not kernel threads) to free memory.

Strategy of Overcommit

There are three strategies for overcommit under Linux (Documentation/vm/overcommit-accounting):

0. Heuristic strategy. Reasonable overcommit will be accepted, unreasonable overcommit will be rejected.

1. Any overcommit will be accepted.

two。 Commit is rejected when the system allocates more memory than the swap+N%* physical RAM (N% is determined by vm.overcommit_ratio).

The policy for overcommit is set through vm.overcommit_memory.

The percentage of overcommit is set by vm.overcommit_ratio.

# echo 2 > / proc/sys/vm/overcommit_memory

# echo 80 > / proc/sys/vm/overcommit_ratio

When oom-killer occurs, linux chooses which processes to kill

The function for selecting a process is the oom_badness function (in mm/oom_kill.c), which calculates the number of points for each process (0,1000).

The higher the points, the more likely the process is to be killed.

The number of points per process is related to oom_score_adj, and oom_score_adj can be set (- 1000 lowest, 1000 highest).

Error 2:

WARNING: The TCP backlog setting of 511 cannot be enforced because / proc/sys/net/core/somaxconn is set to the lower value of 128.

Echo 511 > / proc/sys/net/core/somaxconn

Error 3:

16433 Jun X 12 it was originally set to 14 52 it was originally set to 37.734 * Increased maximum number of open files to 10032.

The default value of the newly installed linux is only 1024. When the load is heavy, error: too many open files often appears.

Ulimit-a: use all the limit values that you can view the current system

Vim / etc/security/limits.conf

Add at the end of the file

* soft nofile 65535

* hard nofile 65535

Execute su or re-close the connection and then execute ulimit-a to view the modified results.

Failover mechanism

After starting the cluster, the cluster program defaults to adding the cluster information to the master-slave sentinel.conf file

Lord:

Port 26379

Pidfile "/ usr/local/redis/var/redis-sentinel.pid"

Dir "/ usr/local/redis/data/sentinel"

Daemonize yes

Logfile "/ usr/local/redis/var/redis-sentinel.log"

Sentinel myid aeff525d03a2234ef834808f7991761db03a1973

Sentinel monitor mymaster 10.1.0.160 6379 2

Sentinel parallel-syncs mymaster 2

Sentinel auth-pass mymaster 20170310

# Generated by CONFIG REWRITE

Sentinel config-epoch mymaster 0

Sentinel leader-epoch mymaster 0

Sentinel known-slave mymaster 10.1.0.71 6379

Sentinel known-slave mymaster 10.1.0.161 6379

Sentinel current-epoch 0

From 1:

Port 26379

Pidfile "/ usr/local/redis/var/redis-sentinel.pid"

Dir "/ usr/local/redis/data/sentinel"

Daemonize yes

Logfile "/ usr/local/redis/var/redis-sentinel.log"

Sentinel myid 01b1b7674abe648f6a2344fc5610e73b7e87cb8a

Sentinel monitor mymaster 10.1.0.160 6379 2

Sentinel config-epoch mymaster 0

# Generated by CONFIG REWRITE

Sentinel leader-epoch mymaster 0

Sentinel current-epoch 0

From 2:

Port 26379

Pidfile "/ usr/local/redis/var/redis-sentinel.pid"

Dir "/ usr/local/redis/data/sentinel"

Daemonize yes

Logfile "/ usr/local/redis/var/redis-sentinel.log"

Sentinel myid f1589f48079b3b3b536add4e2e01a36304aeba8c

Sentinel monitor mymaster 10.1.0.160 6379 2

Sentinel config-epoch mymaster 0

# Generated by CONFIG REWRITE

Sentinel leader-epoch mymaster 0

Sentinel current-epoch 0

Simulate the main fault

[root@show160 redis] # / usr/local/redis/bin/redis-cli-p 6379

127.0.0.1 6379 > AUTH 20170310

OK

127.0.0.1 purl 6379 > DEBUG SEGFAULT

Could not connect to Redis at 127.0.0.1:6379: Connection refused

Not connected > quit

From the Sentinel configuration file, you can see that the current main library has changed.

4. Summary

Redis's Sentinel port 26379 uses redis-cli to connect to view Sentinel related information. To connect to this highly available redis, you can use an official connection client. The use of Sentinel Monitoring automatically switches over the slave when the master fails, and becomes the slave when the master starts. At least 3 sentinels and 3redis nodes are required to allow one node to be attached and to ensure service availability.

Reference:

Https://redis.io/topics/sentinel

Http://www.redis.cn/topics/sentinel.html

Http://www.majunwei.com/view/201610302123020678.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.