Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of "how to deploy a simple Sentinel system in Redis". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "how to deploy a simple Sentinel system in Redis" can help you solve the problem.

First, function and structure 1. Action

Before introducing Sentinel, let's review the technologies related to the high availability of Redis from a macro point of view. They include persistence, replication, sentinels and clusters, and the main roles and problems to be solved are:

Persistence: persistence is the simplest highly available method (sometimes not even classified as a highly available means). Its main function is to back up the data, that is, to store the data on the hard disk, ensuring that the data will not be lost due to the exit of the process.

Replication: replication is the basis of highly available Redis, and both Sentinels and clusters achieve high availability on the basis of replication. Replication mainly realizes the multi-machine backup of data, as well as load balancing and simple fault recovery for read operations. Defects: failure recovery can not be automated; write operations can not be load balanced; storage capacity is limited by a single machine.

Sentinel: on the basis of replication, Sentinel achieves automated fault recovery. Defect: the write operation cannot be load balanced; the storage capacity is limited by a single machine.

Cluster: through clustering, Redis solves the problem that write operations cannot be load balanced and storage capacity is limited by a single machine, and achieves a relatively perfect high availability solution.

Let's go back to the sentinels.

Redis Sentinel, or Redis Sentinel, was introduced in version 2.8 of Redis. The core function of the Sentinel is the automatic failover of the primary node. The following is a description of the Sentinel function in the official Redis documentation:

Monitoring (Monitoring): the Sentinel constantly checks whether the master node and slave node are functioning properly.

Automatic failover (Automatic failover): when the master node does not work properly, the Sentinel starts an automatic failover operation, which upgrades one of the slaves of the failed master node to the new master node and causes the other slave nodes to copy the new master node.

Configuration provider (Configuration provider): the client obtains the primary node address of the current Redis service by connecting to the sentinel during initialization.

Notification (Notification): the sentry can send the result of the failover to the client.

Among them, the monitoring and automatic failover functions enable the sentry to detect the failure of the primary node in time and complete the transfer, while the configuration provider and notification functions need to be reflected in the interaction with the client.

Here is an explanation of the use of the word "client" in the article: in the previous article, whenever you access the redis server through API, it will be called the client, including redis-cli, Java client Jedis, etc. For ease of distinction, the client in this article does not include redis-cli, but is more complex than redis-cli: redis-cli uses the underlying interfaces provided by redis, and the client encapsulates these interfaces and functions to make full use of the configuration provider and notification features of the Sentinel.

two。 Architecture

A typical Sentinel architecture diagram is as follows:

It consists of two parts, the sentinel node and the data node:

Sentinel node: the Sentinel system consists of one or more Sentinel nodes, which are special redis nodes and do not store data.

Data nodes: both master and slave nodes are data nodes.

II. Deployment

This section will deploy a simple Sentinel system consisting of 1 master node, 2 slave nodes and 3 Sentinel nodes. For convenience: all of these nodes are deployed on a single machine (LAN IP:192.168.92.128), distinguished by port numbers; the configuration of nodes is as simple as possible.

1. Deploy master and slave nodes

The master-slave node in the Sentinel system is the same as the ordinary master-slave node configuration and does not require any additional configuration. The following are the configuration files for the master node (port=6379) and 2 slave nodes (port=6380/6381), both of which are relatively simple and will not be described in detail.

# redis-6379.confport 6379daemonize yeslogfile "6379.log" dbfilename "dump-6379.rdb" # redis-6380.confport 6380daemonize yeslogfile "6380.log" dbfilename "dump-6380.rdb" slaveof 192.168.92.128 6379#redis-6381.confport 6381daemonize yeslogfile "6381.log" dbfilename "dump-6381.rdb" slaveof 192.168.92.128 6379redis-server redis-6379.confredis-server redis-6380.confredis-server redis-6381.conf

After the node starts, connect the master node to see if the master-slave status is normal. After the configuration is complete, start the master node and the slave node in turn:

two。 Deploy Sentinel Node

The Sentinel node is essentially a special Redis node.

The configuration of the three sentinel nodes is almost exactly the same, and the main difference lies in the different port numbers (26379 / 26380 / 26381). The configuration and startup mode of the 26379 nodes are described below. The configuration part is as simple as possible. More configurations will be described later.

# sentinel-26379.confport 26379daemonize yeslogfile "26379.log" sentinel monitor mymaster 192.168.92.128 6379 2

There are two ways to start the Sentinel node, and the two functions are exactly the same: the meaning of the sentinel monitor mymaster 192.168.92.128 6379 2 configuration is that the Sentinel node monitors the primary node, which is named mymaster, and the meaning of the last 2 is related to the failure determination of the primary node: at least two Sentinel nodes are required to agree to determine the failure of the primary node and fail over.

Redis-sentinel sentinel-26379.confredis-server sentinel-26379.conf-sentinel

3. Summary

After configuring and booting in the above way, the entire Sentinel system is started. Can be verified by connecting the sentinel node through redis-cli

In the process of building the Sentinel system, there are several points to pay attention to:

The main results are as follows: (1) the master-slave node in the sentinel system is no different from the ordinary master-slave node, and the fault detection and transfer are controlled and completed by the sentry.

(2) Sentinel nodes are essentially redis nodes.

(3) each sentinel node only needs to configure the monitoring master node to automatically discover other sentinel nodes and slave nodes.

(4) during the sentinel node startup and failover phase, the configuration files of each node will be rewritten (config rewrite).

Client access to the Sentinel system

While the previous section demonstrated the two main roles of sentinels: monitoring and automatic failover, this section combines the client side to demonstrate the other two roles of sentinels: configuration providers and notifications.

1. Code example

Before introducing the principle of the client, take the Java client Jedis as an example to demonstrate how to use it: the following code can connect to the Sentinel system we just built and perform various read and write operations (the code only shows how to connect the Sentinel, exception handling, resource shutdown, etc.).

Public static void testSentinel () throws Exception {String masterName = "mymaster"; Set sentinels = new HashSet (); sentinels.add ("192.168.92.128VV 26379"); sentinels.add ("192.168.92.128RV 26380"); sentinels.add ("192.168.92.128Rod 26381"); JedisSentinelPool pool = new JedisSentinelPool (masterName, sentinels) / / A lot of work has been done in the initialization process: Jedis jedis = pool.getResource (); jedis.set ("key1", "value1"); pool.close ();} Jedis client provides good support for Sentinel. As the above code shows, we just need to provide the Jedis with the Sentinel node collection and masterName to construct the JedisSentinelPool object; then we can use it like a normal redis connection pool: get the connection through pool.getResource () and execute the specific command. two。 Client principle

In the whole process, our code does not need to explicitly specify the address of the master node to connect to the master node; without any indication of failover in the code, we can automatically switch the master node after the Sentinel completes the failover. This can be done because the relevant work has been done in the JedisSentinelPool constructor, which mainly includes the following two points:

(1) traverse the Sentinel node to get the information of the master node: traverse the Sentinel node and obtain the information of the master node through one of the Sentinel nodes + masterName. This function is achieved by calling the sentinel get-master-addr-by-name command of the Sentinel node. An example of this command is as follows:

Once you get the master node information, stop traversing (so generally traversing to the first Sentinel node, the loop stops).

(2) increase the monitoring of the sentry: in this way, when a failover occurs, the client can receive a notification from the sentry, thus completing the handover of the master node. The specific approach is to use the publish and subscribe function provided by redis to open a separate thread for each Sentinel node, subscribe to the + switch-master channel of the Sentinel node, and reinitialize the connection pool when a message is received.

3. Summary

Through the introduction of the client principle, we can deepen the understanding of the Sentinel function:

(1) configuration provider: the client can obtain the information of the master node through the sentry node + masterName, where the role of the sentry is to configure the provider.

It is important to note that the Sentinel is only a configuration provider, not an agent. The difference between the two lies in: if it is a configuration provider, the client will establish a connection to the master node directly after obtaining the information of the master node through the sentry, and subsequent requests (such as set/get) will be sent directly to the master node; if it is an agent, every request from the client will be sent to the sentry, and the sentry will process the request through the master node.

As an example, it is well understood that the role of a sentry is to configure a provider, not an agent. In the previously deployed Sentinel system, modify the configuration file of the Sentinel node as follows:

Sentinel monitor mymaster 192.168.92.128 6379 2 changed to sentinel monitor mymaster 127.0.0.1 6379 2

(2) Notification: after the failover is completed, the sentry node will send the new master node information to the client so that the client can switch the master node in time. Then, if you run the client code on another machine in the LAN, you will find that the client cannot connect to the master node; this is because the Sentinel, as the configuration provider, queries the address of the master node through it to 127.0.0.1 redis 6379, and the client will establish a redis connection to 127.0.0.1 PVL 6379, which naturally cannot be connected. If the sentinel had been an agent, the problem would not have arisen.

IV. Basic principles

The basic methods of Sentinel deployment and use are introduced earlier, and this part introduces the basic principles of Sentinel implementation.

1. Commands supported by Sentinel nod

As a redis node running in special mode, the command supported by Sentinel node is different from that of ordinary redis node. In operation and maintenance, we can query or modify the Sentinel system through these commands, but more importantly, the Sentinel system can not achieve various functions such as fault discovery and failover without the communication between Sentinel nodes. a large part of the communication is achieved through the commands supported by the Sentinel node. The main commands supported by the Sentinel node are described below.

(1) basic query: through these commands, you can query the topology, node information and configuration information of the Sentinel system.

Info sentinel: get the basic information of all master nodes monitored

Sentinel masters: get the details of all master nodes monitored

Sentinel master mymaster: get the details of the monitored master node mymaster

Sentinel slaves mymaster: gets the details of the slave node of the monitored master node mymaster

Sentinel sentinels mymaster: get the details of the sentinel node of the monitored primary node mymaster

Sentinel get-master-addr-by-name mymaster: obtain the address information of the monitored master node mymaster, which has been described earlier

Sentinel is-master-down-by-addr: Sentinel nodes can use this command to ask whether the master node is offline, so as to judge whether the primary node is offline objectively.

(2) add / remove monitoring to the primary node

Sentinel monitor mymaster2 192.168.92.128 16379 2: exactly the same as the sentinel monitor function in the configuration file when deploying the Sentinel node, without further details

Sentinel remove mymaster2: cancels the monitoring of the primary node mymaster2 by the current Sentinel node

(3) forced failover

Sentinel failover mymaster: this command forces a failover of the mymaster, even if the current primary node is running well; for example, if the machine of the current primary node is about to be scrapped, it can be failed over in advance through the failover command.

two。 Basic principles

The key to the principle of the Sentinel is to understand the following concepts.

(1) scheduled tasks: each sentinel node maintains 3 scheduled tasks. The functions of the timing task are as follows: to obtain the latest master-slave structure by sending info commands to the master-slave nodes; to obtain the information of other sentinel nodes through the publish and subscribe function; and to determine whether to go offline by sending ping commands to other nodes for heartbeat detection.

(2) subjective downline: in the scheduled task of heartbeat detection, if other nodes do not reply for a certain period of time, the sentry node will take it off subjectively. As the name implies, the subjective downline means that a sentinel node "subjectively" judges the downline; the subjective downline corresponds to the objective downline.

(3) objective offline: after subjectively deactivating the master node, the sentinel node will ask other sentinel nodes about the status of the master node through the sentinel is-master-down-by-addr command; if it is judged that the number of sentinels offline of the primary node reaches a certain value, the primary node will be objectively offline.

It should be noted that objective offline is the only concept of the master node; if the slave node and the sentry node fail, after being subjectively offline by the sentry, there will be no subsequent objective offline and failover operations.

(4) Election leader sentinel node: when the master node is judged to be offline objectively, each sentinel node will negotiate and elect a leader sentry node, and the leader node will fail over it.

All sentinels monitoring the primary node may be selected as leaders, and the election algorithm is Raft algorithm; the basic idea of Raft algorithm is first-come-first-served: that is, in a round of elections, Sentinel A sends an application to B to become a leader, and if B does not agree to other sentinels, it will allow A to become a leader. The specific process of the election is not described in detail here. Generally speaking, the selection process of the Sentinel is very fast. Whoever completes the objective offline first will generally become the leader.

(5) failover: the elected leader sentinel starts the failover operation, which can be divided into three steps:

Select a new master node among the slave nodes: the principle of selection is to filter out the unhealthy slave node first; then select the slave node with the highest priority (specified by slave-priority); if the priority cannot be distinguished, select the slave node with the largest replication offset; if still unable to distinguish, select the slave node with the lowest runid.

Update the master-slave status: use the slaveof no one command to make the selected slave node the master node, and the slaveof command to make other nodes their slave nodes.

Set the offline master node (that is, 6379) as the slave node of the new master node, and when 6379 comes back online, it will become the slave node of the new master node.

Fifth, configuration and practice suggestions 1. Configuration

Several configurations related to Sentinels are described below.

(1) sentinel monitor {masterName} {masterIp} {masterPort} {quorum}

Sentinel monitor is the core configuration of the Sentinel, which is explained in the previous description of the deployment of the Sentinel node, where masterName specifies the name of the master node, masterIp and masterPort specify the address of the master node, and quorum is the threshold for determining the number of Sentinels that are objectively offline of the master node: when the number of Sentinels offline of the master node reaches quorum, the master node is objectively offline. It is recommended that the value be half the number of sentinels plus 1.

(2) sentinel down-after-milliseconds {masterName} {time}

Sentinel down-after-milliseconds is related to the judgment of subjective downline: the sentry uses the ping command to detect the heartbeat of other nodes, and if the other nodes do not reply beyond the time allocated by down-after-milliseconds, the sentry will take them off subjectively. This configuration is effective for the subjective downline decision of master node, slave node and sentry node.

The default value of down-after-milliseconds is 30000, that is, 30s, which can be adjusted according to different network environments and application requirements: the higher the value, the looser the judgment of subjective downline. The advantage is that the possibility of misjudgment is smaller, but the disadvantage is that the time for fault detection and failover is longer, and the waiting time for the client is also longer. For example, if the application requires high availability, the value can be appropriately reduced to complete the transfer as soon as possible when a failure occurs; if the network environment is relatively poor, the threshold can be appropriately raised to avoid frequent misjudgments.

(3) sentinel parallel-syncs {masterName} {number}

Sentinel parallel-syncs is related to replication of slave nodes after a failover: it specifies the number of slaves to initiate a replication operation to a new master node at a time. For example, suppose that after the master node switch is complete, three slave nodes will initiate replication to the new master node; if parallel-syncs=1, the slave node will start replication one by one; if parallel-syncs=3, the three slave nodes will start replication together.

The higher the value of parallel-syncs, the faster the replication time of the slave node, but the greater the pressure on the network load and hard disk load of the master node; it should be set according to the actual situation. For example, if the load of the master node is low and the slave node has higher requirements for service availability, you can increase the value of parallel-syncs appropriately. The default value for parallel-syncs is 1.

(4) sentinel failover-timeout {masterName} {time}

Sentinel failover-timeout is related to the judgment of failover timeout, but this parameter is not used to judge the timeout of the entire failover phase, but the timeout of several sub-stages. For example, if the time for the master node to promote the slave node exceeds timeout, or the time for the slave node to initiate a replication operation (excluding the time for replicating data) to the new master node exceeds timeout, it will lead to the failure of the failover timeout.

The default value for failover-timeout is 180000, or 180s; if it times out, the value will double next time.

(5) in addition to the above parameters, there are some other parameters, such as those related to security verification, which are not introduced here.

two。 Practical suggestions

(1) the number of Sentinel nodes should be more than one, on the one hand, increase the redundancy of Sentinel nodes to avoid the Sentinel itself becoming a highly available bottleneck; on the other hand, reduce the misjudgment of downline. In addition, these different sentinel nodes should be deployed on different physical machines.

(2) the number of sentinel nodes should be odd, so it is convenient for sentinels to make "decisions" through voting: leader election decisions, objective offline decisions, and so on.

(3) the configuration of each sentinel node should be consistent, including hardware and parameters; in addition, all nodes should use ntp or similar services to ensure accurate and consistent time.

(4) Sentinel's configuration provider and notification client function can only be realized with the support of the client. If the library used by the developer does not provide the corresponding support, the Jedis; mentioned above may need to be implemented by the developer himself.

(5) when nodes in the Sentinel system are deployed in docker (or other software that may perform port mapping), special attention should be paid to the fact that port mapping may cause the Sentinel system to fail to function properly, because the work of the Sentinel is based on communication with other nodes, while the port mapping of docker may prevent the Sentinel from connecting to other nodes. For example, Sentinels discover each other and rely on their declared IP and port. If one Sentinel An is deployed in a port-mapped docker, then other Sentinels cannot connect to A using the port claimed by A.

That's all for "how to deploy a simple Sentinel system in Redis". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.