Shulou(Shulou.com)06/02 Report--

In this article, we mainly study the basic principles and logical architecture of heartbeat high availability cluster.

Ll article navigation

The basic principles of heartbeat

Introduction to the version of heartbeat

Terms related to heartbeat

Cluster components of heartbeat

Heartbeat connection of heartbeat

Configuration file for heartbeat

Ll requirements

Master the related components and simple configuration of heartbeat high availability cluster

The basic principles of heartbeat

Heartbeat is a cluster software, which is mainly composed of two core parts: heartbeat information detection and resource management.

In the cluster built by heartbeat, each server will send heartbeat information (messages) to other cluster nodes and collect and analyze them to judge the status of the node and whether the node is valid or not. When the server cannot detect the heartbeat information of other nodes or can not connect through the network within a specified period of time, it will think that the other node is invalid. at this time, the server needs to start the resource takeover module to take over the services and resources on the failed node.

Heartbeat can only perform heartbeat information detection and resource monitoring, and does not monitor other resources and application services. To monitor the availability of other resources and application services, you need to install third-party plug-ins, such as ipfail, ldirectord, etc.

Similarly, heartbeat cannot monitor problems with the operating system itself. If the primary node can not send heartbeat information to the standby node due to operating system problems, the standby node can not receive the information of the primary node, so it is considered that the primary node has failed. At this time, the resource takeover module will be started to take over the services and resources of the primary node. However, the resources and services of the primary node are not released, at this time, the primary and standby nodes will compete for resources, which may lead to data corruption of shared resources or file system crash. For linux systems, to solve this problem, you need to open the watchdog module in the kernel. After opening this module, watchdog will regularly perform write operations to the / dev/watchdog device file to determine whether the system is running properly. If watchdog thinks the kernel hangs, it automatically restarts the system, freeing up the node's services and resources.

Introduction to the version of heartbeat

Heartbeat is divided into v1.x and v2.x, and the latest version is v3.x.

The v2.x version is compatible with the v1.x configuration file, and v2.x is much more powerful than v1.x from a functional point of view. V3.x is the revised version of v2.x, which mainly fixes some bug in v2.x. The most important difference is to change the pacemaker of CRM Explorer and integrate some modules into pacemaker (pacemaker). Based on CentOS6.5, from the installation point of view, the heartbeat-pils package conflicts with the cluster-glue package. If you use yum installation, you will not install the heartbeat-pils package. The default is to install the cluster-glue package, but heartbeat v2.x cannot call the cluster-glue package that is too high. So during installation, you need to resolve the dependencies manually.

1 、 heartbeat v1.x

The resource file of v1.x is haresources, and the configuration interface is also more haresources.

Heartbeat v1.x allows cluster and node resources to be configured through two files under / etc/ha.d.

Ha.cf: define cluster nodes, time intervals for failure detection and handover, cluster logging mechanism, and Fence methods.

Haresources: define cluster resource groups. Each line can define failed over cluster nodes and a set of resources, including IP address and text.

Component systems, application services and storage, etc.

2 、 heartbeat v2.x

Heartbeat v2.x has made a big change compared with v1.x, introducing the cluster resource manager CRM, and 2.x is based on CRM managed

The resource file is changed to cib.xml. Due to the introduction of CRM, a process runs independently on the running heartbeat service node

Crm, users can send requests with this process. On the server side, you need to run the crmd process, which listens on a socket

The port is 5560. Users can send requests to the server through the client crm. Crm is actually a crm shell (crmsh) life.

Command line interface, through which users can configure cluster services; in addition, heartbeat v2.x also adds GUI graphics configuration tool

The name of this module is heartbeat-gui, which allows users to configure cluster services in a graphical scenario.

V2.x supports Resource Agent (resource agent) in the form of LSB, OCF, STONITH, etc.

V2.x supports the configuration and management tools of the GUI graphical interface.

V2.x monitors multiple resource groups independently, eliminating the need for third-party scripts such as mom or ldirctored.

V1.x only supports dual nodes, while v2.x supports up to 16 based on CRM (Cluster Resource Manager) explorer mode

Nodes. This mode uses the configuration of XML-based CIB (Cluster Infomation Base) resource information. The cib file will be found in each section

Automatic point-to-point replication, you can do the following:

Configuration and monitoring of cluster nodes

Configuration of stickiness, constraints, groups and dependencies of cluster resources

Logging, monitoring, arbitration and Fence standard management

When the service fails or the set criteria are met, some actions need to be performed

3 、 heartbeat v3.x

After the v3.x version, heartbeat was functionally split, and each functional module was developed as a different component independently. In configuration

It is basically the same as v2.x version. Its architecture is divided into heartbeat, pacemaker (pacemaker) and cluster-glue (cluster bonding).

They can work in conjunction with other modules.

Heartbeat: responsible for maintaining communication and information transmission between cluster nodes.

Pacemaker: also known as CRM, it is the control center that manages the HA through which clients configure and monitor the entire cluster.

Cluster-glue: equivalent to a middleware that associates heartbeat with pacmaker, mainly including LRM and STONITH

Resource agent: used to control the start and stop of the service and to monitor the collection of scripts for the scripting service; LRM calls these scripts to

Realize the start, stop and restart of resources.

Terms related to heartbeat

Node (node)

A separate host running the heartbeat process becomes the node; node is the core component of HA, and each node runs its own independent operating system and heartbeat services. In heartbeat cluster software, nodes are divided into master node and standby / backup node. Master node and standby node have their own hostname and have their own set of resources. For example: file system, disk, ip address, application service and so on. The primary node generally runs one or more services, while the standby node belongs to the monitoring state.

Resources (resource)

A resource is a controllable entity in a node, and when the primary node fails, the standby node will take over the resources on the primary node. Entities in heartbeat services that can be treated as resources are:

1. File system, disk partition

2. NFS network storage system

3. IP address

4. Application services, such as httpd, mysqld, etc.

Event (event)

Events refer to things that can happen in a cluster. For example, node system failure, network communication failure, network card failure, application failure and so on, these events may lead to the transfer of resources in the node. HA's testing is also based on these events.

Action (action)

Action refers to the way the HA responds when the event occurs, and it is controlled by the shell script. For example, when the primary node fails, the backup node will shut down or start the service by implementing the set script, thus taking over the resources of the failed node.

Cluster components of heartbeat

The whole communication module of heartbeat:heartbeat software, the communication between all nodes is completed by this module. This module will start different event handler according to different types of communication, and when the monitor hears different communication requests, it will be handed over to different handler for processing. This can be seen in this heartbeat log.

CCM (cluster consensus menbership): cluster consensus member; it serves as a connecting link between the preceding and the following. Monitor the underlying heartbeat information. When the heartbeat information cannot be monitored, the vote count and convergence status information of the cluster will be recalculated, and the result will be sent to the upper layer for processing. What decision will the upper layer take based on this result. Ccm can also generate a topology overview of a cluster node, taking this node as a reference to ensure that the node can take corresponding actions under special circumstances.

CRM (cluster resource manager): cluster resource manager, also known as pacemaker. It mainly manages the resource allocation information of cluster node services and realizes the allocation and scheduling of resources. According to the configuration information and running status of the resource, CRM decides which node the resource should run on. However, CRM does not directly participate in the action, but leaves it to other components to complete. CRM transfers the basic information of each member node collected by the heartbeat component to CCM to update the membership information of the whole cluster. Command LRM to perform operations such as start | stop | restart | status | monitor on the resources of the current node, and at the same time receive feedback from LRM and make corresponding decisions for follow-up work. CRM is also responsible for recording the information returned by each component into the log file by calling the set logging program.

LRM (local resource manager): local resource manager; it is a heartbeat module that directly operates resources in cluster management; it is responsible for starting, stopping, restarting or monitoring resources. This module currently supports four types of resource agent: heartbeat itself, LSB, OCF, and STONITH;. The script locations are as follows:

Heartbeat:/etc/ha.d/resource.d/

LSB:/etc/rc.d/init.d/

OCF:/usr/lib/resource.d/heartbeat/

STONITH (Shot The Other Node In The heat): commonly known as "head shot"; this way is directly through the control of the power supply

Switch the way in which a node is shut down or restarted. In the heartbeat high availability cluster, if the master node is unable to respond to the heartbeat information with the slave node due to special reasons, the slave node will think that the master node has failed and will immediately start the resource takeover module to take over the resources on the master node, resulting in resource contention between the master and slave nodes in the cluster. In order to avoid file damage or text system collapse caused by resource contention, at this time, after the standby node takes over the resources, it will send shutdown or restart commands to the primary node through the network, thus allowing the primary node to release resources. This is what we call a "head shot". It should be noted that the STONITH mechanism requires hardware support.

LRM controls resources by calling various scripts in the above path. Scripts can be defined by themselves and only need to follow specific parameters: start | stop | restart | status

PE (policy engine): policy engine; mainly responsible for calculating and analyzing the information sent by CRM compared with the parameter configuration in the configuration file, and sending the calculated and analyzed results to TE through CRM to analyze the subsequent actions to be performed. The information that PE needs to calculate and analyze is mainly which nodes, the status of each node, which resources are currently managed, which node each resource is currently running in, and the status of each resource in each node.

TE (translation engine): mainly analyzes the calculation results of PE, and then converts them into subsequent corresponding operations according to the configuration information.

PE and TE do not communicate directly, they both transmit information through CRM, and PE and TE are started only on the node in active.

CIB (cluster infomation base): cluster information base; CIB is the resource configuration in the cluster, the initial state of node information and the collection center after changes, is an ever-changing information base. When the information of resources and nodes in the cluster changes and is collected by CIB, CIB will immediately update the cluster information base and distribute it to other nodes. However, the distribution action is not done by CIB itself, but by heartbeat components. The configuration file name of CIB is cib.xml,CIB when running, the configuration information in cib.xml resides in the memory of DC (desginnated coordinator, master node), and only DC will often read and modify the contents of this file to ensure that the resources and node information of each node are updated to the latest state. Cib.xml configuration files on other nodes are copied from DC through heartbeat components.

Heartbeat connection of heartbeat

Heartbeat deployment requires at least two servers. What physical link is used to connect the heartbeat information of these two servers?

1. Serial cable (preferred, the disadvantage is that the distance should not be too long)

2. Connect an Ethernet cable to the network card of two servers (recommended)

3. Two servers are connected to the same switch (the second choice increases the failure rate of the switch equipment; at the same time, the line is not a dedicated heartbeat and is easily affected by other data transmission)

Configuration file for heartbeat

The default configuration file directory for heartbeat is / etc/ha.d/,. This directory contains three main configuration files: authkey, ha.cf, and haresources.

Configuration file

Action

Description / etc/ha.d/authkeyheartbeat authentication file

Highly available server pairs authenticate peers according to this file / etc/ha.d/ha.cfheartbeat basic parameter configuration file configuration heartbeat basic parameters / etc/ha.d/haresourcesheartbeat resource configuration file configuration IP resources and scripts, etc.

Some parameters of ha.cf configuration file are described in detail:

Autojoin none

# nodes in the cluster will not join automatically

Logfile / var/log/ha-log

# specify the log location of the named heartbaet

Keepalive 2

# specify a heartbeat interval of 2 seconds (that is, a broadcast is sent on the eth2 every two seconds)

Deadtime 30

# specify that after the standby node does not receive the heartbeat signal of the primary node within 30 seconds, it will immediately take over the service resources of the primary node

Warntime 10

# specifies that the heartbeat delay is ten seconds. When the backup node cannot receive the heartbeat of the primary node within 10 seconds, a warning log will be written to the log, but the service will not be switched at this time.

Initdead 120

# on some systems, it takes some time for the network to work properly after the system is started or rebooted. This option is used to resolve the time interval caused by this situation. The value is at least twice that of deadtime.

Udpport 694

# set the port used for broadcast communication. 694 is the default port number.

Baud 19200

# set the baud rate of serial communication

Bcast eth0

# Linux indicates that the heartbeat is broadcast over Ethernet and is broadcast on the eth0 interface.

# mcast eth0 225.0.0.1 694 1 0

# Udp Multicast of the Nic eth0 is used to organize the heartbeat, which is generally used when there is more than one backup node. Bcast, ucast and mcast represent broadcast, unicast and multicast, respectively, and are the three ways to organize the heartbeat.

# ucast eth0 192.168.1.2

# the udp unicast of the network card eth0 is used to organize the heartbeat, and the IP address should be the IP address of the other computer.

Auto_failback on

# is used to define whether the service will be automatically switched back when the primary node is restored. The two hosts of heartbeat are the primary node and the backup node respectively. Under normal circumstances, the primary node occupies resources and runs all the services. in case of failure, the resources are handed over to the backup node and the service is run by the backup node. If this option is set to on, once the primary node resumes operation, it automatically acquires resources and replaces the backup node. If this option is set to off, then when the primary node is restored, it will become the backup node, and the original backup node will become the primary node.

# stonith baytech / etc/ha.d/conf/stonith.baytech

The main function of # stonith is to detach the nodes with problems from the cluster environment, thereby releasing cluster resources and avoiding the occurrence of two nodes competing for one resource. Ensure the security and integrity of shared data.

# watchdog / dev/watchdog

# this option is optional and uses Heartbeat to monitor the running status of the system. To use this feature, you need to load the "softdog" kernel module into the kernel to generate the actual device files, and if you do not have this kernel module in the system, you need to specify this module and recompile the kernel. After compilation, enter "insmod softdog" to load the module. Then enter "grep misc / proc/devices" (which should be 10) and "cat / proc/misc | grep watchdog" (which should be 130). Finally, the device file is generated: "mknod / dev/watchdog c 10130". You can use this feature

Node node1.magedu.com

# Host node hostname, which can be viewed by the command "uanme-n".

Node node2.magedu.com

# standby node hostname

Ping 192.168.12.237

# choose the ping node. The better the ping node, the stronger the HA cluster. You can choose a fixed router as the ping node, but it is best not to select the members of the cluster as the ping node. The ping node is only used to test the network connection.

Ping_group group1 192.168.12.120 192.168.12.237

# similar to ping ping, a set of ip addresses

Apiauth pingd gid=haclient uid=hacluster

Respawn hacluster / usr/local/ha/lib/heartbeat/pingd-m 100-d 5s

# this option is an optional configuration that lists the processes started and shut down with heartbeat, which are typically plug-ins integrated with heartbeat and can be automatically restarted if they fail. The most commonly used process is pingd, which is used to detect and monitor the status of the network card and needs to cooperate with the ping node specified in the ping statement to check the connectivity of the network. Where hacluster indicates the identity of starting the pingd process.

# the following configuration is the key, that is, to activate crm management and start using v2 style format

Crm respawn

# Note, you can also use crm yes, but if you write it this way, if there is a problem with the later cib.xml configuration

# will cause heartbeat to restart the server directly, so it is recommended to use respawn when testing

# the following is the compression of the transmitted data, which is optional

Compression bz2

Compression_threshold 2

This is the end of this article, welcome to point out the problems, the follow-up will continue to improve.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.