Shulou(Shulou.com)06/02 Report--

The process of Microsoft WSFC failover was introduced to you in the foundation of WSFC fundamentals before. let's review it.

1. Deploy and configure cluster nodes as required to ensure that the cluster server uses redundancy technology to eliminate a single point of failure of the server, network, and storage

two。 Ensure that all nodes in the cluster can access shared storage

3. Cluster applications write application data to cluster shared storage

4. The administrator added the function roles above the node 1 server. After the addition, the node 1 server cluster database records the new role functions and associated information, and will later synchronize the information to other nodes 2 and the cluster arbitration disk.

5. Network-wide handshake detection between cluster nodes according to predetermined heartbeat detection frequency

6. When the server suddenly shuts down in node 1, the heartbeat detection frequency of node 2 reaches the threshold and it is determined that node 1 has gone offline.

7. After node 2 determines that node 1 has gone offline, it will view the cluster application currently hosted by the node 1 server based on the synchronized cluster database information, reapply the cluster application with the associated IP address, and put the cluster disk online at node 2.

8. The client accesses the cluster name normally and uses the cluster service, but the cluster application of the original node 1 is now provided by node 2, and the failover ends

In this article, we will focus on the occurrence of cluster failover. you can see that starting from step 5, the cluster begins to conduct a health check, and then uses the health check results to determine whether the cluster node should perform a failover.

This is necessary in any high availability cluster. As long as the cluster is highly available, there must be a mechanism for node health detection, which can be ping detection, handshake detection, API detection. In short, we need a mechanism to accurately ensure the health of nodes.

Whether the node health detection mechanism is accurate or not is the core of a cluster, and the cluster health detection mechanism that can really be used should accurately reflect the health status of the node. at the same time, the threshold of health monitoring can be changed flexibly according to the actual situation.

Node health detection is the basis for node failover in failover. If the node health detection reaches a certain threshold, the cluster will determine the node failure, which will cause all the cluster resources carried on the node to perform failover operations, and failover operations will sometimes produce downtime. Therefore, the frequency of cluster node health detection and the threshold for fault detection will be determined. There must be an environmental assessment.

Cluster health detection is divided into two categories, one is node detection, the other is resource detection, node detection is more simple, do not need to set up a lot of retry logic, as long as the health detection mode specified by our cluster, you do not let the cluster detect within the specified time and the specified number of times, the cluster will think that you are malfunctioning and can no longer provide services, and you need failover for all applications.

Compared with node detection, resource detection is more complex. Node detection can be understood as outer layer detection. For level detection of a node, if node detection fails, all resources on the node failover, while resource detection pays more attention to resource-level health. As Lao Wang explained in the misalignment of logs, each resource in the cluster will have its own resource.dll, and the cluster is in normal operation. RHS detects the lookalive and isalive of each resource according to the resource.dll definition of each resource. The detection methods of lookalive and isalive for each resource are different. Lookalive is only a rough detection, while isalive is more in-depth. The result of isalive detection will be reported to RCM,RCM and will retry again according to the failure policy of the resource, try to start the resource many times on the machine, or transfer the resource to another node to run.

To sum up, the health test results of the node determine whether all the applications above the node will be failed over, and the health test results of resources determine whether a single resource should be operated in accordance with the fault strategy. Resource detection is more complex. According to the detection method defined by resource dll, we should consider a variety of situations to detect lookalive and isalive. Administrators can also set failure policies for some resources, such as transferring to other nodes without trying to restart, or restarting resources with three failed attempts and then transferring to other nodes.

In the past, the failure of RHS detection of a single resource may also affect other cluster resources. after 2008, most programs have been isolated to separate RHS processes for detection, so there are few scenarios in which a single resource detection affects other resources.

The impact area of node health detection results is greater, and the impact surface of resource health detection is only a single resource. That's all you need to remember.

After introducing some basic theories, let's focus on node health detection.

Since WSFC 2012, Microsoft has changed the health of the node. Instead of simply performing a ping operation, Microsoft will perform a real handshake.

For example, there are two nodes in the current cluster. By default, they do a network-wide test every second, and each node and node will be detected before. If the handshake detection is initiated by node 1, it will actually perform a handshake and send a 134byte detection signal through UDP 3343 port to ask node 2 if you are there, node reply, I am here, so are you, are you there, node 1 replied that I am also there. At the end of a test here, WSFC's node health check is performed every other second between all nodes to ensure that no node will be omitted. Node health detection is mainly through UDP 3343 port.

Well, the detection of the UDP 3343 port is carried out on the network card in the cluster. The answer is that all the networks that have been checked for cluster communication. After the WSFC 2008 era, there are only three network types for the cluster.

0 clusterless communication

1 enable client and cluster communication

3 is used for cluster communication only

By default, when we create a cluster or add a cluster, there is a network topology generator inside the cluster to help us automatically enlighten the cluster network type of the network card and generate the intra-cluster network communication topology.

For example

If ISCSI Initiator is detected on the network card, it is automatically inspired to communicate without cluster for type 0, ensuring that the storage network is only used to exchange storage data.

If it is detected that a gateway is configured on the network card, the topology generator will assume that your card is intended to communicate with external clients, so it will automatically set the network card with the gateway to cluster network type 3.

If it is detected that there is no gateway on the network card, the topology generator will think that your card is only for internal cluster communication, does not need to provide external services, and will be set to cluster network type 1.

It should be noted that cluster communication within the cluster will do three things next.

1. Perform a network-wide inspection of the health status of all nodes in the cluster

two。 Perform update synchronization of the cluster database

3. Perform synchronization of cluster CSV metadata

These three operations all require very high network quality, especially CSV metadata and cluster database. If CSV metadata and cluster database are frequently lost because of poor network quality, the cluster database will be updated slowly, which may be reflected in the fact that the operation will return results for a long time, or the CSV operation is inefficient to write data.

Therefore, it is recommended that for the cluster network card of cluster network type 3, the network quality should be planned to ensure that packet loss will not occur. By default, these three kinds of traffic will be done by the network card of cluster network type 3. If the network card of cluster network type 3 fails, then the cluster re-planning traffic through the network topology generator is performed by network type 1, but network type 0 is not allowed to do so.

We usually need to configure some additional things for the network card of cluster network type 3, such as do not set the network card DNS server, WINS server or default gateway, cancel the DNS registration of the network card of cluster type 3, disable Netbios, adjust the order of the network card, make sure that the network card of cluster type 1 is at the front, and ensure that the network card of cluster type 3 is behind. Most of these settings are to ensure that after failover. The application of online DNS can register directly and quickly using cluster type 1 to ensure that network type 1 is always the highest priority when outbound network connection, and that the external network card has the highest priority when accessing AD for authentication. It is no longer important for the order of network cards after WSFC 2012. In the era of WSFC 2016, the access order setting of network cards has been cancelled. Therefore, most scenarios only need to remove the DNS and Netbios registration of the cluster type 3 network card.

If the WSFC 2016 cluster is an AD-dependent cluster, such as a SQL Server cluster based on domain authentication, you are worried that if the network card order is not designed, if the selection is carried out by the cluster type 3 network card when performing AD domain verification, it will time out because the network card cannot contact AD, which will affect the performance of cluster applications that depend on AD. Then you can choose to modify the network card interface metric through the Powwershell command to achieve the purpose of previously controlling the priority of the network card order.

The configuration command is as follows. The lower the metric, the higher the priority.

Set-NetIPInterface-InterfaceAlias "LAN"-InterfaceMetric 1

Set-NetIPInterface-InterfaceAlias "CLUS"-InterfaceMetric 2

Set-NetIPInterface-InterfaceAlias "ISCSI"-InterfaceMetric 3

If the cluster application is not dependent on AD, or if the cluster is deployed as a workgroup model, there is no need to set it up at all.

For the node health detection mentioned in this article, the NetFT component will build the detection topology. NetFT usually refers to Failover Cluster Virtual Adapter. When we install the cluster, the hidden device in the device manager shows that there is such a virtual network adapter. You can also see this network card with ipconfig / all, but the IP address is not configured. The main function of this cluster virtual network adapter is to help us build a highly available topology for network communication in the cluster. For example, when heartbeats are detected between our cluster nodes, NetFT will help us rebuild this topology from time to time. For example, node 1 and node 2 each have two network cards, one dedicated to the cluster network and one for the cluster network + management network. Then when NetFT detects that if the dedicated cluster network cannot perform heartbeat detection, it will dynamically switch to another network card to help us with heartbeat detection. The main function of NetFT is to help administrators automatically build the pre-existing communication topology of the cluster, to the maximum extent possible to ensure that the cluster network can communicate normally.

We already know that node health detection is based on the actual handshake detection initiated by UDP 3343 port, but how often is it detected? the detection time and threshold can be configured and modified. In the end, we need to modify the more appropriate threshold according to the actual scene.

For example, if the requirement of the scenario is that the application must be highly available, the node and the application are very important, there must be no problems, there is no instantaneous interruption, and the network condition is very good, then we can set the node detection to be more strict. For example, if the node is detected every other second and fails five times, it marks the node as a failure state and fails over all the resource applications above.

If the scenario cannot avoid instantaneous interruption, for example, if the customer's network card is unstable, it will suddenly break off and will soon be ready, the system is very slow, and the detection signal maybe will not respond so quickly. At this time, you can also set the detection threshold to be more relaxed. For example, for this kind of instantaneous interruption scenario, set the failover operation to be triggered only when 20 detection failures occur.

Microsoft's advice is first of all, do not change the testing time, always keep the heartbeat of the whole network every other second, the second is that the number of detection failures should not be changed too loosely, the maximum recommended setting is 20, if the setting is more than 20, too many times, it will lead to application downtime for a long time before it will be discovered, delayed downtime, and all the above applications will be affected Therefore, Microsoft recommends a loose threshold of up to 20 times.

In fact, changing the threshold detected by such a node is very simple, a command thing, but it is more important for us to think about the health status of the application and choose the most appropriate threshold.

If your environment is very strict about the availability of the application, once there is a problem with the node, the application needs to be immediately failed over to other nodes, and the network quality in your environment is very high, and there will be no packet loss or instantaneous interruption, then you can set the detection threshold to 5 or 10. The advantage is that the application always runs on the most reliable node, as long as the application of the current operating node detects lost for 5 times. Immediately failover to other nodes, but the problem is that we must ensure that there is no instantaneous interruption in the network environment. In the event of an instantaneous interruption, the application will failover frequently. Therefore, the premise of setting a strict detection threshold must be strict requirements for application availability, and the network environment is stable enough.

If your environment, the network quality is not high, there will indeed be instantaneous interruptions, and sometimes the detection signal will not respond in time, then you can choose to set the detection threshold to be looser. The maximum recommended setting is 20 times of signal loss before failover. This is a Microsoft best practice. The advantage of setting a loose detection threshold is to ensure that the node will not be failed over under normal circumstances, and 20 signal detection failures are allowed. Maybe is due to the long distance of the node or the case of instantaneous interruption. The disadvantage is that if the threshold is too loose, the downtime will be delayed. For example, if the node does go down, but the failover is triggered after 60 tests, the failover should have been done after 5 tests. Therefore, the additional 55 tests are additional downtime.

For setting a loose threshold for node detection, one level is to solve the situation of cross-subnet detection, such as 2 nodes in Beijing and 2 nodes in Guangzhou, a cluster with four nodes across regions and subnets. because the network link is too complex, five tests sometimes cause detection failure and trigger failover, so you should adjust their detection threshold, but no more than 20 at most. In addition, the loose detection threshold is also adjusted to deal with the problem of instantaneous interruption. If the quality of the network environment is not high and stable, there will be inevitable packet loss, but you do not want to fail over frequently. Then you can also adjust the detection threshold to 20.

For the case of instantaneous interruption, there is a new technology in WSFC 2016, namely VM resiliency, which is turned on by default. For virtualized clusters, Microsoft defaults to the problem of instantaneous interruption in the environment, so two new states are added to virtual machine resources. If an instantaneous failure occurs within 240s, the node enters an isolated state and the virtual machine is in an unmonitored state. The virtual machine can still run or pause normally. If the node is quarantined three times within an hour, the cluster believes that there is a problem with the node and needs to be thoroughly investigated, so it will be placed in a quarantine state and migrate all virtual machines to other nodes in real time before 7200 seconds. The quarantine node will be checked and will not join the cluster normally.

The VM resiliency technology in WSFC 2016 mainly puts forward a new idea. Microsoft knows that there are many instantaneous interruptions for virtualized clusters, which often lead to failover due to network quality, so it adds a new process from isolation to quarantine to normal repair. But this feature is only valid for VM resources, and the default isolation time is too long. If you want to use this feature, you need to know it first. Lao Wang wrote in detail on the previous blog, and then need to adjust these thresholds, 240s, quarantine for 3 times, quarantine for 7200 seconds. If you don't want this feature, you can turn it off. The cluster returns to the previous failover with full reference to the health detection threshold setting. It should be noted that health detection is effective for nodes. Once the health detection threshold is reached, all node resources will be failover, while in the isolated state, only virtual machines will be left unmonitored. So you can choose the function to use according to the actual scene.

OK, the theories involved have been explained clearly, and then it is much easier to look at the operation command.

The parameters involved in adjusting the health detection threshold are as follows

Delay is the detection frequency, and Threshold is the detection threshold.

In addition to what is listed in the following table, there is another situation in the 2012 era, that is, if the node adds the Hyper-V role as a Hyper-V cluster, then the SameSubnetThreshold will be automatically set to 10 and the CrossSubnetThreshold will be automatically set to 20. It may be because Microsoft considers that there may be many virtual machines on the Hyper-V, 5 will lead to frequent failover, the failover time is long, and the instantaneous outage is also taken into account. 10 and 20 are the best.

Parameter WSFC2012R2WSFC2016 maximum SameSubnetDelay1 second 1 second 2 seconds SameSubnetThreshold5 heartbeat 10 heartbeats 120 heartbeats CrossSubnetDelay1 seconds 1 seconds 4 seconds CrossSubnetThreshold5 heartbeats 20 heartbeats 120 heartbeats CrossSiteDelayNA1 4 seconds CrossSiteThresholdNA20 heartbeats 120 heartbeats

Before WSFC 2016, when we wanted to adjust the node health detection threshold, there were only the same subnet and cross-subnet. These four options can be set, and CrossSite is a new feature.

The Site here is based on the site we defined by defining the failure domain, so cross-site heartbeat detection can be placed under site awareness.

In WSFC 2016, if we set up these three scenarios, namely, the same subnet, cross-subnet and cross-site, their effective priorities are also different in different scenarios.

If the cluster node is in the same site and in the same subnet, the threshold of the same subnet takes effect.

If the cluster node is in the same site and two different subnets, the cross-subnet threshold takes effect.

If the cluster nodes are in different sites and different subnets, the cross-site threshold will override the cross-subnet threshold.

If the cluster nodes are in different sites and in the same subnet, the cross-site threshold will override the threshold of the same subnet

The biggest difference between here and before is that there are more concepts of different sites. If you decide that the nodes are at different sites, then the cluster will think that they are far away from each other and need to apply the detection threshold of different sites, even if they are on the same subnet.

This is very suitable for the stretch vlan scenario, that is, although some cluster nodes are far away, they exist under the same subnet, so you cannot control it. It is said that other nodes have to carry out relatively loose signal detection for this remote node, because they are all in the same subnet, so only SameSubnetThreshold will be applied, but this node will also carry applications. If it is because the network link is too long or instantaneous interruption, the signal detection is inaccurate. If a failover occurs, there is no way to control it. In the end, we can only ask to change the subnet, but in WSFC 2016, we can logically define the node to another Site, so that, although it is still on the same subnet, once the remote node is tested for health, a loose detection policy across Site will be applied.

# obtain settings related to health detection of WSFC nodes

Get-Cluster | fl * delay*

Get-Cluster | fl * threshold*

# adjust the heartbeat detection threshold in the same subnet to 15

(Get-Cluster) .SamSubnetThreshold = 15

# adjust the threshold of cross-subnet heartbeat detection to 15

(Get-Cluster) .CrossSubnetThreshold = 15

The current environment continues the previous blog. HV01 HV02 belongs to Beijing site, HV03 and HV04 belong to Shanghai site.

Looking at ClusterLog, you can see the health inspection topology built by NETFT

Apply SamSubnetThteshold detection strategy for the same subnet and the same site

Apply CrossSiteThteshold detection strategy to different subnets across sites

Apply CrossSiteThteshold detection strategy for different Site in the same subnet

# manually move HV01 to Shanghai site

Check again and find that the CrossSiteThteshold test standard has been applied from 18.0.0.9 to 18.0.0.10!

Apply cross-Site node detection strategy for different subnets, but one of the nodes exits the Site.

Apply the cross-subnet node detection strategy for the same site, but one of the nodes is moved across subnets

Move HV01 back to Beijing site

Modify HV01 to Shanghai subnet IP

Check ClusterLog and find that CrossSubnetThreshold has been applied to the health detection of other nodes to HV01 nodes.

Once the 18.0.0.0 network segment fails heartbeat detection, NetFT will reroute other networks for health detection. Once the cross-subnet network segment is selected and is in the same Site, then the cross-subnet health detection policy will be applied.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.