Shulou(Shulou.com)06/02 Report--

We can see that Lao Wang spent three pages on arbitration. Lao Wang thinks it is worth it, because in Lao Wang's view, managing a WSFC cluster is nothing more than designing the architecture, and then maintaining the availability of the cluster on a daily basis, performing cluster detail management, detailed log analysis, update and migration, etc., among which maintaining the continuous availability of the cluster is the most common thing we see in the management cluster, and whether the cluster is available or not, and arbitration. Witness, voting is directly related to this.

Very often, if you don't know what arbitration is, you don't know what's going on when the cluster stops, so Lao Wang spends more space on carefully dissecting the arbitration technology, trying to make people understand it thoroughly. Two more pages have been spent in the form of scenarios to combine 2012 dynamic arbitration technology with other cluster arbitration technologies to reproduce the operation in some scenarios. I believe friends who have seen it carefully will get something.

Well, friends who have seen it may think that dynamic arbitration is a good technology, ah, to help us automatically adjust the vote to ensure that the cluster can stand to the last node, and most people will say that 2012 will start to have dynamic arbitration. is it certain that the cluster can be supported to the last node? In fact, it is not necessarily, we still need to look at it calmly. After Lao Wang's research, it is found that there are two scenarios in which dynamic arbitration will not be supported to the last node.

In the first scenario, Lao Wang also mentioned it in the first article of dynamic arbitration, and attached a picture description, assuming that the current cluster still has two nodes running without witness disk, using majority node arbitration and enabling dynamic arbitration. By default, dynamic arbitration randomly selects a node to remove the vote.

At this time, it is divided into three situations

Case 1. If the non-voting node is powered off, the cluster can operate normally.

Case 2. The operating system of the voting node shuts down normally, the number of votes can be exchanged normally, and the cluster can operate normally.

Case 3. The voting node is powered off, the cluster cannot run, and the number of votes is too late to be exchanged, so compulsory arbitration is required.

Once there is situation 3, in fact, Lao Wang feels that the situation will be very common. once node 1 is selected and node 2 does not vote, suddenly node 1 is powered off, and node 2 will go offline because it has not been swapped for node voting. The whole cluster is shut down, and only forced startup can be made at this time.

Therefore, in the case of most nodes and no witness disk, there is a certain probability that whether the cluster can make it to the last node or not. There is a 66% chance that you will encounter situations 1 and 2, and the cluster will operate normally. If you encounter case 3, you will lose the effect of standing to the last node, and you still need to use compulsory arbitration.

Microsoft must have discovered this problem, so Microsoft added the technology of dynamic witness to the dynamic arbitration technology at the beginning of 2012R2, that is, in the case of witness, we can always adjust the vote of witness dynamically according to the change of nodes to ensure that the cluster is always odd. No matter what the situation is, even in case 3 mentioned above, there are two nodes left, and one of them suddenly lost power. But as long as another node can contact the witness, the cluster can still stand to the last node.

It's also true to say that if witness is always there, the cluster can indeed support to the last node, but if we consider it in the light of the actual environment, in case we use shared witness or disk witness, we need to ensure their availability, what will happen if we suddenly witness that we can't get in touch with each other, and whether my cluster can still support to the last node.

According to Lao Wang's actual test, a problem that is easy to be ignored has been found.

I show it to you through the actual test.

Time Node 1: four nodes of the cluster + shared witness all survived, with a total of five votes

Time node 2 downtime one node, dynamic witness automatically remove one vote, a total of three votes

Time node 3 downtime a node dynamic witness automatically adds one vote, a total of three votes

At this time, a network failure occurs and the shared witness is unable to connect. We directly cancel the shared status of the shared witness.

At this time, if you run the check vote command on the two surviving nodes, you can see that it is still 2 nodes + 1 witness vote.

Although the log has mistakenly reported that the shared witness resource cannot be accessed, the event managers of both nodes will be jammed by the log of file sharing failure.

Time node 4, the remaining two nodes are down.

You can see that the whole cluster has been shut down.

At this point, only mandatory arbitration starts the cluster node.

After the forced start of the cluster, the normal communication between node 1 and node 2 comes online. You can see that the cluster is still flooded by logs that file sharing cannot go online. We can try to configure the cluster arbitration mode to non-witness, that is, majority node mode for mitigation.

The attempt to configure most nodes will fail, reminding us that the cluster cannot form a quorum now.

At this time, only when one more node joins, the majority arbitration can be formed normally, and it can be configured as the majority node arbitration model.

At this time, when the third node goes down again, the group assembly dynamically arbitrates one of the two nodes to ensure that the cluster always votes odd, and the problem caused by the failure of the shared witness has been solved. at this time, there is a 66% chance that the two nodes can hold on to the last node without compulsory arbitration.

We can see that the key here is that the time node three, the three nodes become two nodes, and then the shared witness suddenly expires. In an ideal situation, the cluster dynamic arbitration should sense the failure of the shared witness, and then re-adjust the number of cluster votes. Randomly select one vote to survive.

However, the actual situation is that when the shared witness suddenly fails, the cluster arbitration does not feel it, and then make a dynamic arbitration adjustment. The check command will find that it is still 2 node tickets + 1 witness ticket. In fact, the shared witness is gone at this time. Check the log to see that the shared witness has failed.

However, the cluster did not remove the witness vote, nor did it dynamically adjust to 1 vote, so if another node cluster goes down, Lao Wang guesses that the key here is that when the shared witness fails, the status is caused by "failure". The cluster did not remove the witness vote, nor did it dynamically adjust the node vote. This is very dangerous. in this abnormal working situation, one more bad node will force the cluster to start.

So Lao Wang is wondering if dynamic arbitration favors disk witness and does not attach importance to shared witness? Does shared witness have this problem besides time zoning? So soon Lao Wang tried to witness the disk again.

Time node 3: in the case of disk witness, there are three nodes left in the cluster to survive, when one node is down, and then the cluster disk is disabled.

At this time, although the witness disk has been disabled, the cluster will not immediately perceive that the visible state is still online.

After a period of time, the status will become online and the arbitration disk will try to hang on each node one by one according to the failure strategy, but this shows that the number of votes and the number of node votes have not been dynamically adjusted.

Finally witnessed that the disk became a failure state, but still did not adjust the number of witness votes and node votes.

Therefore, it can be seen that when the witness disk suddenly fails and cannot be accessed, the dynamic arbitration of the cluster is no longer working properly, regardless of whether the witness disk becomes online suspended or fails, as long as one of the nodes is broken. after a while, the cluster will decide that the current 55 cannot form arbitration and shut down the cluster.

The last node tried to form a cluster but failed after a few seconds because there was no arbitration operation that could not be performed and there was no majority vote.

So you can see that both shared witness and disk witness are faced with this problem.

That is, when the cluster is left from 3 nodes to 2 nodes, the cluster will suddenly lose contact, and the cluster will not adjust the vote dynamically. When 1 node goes down again, the cluster will be closed, requiring manual compulsory arbitration, and should be switched to the majority node arbitration mode to prevent it from happening again.

After the shared witness is lost, the shared witness failure will be written directly in the log, but the dynamic arbitration will not adjust the witness and the node's vote.

On the other hand, according to the disk policy, the disk witness will try to suspend online first, and then the status will fail, but dynamic arbitration will never dynamically adjust the witness and node voting after the cluster disk is lost.

Unless the disk witness state becomes offline, in an ideal case, the disk witness failure will be offline, and then the vote is released, and the cluster senses the loss of witness votes and dynamically adjusts the number of votes of one node. Now the cluster is odd-one vote.

However, according to Lao Wang's observation, when 3 leaves 2 and disk witness suddenly fails, the state of disk witness will always fail and will not become offline.

If the disk witness situation makes the offline, Lao Wang tries to find that only when the disk is in a normal state, you can manually change the status to offline. In the case of the disk witness being offline normally, we will remove the witness vote as we expected, and then randomly remove the vote of a node.

Therefore, when this kind of witness suddenly disappears, the problems faced by the shared witness and the disk witness are the same, and there is no biased relationship. Lao Wang feels that this should be a bug of the dynamic arbitration detection mechanism. When the witness is suddenly lost, it can be set to a failure state, but it should be able to dynamically remove the number of votes witnessed in the failure state and dynamically adjust the node vote. The loss of contact with a witness should not lead to the abnormal work of the whole dynamic arbitration.

Therefore, when using dynamic arbitration, you need to consider the following two problems that may be encountered but easily overlooked.

Pure use of the majority of nodes, dynamic arbitration to adjust the number of nodes, when there are 2 nodes, there is a 66% chance that the cluster can survive to the last node normally. When the selected voting node suddenly goes down, the cluster shuts down. You need to manually force the cluster to start.

Use witness plus node vote, dynamic arbitration + dynamic witness. When there are 2 scenarios left, witness suddenly lost contact, witness will not remove its own vote, dynamic arbitration will not automatically adjust to 1 vote, if one more node is down, the cluster will shut down. When the other two nodes are restored, you can manually switch to the majority node arbitration model. In this way, when there are 2 scenarios left in 3, the vote will be automatically adjusted to 1 vote, and the chance of survival to the last node scenario will be about 66%. Then, since we are forced to start the cluster, even if we restore it later when we witness it, the forced-started cluster database will also cover the database that witnessed the disk.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.