Shulou(Shulou.com)06/01 Report--

The bebeOracle CSSD (Cluster ServicesSynchronization Daemon) process monitors the health of cluster nodes through two independent heartbeats: network heart beat and disk heart beat

Network heartbeat: a thread of the CSSD process of each node in the cluster sends a TCP protocol heartbeat signal to all nodes in the RAC (including itself) through inteconnect every second. If there is a problem with the heartbeat signal transmission, the self-correcting mechanism of the TCP protocol will retransmit the packet. In this case, oracle does not pay attention to the transmission details.

Disk heartbeat: the CSSD process of each node in the cluster maintains the heartbeat signal by reading and writing data at a specific location on the voting disk. In addition, each node monitors the heartbeat data blocks maintained by the CSSD processes of other nodes. If a node is found to have not written to the disk heartbeat within a specified period of time, the node is judged dead. If a node is in an unknown state, other nodes will also be expelled from the cluster by updating its kill block status on its voting disk.

In general, the network heartbeat starts every second, and if a node does not respond beyond the time set by the parameter css_miscount time, it will be kicked out of the cluster. Similarly, each node in the cluster reads and writes a specific area of voting disk per second, and the node with a timeout response is also kicked out of the cluster. Here are the possible conditions and consequences of the heartbeat faced by the node:

Brain fissure: in a RAC environment, all oracle instances communicate with each other over a high-speed private network. The links in this private network should be configured to be redundant and used only for oracle block transmission between instances. Take the RAC cluster consisting of two instances running on two machines as an example. When the servers in the RAC cluster are running normally and the two instances themselves are running normally, when the two oracle instances cannot communicate with each other through the VPC, if no prevention is taken, the two instances will run independently and provide services. Because the two instances lose communication with each other, they will think that the other is dead, so they no longer use the original mechanism to protect the shared data, which will lead to confusion of the shared data. This is a brain fissure.

IOfencing: if the operating system is normal but the oracle instance fails, the recovery process may execute the remaining writes from the previously failed instance. Because write operations cannot maintain the correct write order in this case, they may cause data confusion and must be prevented. Therefore, in a clustered environment, if a node fails, it must be prevented from writing to shared storage. This method is called IO fencing.

Simple majority principle: in order to maintain the normal operation of the RAC cluster, more than half of the configured arbitration disks must operate properly.

The following two examples are used to understand the mechanism of the arbitration panel when there is a problem with the heartbeat.

Example 1, as shown in the following figure, a three-node cluster with three arbitration disks. The network communication (network heartbeat) between Node3 and the other two nodes is interrupted. However, by reading the status of the arbitration disk, the CSSD process finds that all nodes can write the arbitration disk, indicating that there is a brain fissure. Node 1 and node 2 then update the node 3 kill block status on the arbitration plate. Then the CSSD process of node 3 knows that it should exit the cluster by reading the corresponding kill block on the arbitration disk. So execute IO fencing, and then close yourself.

In example 2, in a two-node cluster with three arbitration disks, there is a problem with disk heartbeat, resulting in node 1 having access to two arbitration disks and node 2 having access to one arbitration disk. In this case, based on the simple majority principle, node 1 (two arbitration plates) will issue a suicide instruction to node 2. In this way, node 2 exits the cluster.

A further problem in example 2 is that if there is an arbitration disk failure that makes it inaccessible to both nodes, but the other two arbitration disks are accessible to both nodes. In this case, how to determine which node to exit? At this point, because the arbitration disk that can be accessed by the two nodes satisfies the simple majority principle, if the Network heartbeat is normal, the RAC will run normally and will not kick out any nodes. What happens if, unfortunately, there is another arbitration disk failure that makes it inaccessible to both node?

At this time, the network is normal, but the arbitration plate is lower than the simple majority, so the RAC must be reconstructed, but the conditions of the two nodes are the same. RAC chooses to keep the node with the lowest node number, in this case, the node1 is retained and the node2 is kicked out.

Most of the content of this article comes from the English website https://nitishanandsrivastava.wordpress.com/, which needs scientific access to the Internet. I have cut and extended it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.