Shulou(Shulou.com)06/01 Report--

Recently, there have been feedback from friends and little y:

One of their IBM X86 servers (now owned by Lenovo) suffered hardware damage, and when the maintenance staff collected diagnostic logs to the manufacturer through the management port, the server ran a good set of ORACLE 11.2 RAC databases.

Suddenly a node restarted.. What's going on?

Hearing this, Xiao y basically guessed the cause. Similar problems have been analyzed and dealt with several times before, and the analysis process is not complicated, but I did not expect that similar failures are still happening.

Therefore, it is necessary to suggest this risk!

Here, Xiao y will share with you a case handled in the past. At the end of the article, we will give specific risk tips for the combination of X86 server and RAC. I hope you can understand and prevent early, avoid stepping on the pit and hurting others and yourself.

Here comes the problem!

On Friday, at 11:00 in the evening, the phone rang. It was the phone of a service provider. It seemed that something big had happened. I was refreshed.

"Little y, a set of 11.2.0.3 2-node RAC on linux

The Node 2 database restarted itself this afternoon, and then got up by itself.

I hung up again a few hours ago, but I haven't got up yet!

The private network IP of the two nodes ping each other for a while, and you can communicate with ping!

Why don't you connect it remotely and deal with it? "

BTW:

Yes, you read it right. It's the service provider, not the end customer.

Xiao y's company not only provides database expert services directly to customers, but also provides third-line support for other service providers and software developers, but Xiao y can't sleep at night when there is a lot of performance pressure. I also ask you brothers to help recommend and introduce ah.

Analysis and recovery of faults

Verification Node 2 failed to start

Time is urgent. After remote connection, it is found that Node 2 is indeed dead, so startup starts the database directly and has a look.

In theory, after the startup command goes down, you can see the information that SGA allocates and starts to mount soon.

But the order has been down for a minute, and there is no output, which is really not good.

In the end, the startup command remained unresponsive for a long time after typing, and exited after about 10 minutes after reporting an ORA-03113 error.

As shown in the following figure:

It seems that an exception has been encountered during database startup and you need to continue to analyze the alert log.

Check the alert log of the Node 2 database

Intercept the key information of altert log, as shown in the following figure:

You can see:

Because the Lmon process of node 2 timed out to communicate with the LMON process of node 1 through ipc, in short, the RAC of the two nodes could not communicate, so they could not join the cluster. Therefore, it is necessary to further check whether the private network communication between the two nodes is normal.

Obtain the IP address of the private network communication between the two nodes

They mentioned earlier that the private network IP of the two nodes can be connected by ping. I guess they have the wrong IP in ping.

Because, starting from 11.2.0.2 (inclusive), ORACLE private network communication no longer directly uses the address we configured on the private network card (such as 192.168), but uses the address of the 169.254 network segment bound on the private network card by ORACLE after GRID startup. Confirmed that they really ping is 192.168 this IP, this IP can PING communication is not enough.

Issue the following command to obtain the IP address used for communication between the two nodes in the private network as follows:

In other words, the real addresses of the two RAC nodes for private network communication are:

Node 1 uses a private network address of 169.254.220.75 instead of 192.168.x.x

Node 2 uses a private network address of 169.254.106.133 instead of 192.168.x.x

In other words, the IP before and after GRID starts is as follows:

Node1

Node2

Remarks

Bond0

192.168.1.1

192.168.1.2

IP that exists before and after GRID startup

Bond0:1

169.254.220.75

169.254.106.133

IP that exists only after GRID is started

RAC and ASM communication use

Check whether the private network communication between the two nodes is normal.

As can be seen from the above picture:

The actual private network communication address of 169.254 is not ping between the two nodes! This is why the database instance of Node 2 cannot be clustered!

Thinking time

At this point, we might as well summarize it with a table:

The bond0 is a VPC Nic, 192.168 is manually configured, and the 169.254 IP is tied up after GRID:

Node1

Node2

Bond0

192.168.1.1

192.168.1.2

You can ping each other.

Bond0:1

169.254.220.75

169.254.106.133

Ping can't communicate with each other.

What's going on?

In fact, it is very simple, don't worry, the cause of the problem is behind, when to turn down, it is up to you to decide.

……

……

……

……

……

……

……

……

……

……

……

……

……

……

……

……

Then what is the reason for the disconnection between the two addresses?

We further examined the routing of the two nodes and found an exception. As shown below

Check and find that the private network route on node 1 is lost, resulting in the private network between the two nodes can not communicate normally, and then can not join the cluster normally.

And there is a 169.254 route on node 2!

Solve the problem temporarily

At Node 1

# route add-net 169.254.0.0 netmask 255.255.0.0 dev bond1

After implementing the solution on node 1, the node 2 database instance starts normally and the problem is solved.

At this point, some students asked, can we stop all the GRID of the two nodes and restart them all to restore them?

The answer is yes!

Because GRID is restarted, the IP of 169.254 will be re-bound in bond0, and the route 169.254.0.0 will be added.

2016-06-02 23V 05JV 47.457:

[crsd (10403)] CRS-2772:Server 'node2' has been assigned to pool' ora.RACDB'.

2016-06-03 19Suzhou 3648.517:

[/ oracle/app/11.2.0.3/grid/bin/orarootagent.bin (8641)] CRS-5018: (: CLSN00037:)

1) 19:36:25, the node 1 USB0 network card is assigned the IP of 169.254.95.120

2) 19:36:48, the Node 1 orarootagent process found that the 169.254 IP assigned on the USB0 collided with the IP communicating with the RAC cluster, and then deleted the 169.254 route, causing the two nodes RAC to fail to communicate.

3) 19:41:24, node 2 reported an IPC communication timeout and was expelled from the cluster

4) due to the loss of 169.254 of the route of node 1, node 2 is unable to communicate with node 1 and has been unable to join the cluster

5) after manually adding 169.254 routes to node 1, the problem is solved.

It is not difficult to see that this behavior is normal!

We use "Removed unused HAIProute: 169.254.95.0" as the search keyword on METALINK, and the following article on MOS also describes this behavior, which has been speculated to be verified.

Oracle RAC HameA Failure Causes Oracle Linux Node Eviction On Server With IBM Integrated Management Module (IMM) (document ID 1629814.1)

Risk hint

Risk hint

In a version with ORACLE11.2.0.2 or above deployed

Because the ASM and DB of the cluster use 169.254.x.x as the IP of the cluster private network communication

[after GRID starts, bind 169.254.x.x this IP to the private network card and add the route of 169.254.0.0]

At present, it is known that after the IBM X86 server has installed the operating system, there is a network card such as USB0, which is used to communicate with IMM. IMM is the management port of the server, and the hardware is managed through the LAN of the USB network.

When the USB0 network card is activated, 169.265.95.120,118 this IP will be allocated, which will cause the RAC cluster route to be destroyed, which will lead to the failure of DB/ASM unable to communicate and restart / node eviction.

# cat / var/log/messages* | grep dhclient | grep "bound to 169.254"

If so, enter the prevention link.

2) issue the following command to check whether the system currently has the IP of the network segment in which the non-RAC private Nic is assigned 169.254

# ifconfig-a

..

Usb0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX

# vi

# / sbin/ifdown usb0

# / sbin/ifup usb0

# / sbin/ifconfig usb0

This article is reproduced in Zhongyi Antu.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.