Shulou(Shulou.com)06/03 Report--

Today, sorting out the previous operation and maintenance information, I found a blade server (running vmware virtualization) accident handling process, so record it, memo.

I. event handling process

At 14:10, I was informed by the computer room operation and maintenance engineer that there was a downtime of many servers on the Opmanager monitoring system, and they were all virtual machines.

Notify the operation and maintenance engineer in the computer room at 14:12 to check whether there is an alarm on the HP blade server, and log in to vcenter remotely for inspection. Remote view shows that an alarm appears in ESX04 (10.203.11.64). The alarm message is shown below:

Notify the engineer at 14:15 that there is an alarm on ESX04, then confirm that the blade server is alive, and enter the computer room to confirm that there is an alarm on the hardware on the device.

14:16 check the logical network interface for anomalies

As shown in the following figure, two network cards are found to be offline

14:18 inspection of other blades, found that the corresponding network card ESXI02, found to be normal

Log in to the HP Blade Management console at 14:20 and no server alarm information is found.

14:19 try to change the mode of vmnic6 and vmnic7 network cards with reference to other EXSI. This operation will not take effect.

Changing the network card mode does not take effect

14:27 manual migration of virtual machines to other hosts at ESX04 failed.

14:58 shut down all virtual machines on the ESX04 host

Restart the ESXI host at 15:20, and HA automatically migrates the open virtual machine to other EXSI hosts to start

15:30 after a successful startup of the ESX04 host, vsphereHA failed to automatically migrate the virtual machine back to the ESX04 host

15:50 manually migrate some virtual machines back to the ESX04 host and observe the running status.

Second, log analysis

1. Log in to the command line of ESXI remotely and view the log of vmkernel:

Note: because esxi4 uses utc time, the time shown in the log will be 8 hours slower than the time time.

/ var/log # cat / var/log/vmkernel.log | grep '2014-12-18) WARNING: ScsiDeviceIO: 1211: Devicenaa.60014380064900f30000800000e40000 performance hasdeteriorated. I latency increased from average value of 3303 microseconds to68755 microseconds.2014-12-18T03:31:54.595Zcpu8:16392) ScsiDeviceIO: 1191: Device naa.60014380064900f30000800000e40000performance has improved. O latency reduced from 68755 microseconds to 13691microseconds.2014-12-18T03:32:32.643Zcpu12:17017) MigrateNet: vm 17017: 2061: Accepted connection from 2014-12-18T03:32:32.643Zcpu12:17017) MigrateNet: vm 17017: 2131: dataSocket 0x4100253292f0 receivebuffer size is 5635602014-12-18T03:32:32.644Z cpu12:17017) WARNING:Migrate: 26262: Invalid message type for new connection: 542393671. Expecting message

As the above log shows: at 13:27, the performance of the host began to decline, and the I-Pot O delay became larger.

2. Check whether there are any related alarms in 10.203.11.100:

As shown in the figure above, it is prompted that the network card status of the esx04 host is incorrect.

3. Other logs collected are as follows, and no exception has been found so far.

The whole process is basically completed at this point, and there is no obvious feature in all the blade servers, which is the only one that has an occasional fuss.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.