Shulou(Shulou.com)06/01 Report--

In a core trading library, the alarm IOwait is more than 30%, which seems to be a very common alarm, but it is actually a mystery. The login host finds that there are a lot of RMAN backup scripts running.

The usual task is less than an hour, and I have been running for more than 6 hours.

The backup task is mounted by nfs. It is suspected that there is a problem with nfs.

Sure enough, when entering the mount directory, ll could not display the results normally and stuck, and then went to the backup server, that is, nfs server, to see if there was anything abnormal. It was found that a monitoring script deployed before, load and io were very low in normal times. It's not the same as today's failure period.

Return to backup client server, ps-ef | grep nfs found that there are many cat processes

[root@trandb1 log] # ps-ef | grep nfs

Root 9700 20 2017? 00:00:00 [nfsv4.0-svc]

Oracle 88889 88888 0 10:05? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9868_1

Oracle 90224 90223 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9888_1

Oracle 90566 90565 0 10:06? 00:00:00 cat. / nfs/full_data_TRANDB_20190619_9872_1

Oracle 90571 90570 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9869_1

Oracle 90576 90575 0 10:06? 00:00:00 cat. / nfs/full_data_TRANDB_20190619_9872_1

Oracle 90584 90583 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9868_1

Oracle 90588 90587 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9884_1

Oracle 90593 90592 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9885_1

Oracle 90597 90596 0 10:06? 00:00:00 cat. / nfs/full_data_TRANDB_20190619_9865_1

Oracle 90606 90605 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9881_1

Oracle 90616 90615 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9871_1

Oracle 90626 90625 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9887_1

Oracle 90631 90630 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9888_1

Oracle 90641 90640 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9871_1

Oracle 90645 90644 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9880_1

Oracle 91999 91998 0 10:06? 00:00:00 cat. / nfs/full_data_TRANDB_20190620_9883_1

Oracle 92488 92487 0 10:06? 00:00:01 cat. / nfs/full_data_TRANDB_20190620_9880_1

Oracle 93837 93836 0 10:07? 00:00:00 cat. / nfs/arch_TRANDB_20190620_9890_1

Oracle 94011 94010 010: 07? 00:00:00 cat. / nfs/full_data_TRANDB_20190620_9886_1

Oracle 94238 94237 0 10:07? 00:00:01 cat. / nfs/full_data_TRANDB_20190619_9865_1

Root 98024 17863 0 10:09 pts/7 00:00:00 grep nfs

Root 130976 20 2017? 00:00:00 [nfsiod]

The pid is dropped through the operating system kill, but it will be derived immediately. After umount dropped the directory, it was gone. I haven't found out the cause for the time being. Record it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.