Shulou(Shulou.com)05/31 Report--

This article will explain in detail the example analysis on Freeshell's identification of NFS frequent jams. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

Freeshell has a two-hour NFS jam. In fact, since the external disk was enabled in August 2014, there have been frequent NFS jams lasting tens of seconds or even minutes, during which the freeshell on the external disk cannot perform any operation, and some freeshell will shut down due to disk operation timeout. It was always thought that NFS bug caused the deadlock, but the cause of the fault was not located and could not be reproduced.

On February 3, through the method of grabbing the package and the strace nfsiod process by tcpdump, it was identified that the problem with NFS server not responding was due to the busy external hard drive (a Seagate 2T green disk).

During the period of NFS jam, the average latency of external disk reads is up to 600ms and the average latency of writes is 1400ms. When I first set the NFS mount parameters, I didn't take into account the busy disk. I just thought that the network delay should not exceed 1 second, so I set the timeout timeo=10, which means 1 second timeout. A NFS request may need to be broken down into multiple disk read requests, and the sum of these requests is likely to be more than 1 second, that is, most read and write requests time out before they are sent to disk, creating the illusion that the NFS server is unresponsive.

Here are the earlier mount parameters:

Vers=3,rw,rsize=32768,wsize=32768,tcp,timeo=10,retrans=5,soft,intr,sec=sys,lookupcache=all,ac,nocto

The timeout is now modified to 30 seconds (timeo=300).

Vers=3,rw,rsize=32768,wsize=32768,tcp,timeo=300,retrans=5,soft,intr,sec=sys,lookupcache=all,ac,nocto

The fstab was modified on February 3, but since the mounted NFS cannot modify the mount parameters (see man nfs), virtual machines on all external disks need to be shut down before NFS can be remounted. Node 1 just hung up on February 4, so all other nodes were rebooted, and the NFS parameters were updated.

This is the end of the article on "example Analysis of Freeshell identifying frequent NFS jams". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.