Shulou(Shulou.com)06/02 Report--

Preface

In the last two days, I logged on to the server and found that it prompted "No space left on device" when downloading files with wget, and even prompted this error when using the tab key to complete.

I have encountered this problem once before, because the disk space is full and a new file cannot be created. Normally, this problem can be solved by deleting some files to free up space.

When I use the df command to view partitions, the results are as follows:

# df-hFilesystem Size Used Avail Use% Mounted on/dev/vda1 29G 29G 0% / udev 10M 0 10m 0% / devtmpfs 101M 232K 100m 1% / runtmpfs 5.0M 05.0M 0% / run/locktmpfs 405M 0405M 0% / run/shm

Seeing here, I thought the disk was really 100% occupied, so I checked the occupancy of each directory:

# du-sh / * 8.8m / bin30M / boot0 / dev5.3M / etc24K / home0 / initrd.img205M / lib4.0K / lib6416K / lost+found8.0K / media4.0K / mnt4.0K / opt0 / proc2.5G / root232K / run5.2M / sbin8.0K / srv0 / sys4.0K / tmp2.6G / usr1.8G / var0 / vmlinuz

Obviously, the total disk footprint is completely less than 10G, and the disk is not theoretically full.

In this case, it is likely that there are too many small files, resulting in a sharp increase in inode, running out of space in the inode area.

If this is the case, a new file cannot be created because a new inode cannot be created to store the meta-information of the file, even if there is free disk space.

Therefore, I validate it with the df command:

# df-ihFilesystem Inodes IUsed IFree IUse% Mounted on/dev/vda1 1.9m 299K 1.6m 17% / udev 123K 299 123K 1% / devtmpfs 126K 249 125K 1% / runtmpfs 126K 4 126K 1% / run/locktmpfs 126K 2 126K 1% / run/shm

As you can see, the inode area is only a small part of it, and there is still a lot of unused space, so it is not a problem that the inode area is full.

When I got here, my heart was very depressed. This problem directly leads to the failure of Apache, MySql and other services, and the server is basically unavailable, but the reason is complicated and confusing.

Finally, the server provider told me that another possibility is that when some files are deleted, they are still occupied by other processes. At this time, the files are not really deleted, but are marked as deleted, and the files will not really be removed from disk until the end of the process.

So I looked at the files occupied by the process through the lsop command:

# lsof | grep deletedmysqld 1952 2982 mysql 5U REG 254Power1 0127 / tmp/ibzMEe4z (deleted) mysqld 1952 2982 mysql 6u REG 254 Magazine 1 0146 / tmp/ibq6ZFge (deleted) mysqld 1952 2982 mysql 10u REG 254 mysql 1 0150 / tmp/ibyNHH8y (deleted) apache2 2869 root 9u REG 254 1 0168 / tmp/.ZendSem.2w14iv (deleted) apache2 2869 root 10w REG 0Magazine 16 0 11077 / run/lock/apache2/rewrite-map.2869 (deleted)... python 3102 root 1w REG 254 122412342132 264070 / var/log/nohup.out (deleted)

Finally found the culprit, the original is running in the background of the Python script, continue to save the output to the / var/log/nohup.out file, the file size actually reached 20 gigabytes!

After running the script in the background in the previous stage, I left it alone. It is estimated that I deleted the nothup.out file during the Python run. Because the file is occupied, it can only be marked as deleted first, but not really deleted, resulting in a full disk.

This lesson reminds me that you can't simply leave the task in the background, especially when using the nohup command, all the output will be constantly added to the same file, and if the process does not terminate itself, it may cause the output file to fill the entire disk.

Summary

The above is the whole content of this article, I hope that the content of this article can bring some help to your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.