Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how Linux finds large files and clears them safely". In daily operation, I believe many people have doubts about how Linux finds large files and clears them safely. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts of "how to find large files and clear them safely". Next, please follow the editor to study!

1 case description?

In the service online environment, there will be some alarms caused by high disk utilization. It may be that a log file is too large and failed to clean up and recycle in time. How to find large directories and files?

How to clean up large files safely?

How to release the occupied disk space quickly?

2 Command 1 (the best command for catalog statistics sorting)

(here, take the current directory. / as an example, statistics top5)

[du-k-- max-depth=1. / | sort-nr | head-N5]

[root@test-001 /] # du-k-- max-depth=1. / | sort-nr | head-n5137450839518./6785876./data2182577./usr1830341./home446856./var//du-k # displays the size of a directory or file in kB; / / du-- max-depth=1 [directory] # shows only the size of the first layer directory under the specified directory (excluding individual files); / / sort-nr # sorts by number size from large to small / / head-n5 # shows the first 5 lines of the content, which is the top5 content shown here; 3 Command 2 (most practical, catalogs and files are sorted together)

(here, take the current directory. / as an example, statistics top5)

(1) particulars and descriptions of orders

[du-sk *. / | sort-nr | head-N5 | awk-F'\ t' {if (1024 * 1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024 * 1024) {printf "%. 2fT\ t% s\ n", $1 / (1024 * 1024 * 1024), $2} else if (1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024) {printf "%. 2fG\ t% s\ n", $1 / (1024 * 1024) $2} else if (1024 * 1024 > $1 & & $1 > = 1024) {printf "%. 2fM\ t\ t% s\ n", $1 2fM 1024, $2} else {printf "% sk\ t\ t% s\ n", $1, $2}}']

[root@test-001 /] # du-sk *. / | sort-nr | head-n5 | awk-F'\ t'{if (1024 * 1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024 * 1024) {printf "%. 2fT\ t% s\ n", $1 / (1024 * 1024 * 1024), $2} else if (1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024) {printf "%. 2fG\ t% s\ n", $1 / (1024 * 1024) $2} else if (1024 * 1024 > $1 & & $1 > = 1024) {printf "%. 2fM\ t\ t% s\ n", $1max 1024, $2} else {printf "% sk\ t\ t% s\ n", $1, $2} '7.13G data2.17G usr1.75G home447.04M var408.50M run//du-sk * # shows the size of each folder and file in the current directory in KB (most commonly used), s for summary, and k for statistical unit KB / /. / # under the current directory / / sort-nr # is sorted in behavior units, sorted according to the size of the numbers from largest to smallest; / / awk-F'\ tSize # is divided by horizontal tabs, and the later program is to convert units, formatting and outputting them in an easy-to-understand form.

(2) for more information on du, head, sort and awk, please refer to the appendices of the existing articles.

(3) reference for using printf command in Linux

/ / the printf command in Linux uses the reference / / https://www.linuxprobe.com/linux-printf-example.html'{ if (1024 * 1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024 * 1024) {printf "%. 2fT\ t% s\ n", $1 / (1024 * 1024 * 1024) $2} else if (1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024) {printf "%. 2fG\ t\ t% s\ n", $1 / (1024 * 1024), $2} else if (1024 * 1024 > $1 & & $1 > = 1024) {printf "%. 2fM\ t% s\ n", $1024, $2} else {printf "% sk\ t\ t% s\ n", $1 $2}}'4 how do I safely delete files using the rm command?

(1) what are the pitfalls in the rm command?

The command rm-rf / # is absolutely inoperable. Delete the files in the root directory, that is, all files in the system will be deleted. If the online service machine is operated, it will be a tragedy! What should I do if I misoperate? Come on, ctrl+c and ctrl+z can keep as much as they can.

Rm-rf / home/apps/logs/ # this is also a sinkhole command! The aim is to delete the log text. As a result, there is an extra space in the bug when writing, do you understand it? This becomes [rm-rf /]!

Buried hidden trouble log clean shell script! The key content of the script is as follows.

Cd ${log_path} rm-rf *

The goal is to go to the log directory and delete all the logs. Hidden trouble: when the directory does not exist, the tragedy occurs!

(2) how to safely use the rm command?

Replace the [rm-rf] command with [mv] in the production environment, and write a script to clean up periodically to simulate the function of the Recycle Bin.

Clean up the log shell script and use logic to connect with & &.

Cd ${log_path} rm-rf *

Use logic to connect with & & and merge into one sentence. The logic of the first half of the sentence fails, and the second half of the command is not executed:

```shell

Cd ${log_path} & & rm-rf *

The complete log cleaning shell script is as follows:

````shellroomroombinbinram bashbaseplate home= "/ home/apps" log_path=$ {base_home} / logscd ${log_path} & & rm-rf * 5 disk usage alarm, but no specific large files can be found?

(1) problem scenario

1 disk utilization monitoring alarm, enter the machine can (df-h) command to see that the disk utilization does exceed the alarm threshold.

2 use the command to view the large directory And go to the directory [du-sk *. / | sort-nr | head-n5 | awk-F'\ t'{if (1024 * 1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024 * 1024) {printf "%. 2fT\ t% s\ n", $1 / (1024 * 1024 * 1024), $2} else if (1024 * 1024 * 1024 > $1 & & $1 > = 1024 * 1024) {printf "%. 2fG\ t% s\ n", $1 / (1024 * 1024) $2} else if (1024 * 1024 > $1 & & $1 > = 1024) {printf "%. 2fM\ t\ t% s\ n", $1 2fM 1024, $2} else {printf "% sk\ t\ t% s\ n", $1, $2}}']

3 still can't find a large file, what should I do?

(2) investigation ideas

1 thinking: is there a file that has been deleted, but the process is still occupying the file, the process is not finished, and the space has not been released?

2 use the "lsof | grep-I deleted" command to view the deleted files that have not been freed, including file size, process and service name, etc.

Lsof (List Open Files) is used to view the files opened by the process, the process that opens the files, the ports opened by the process (TCP, UDP), and to retrieve / restore deleted files. Is a very convenient system monitoring tool, because the lsof command requires access to core memory and various files, so it requires root user rights to execute.

(3) take up disk space and release

Restart the service referred to by the process, and the disk space occupied can be released. Online production operations must be careful not to kill the process directly, evaluate whether there is a restart command for the process service itself, and evaluate whether the service can be restarted.

(4) Appendix to remarks

1 when a file is being used by a process, the user deletes the file, and the file is only deleted from the directory structure, but not from disk.

2 when the process of using this file ends, the file will really be deleted from the disk, freeing up the occupied space. When Linux opens a file, the kernel creates a folder for each process in the / proc/ "/ proc/ {nnnn} / fd/ folder ({nnnn} is pid)" to store information about the process, and its subfolder fd stores the fd (fd:file descriptor) of all files opened by the process.

3 Ctrl + C and Ctrl + Z are interrupt commands. Ctrl + C forcibly interrupts the execution of the program, and the process has been terminated; Ctrl + Z means to abort the task (meaning pause), he is still in the process and he just maintains a suspended state.

6 commands for safe cleaning of large files commonly used in production environment

What is the demand for the safe cleaning of large documents in the production environment? It is not only not to affect the normal operation of the service, but also to quickly release the space occupied by the disk (it is not our goal to make the files disappear, our goal is to release the occupied disk space quickly).

Do not use "rm-rf xxx.log"; often use "echo"> xxx.log".

It is assumed that xxx.log is a large file, such as this xxx.log has dozens of GB, "echo"> xxx.log" is to overwrite the contents of the original file with a "" content, so that disk space is instantly freed!

At this point, the study on "how Linux finds large files and clears them safely" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.