Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to find and delete duplicate files in Linux. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

What is fdupes?

Fdupes is a tool under Linux written by Adrian Lopez in the C programming language and distributed under the MIT license. The application can find duplicate files in specified directories and subdirectories. Fdupes identifies duplicates by comparing the MD5 signatures of files and comparing files byte by byte. Fdupes has a variety of options to list, delete and replace hard links with copies of files.

The file starts in the following order:

Size comparison > partial MD5 signature comparison > complete MD5 signature comparison > byte by byte comparison

Install fdupes to Linux

On Debian-based systems such as Ubuntu and Linux Mint, install the * * version of fdupes with the following command.

$sudo apt-get install fdupes

On CentOS/RHEL-and Fedora-based systems, you need to open the epel repository to install the fdupes package.

# yum install fdupes# dnf install fdupes [in Fedora 22 and beyond]

Note: since Fedora 22, the default package manager yum has been replaced by dnf.

How to use the fdupes command

1. For demonstration purposes, let's create some duplicate files in a directory (such as tecmint) with the following command:

$mkdir / home/ "$USER" / Desktop/tecmint & & cd / home/ "$USER" / Desktop/tecmint & & for i in {1.. 15}; do echo "I Love Tecmint. Tecmint is a very nice community of Linux Users." > tecmint$ {I} .txt; done

After executing the above command, let's use the ls command to verify that the duplicate file is created.

Ls-l total 60 tecmint tecmint 65 Aug 8 11:22 tecmint13.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint14.txt-rw-r--r---1 tecmint tecmint 65 Aug 8 11:22 tecmint10.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint11.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint12.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 1 tecmint tecmint 65 Aug 8 11:22 tecmint15.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint1.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint2.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint3.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint4.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint5.txt-rw- Tecmint6.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint6.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint7.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint8.txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint9.txt

The above script creates 15 files named tecmint1.txt,tecmint2.txt... Tecmint15.txt, and the data for each file is the same, such as:

"I Love Tecmint. Tecmint is a very nice community of Linux Users."

2. Now search the tecmint folder for duplicate files.

$fdupes / home/$USER/Desktop/tecmint / home/tecmint/Desktop/tecmint/tecmint13.txt/home/tecmint/Desktop/tecmint/tecmint8.txt/home/tecmint/Desktop/tecmint/tecmint11.txt/home/tecmint/Desktop/tecmint/tecmint3.txt/home/tecmint/Desktop/tecmint/tecmint4.txt/home/tecmint/Desktop/tecmint/tecmint6.txt/home/tecmint/Desktop/tecmint/tecmint7.txt/home/tecmint/Desktop/tecmint/tecmint9.txt/home/tecmint/Desktop/tecmint/tecmint10.txt/ Home/tecmint/Desktop/tecmint/tecmint2.txt/home/tecmint/Desktop/tecmint/tecmint5.txt/home/tecmint/Desktop/tecmint/tecmint14.txt/home/tecmint/Desktop/tecmint/tecmint1.txt/home/tecmint/Desktop/tecmint/tecmint15.txt/home/tecmint/Desktop/tecmint/tecmint12.txt

Use the-r option to recursively search for duplicate files in each directory, including its subdirectories.

It recursively searches all files and folders and takes a moment to scan for duplicate files, depending on the number of files and folders. In the meantime, the entire process is displayed in the terminal, as shown below.

$fdupes-r / home Progress [37780Accord 54747] 69%

4. Use the-S option to view the size of duplicate files found in a folder.

$fdupes-S / home/$USER/Desktop/tecmint 65 bytes each: / home/tecmint/Desktop/tecmint/tecmint13.txt/home/tecmint/Desktop/tecmint/tecmint8.txt/home/tecmint/Desktop/tecmint/tecmint11.txt/home/tecmint/Desktop/tecmint/tecmint3.txt/home/tecmint/Desktop/tecmint/tecmint4.txt/home/tecmint/Desktop/tecmint/tecmint6.txt/home/tecmint/Desktop/tecmint/tecmint7.txt/home/tecmint/Desktop/ Tecmint/tecmint9.txt/home/tecmint/Desktop/tecmint/tecmint10.txt/home/tecmint/Desktop/tecmint/tecmint2.txt/home/tecmint/Desktop/tecmint/tecmint5.txt/home/tecmint/Desktop/tecmint/tecmint14.txt/home/tecmint/Desktop/tecmint/tecmint1.txt/home/tecmint/Desktop/tecmint/tecmint15.txt/home/tecmint/Desktop/tecmint/tecmint12.txt

5. You can use both the-S and-r options to view the size of duplicate files in all the directories and subdirectories involved, as follows:

$fdupes-Sr / home/avi/Desktop/ 65 bytes each: / home/tecmint/Desktop/tecmint/tecmint13.txt/home/tecmint/Desktop/tecmint/tecmint8.txt/home/tecmint/Desktop/tecmint/tecmint11.txt/home/tecmint/Desktop/tecmint/tecmint3.txt/home/tecmint/Desktop/tecmint/tecmint4.txt/home/tecmint/Desktop/tecmint/tecmint6.txt/home/tecmint/Desktop/tecmint/tecmint7.txt/home/tecmint/Desktop/tecmint / tecmint9.txt/home/tecmint/Desktop/tecmint/tecmint10.txt/home/tecmint/Desktop/tecmint/tecmint2.txt/home/tecmint/Desktop/tecmint/tecmint5.txt/home/tecmint/Desktop/tecmint/tecmint14.txt/home/tecmint/Desktop/tecmint/tecmint1.txt/home/tecmint/Desktop/tecmint/tecmint15.txt/home/tecmint/Desktop/tecmint/tecmint12.txt 107 bytes each:/home/tecmint/Desktop/resume_files/r-csc.html/home/tecmint/Desktop/resume_files/fc.html

6. Instead of searching recursively in one or all folders, you can choose to search selectively in two or three folders as required. There is no need to remind you that you can use the-S and / or-r options if necessary.

$fdupes / home/avi/Desktop/ / home/avi/Templates/

7. To delete duplicate files while keeping a copy, you can use the-d option. With this option, you must be extra careful, or the end result may be file / data loss. Solemnly remind me that this operation cannot be resumed.

$fdupes-d / home/$USER/Desktop/tecmint [1] / home/tecmint/Desktop/tecmint/tecmint13.txt [2] / home/tecmint/Desktop/tecmint/tecmint8.txt [3] / home/tecmint/Desktop/tecmint/tecmint11.txt [4] / home/tecmint/Desktop/tecmint/tecmint3.txt [5] / home/tecmint/Desktop/tecmint/tecmint4.txt [6] / home/tecmint/Desktop/tecmint/tecmint6.txt [7] / home/tecmint/Desktop/tecmint / tecmint7.txt [8] / home/tecmint/Desktop/tecmint/tecmint9.txt [9] / home/tecmint/Desktop/tecmint/tecmint10.txt [10] / home/tecmint/Desktop/tecmint/tecmint2.txt [11] / home/tecmint/Desktop/tecmint/tecmint5.txt [12] / home/tecmint/Desktop/tecmint/tecmint14.txt [13] / home/tecmint/Desktop/tecmint/tecmint1.txt [14] / home/tecmint/Desktop/tecmint/tecmint15.txt [15] / home/tecmint/Desktop/tecmint/tecmint12.txt Set 1 of 1 Preserve files [1-15, all]:

You may have noticed that all duplicate files are listed and prompted to delete one by one, or specify a range, or delete them all at once. You can select a range, like below, to delete files within the specified range.

Set 1 of 1, preserve files [1-15 All]: 2-15 [-] / home/tecmint/Desktop/tecmint/tecmint13.txt [+] / home/tecmint/Desktop/tecmint/tecmint8.txt [-] / home/tecmint/Desktop/tecmint/tecmint11.txt [-] / home/tecmint/Desktop/tecmint/tecmint3.txt [-] / home/tecmint/Desktop/tecmint/tecmint4.txt [-] / home/tecmint/Desktop/tecmint/tecmint6.txt [-] / home/tecmint/Desktop / tecmint/tecmint7.txt [-] / home/tecmint/Desktop/tecmint/tecmint9.txt [-] / home/tecmint/Desktop/tecmint/tecmint10.txt [-] / home/tecmint/Desktop/tecmint/tecmint2.txt [-] / home/tecmint/Desktop/tecmint/tecmint5.txt [-] / home/tecmint/Desktop/tecmint/tecmint14.txt [-] / home/tecmint/Desktop/tecmint/tecmint1.txt [-] / home/tecmint/Desktop/ Tecmint/tecmint15.txt [-] / home/tecmint/Desktop/tecmint/tecmint12.txt

8. From a security point of view, you may want to print the output of fdupes to a file, and then check the text file to decide which file to delete. This reduces the risk of accidentally deleting files. You can do this:

$fdupes-Sr / home > / home/fdupes.txt

Note: you should replace / home for the folder you want. At the same time, if you want to search recursively and print the size, you can use the-r and-S options.

9. You can use the-f option to ignore the * files in each matching set.

First list the files in that directory.

$ls-l / home/$USER/Desktop/tecmint total 20 tecmint tecmint 65 Aug 8 11:22 tecmint9 (3rd copy) .txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint9 (4th copy) .txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint9 (another copy) .txt-rw-r--r-- 1 tecmint tecmint 65 Aug 8 11:22 tecmint9 (copy) .txt-rw-r--r -- 1 tecmint tecmint 65 Aug 8 11:22 tecmint9.txt

Then, ignore the * files in each matching set.

$fdupes-f / home/$USER/Desktop/tecmint / home/tecmint/Desktop/tecmint9 (copy). Txt/home/tecmint/Desktop/tecmint9 (3rd copy). Txt/home/tecmint/Desktop/tecmint9 (another copy). Txt/home/tecmint/Desktop/tecmint9 (4th copy). Txt

10. Check the installed version of fdupes.

$fdupes-- version fdupes 1.51

If you need help with fdupes, you can use the-h switch.

$fdupes-h Usage: fdupes [options] DIRECTORY... -r-- recurse for every directory given follow subdirectories encountered within-R-- recurse: for each directory given after this option follow subdirectories encountered within (note the':'at the end of the option, manpage for more details)-s-- symlinks follow symlinks-H-hardlinks normally, when two or more files point to the same disk area they are treated as non-duplicates This option will change this behavior-n-noempty exclude zero-length files from consideration-A-nohidden exclude hidden files from consideration-f-omitfirst omit the first file in each set of matches-1-sameline list each set of matches on a single line-S-size show size of duplicate files-m-summarize summarize dupe information-Q-quiet hide progress indicator-d-delete prompt user For files to preserve and delete all others Important: under particular circumstances, data may be lost when using this option together with-s or-- symlinks, or when specifying a particular directory more than once Refer to the fdupes documentation for additional information-N-- noprompt together with-- delete, preserve the first file in each set of duplicates and delete the rest without prompting the user-v-- version display fdupes version-h-- help display this help message is here on how to find and delete duplicate files in Linux. I hope the above content can be of some help to you. You can learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.