Shulou(Shulou.com)06/01 Report--

Linux how to delete duplicate lines of text, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. Sort command-sorts lines of text files in Linux and Unix-like systems.

2. Uniq command-run or omit duplicate lines on Linux or Unix

Delete duplicate rows using Sort,Uniq and Shell pipes

Use the following syntax:

Sort {file-name} | uniq-u sort file.log | uniq-u

Delete duplicate lines using uniq

The following is a sample test file named garbage.txt displayed using the cat command:

Linuxidc@linuxidc:~/www.linuxidc.com$ cat linuxidc.com.txt

Sample output:

Linux Commune (www.linuxidc.com) registered and launched the website Linux on September 25, 2006. now it has become a widely concerned and supported operating system. IDC is the Internet data center, and LinuxIDC is the data center about Linux. Real-time release of Linux Information Linux Commune is a professional Linux system portal that publishes Linux information in real time, including Linux, Ubuntu, Fedora, RedHat Linux tutorials, Linux certification, SUSE Linux, Android, Oracle, Hadoop and other technologies. Real-time release * Linux information real-time release * * Linux information

Remove duplicate lines from a text file on Linux

Delete all duplicate lines by typing:

Linuxidc@linuxidc:~/www.linuxidc.com$ sort linuxidc.com.txt | uniq-u

Sample output:

IDC is the Internet data center, and LinuxIDC is the data center about Linux. Linux Commune (www.linuxidc.com) registered and opened its website on September 25, 2006. Linux Commune is a professional Linux system portal Linux tutorial, Linux certification, SUSE Linux, Android, Oracle, Hadoop and other technologies. Linux has become a widely concerned and supported operating system that publishes Linux information in real time, including Linux, Ubuntu, Fedora and RedHat.

Here,

-u: check for strict sorting and delete all duplicate rows.

Sort the contents of a file on Linux

Let's assume that you have a file called users.txt:

Cat linuxidc.txt Linux Idc 22-11-72 Zhang San 12-01-69 Li Ai 30-12-88 A Bao 15-08-76 Chen Xiao 05/06/77 Shen Lin 13-02-75 Da Cen 21-10-73 Zhang San 12-01-69

Let's sort and run:

Sort linuxidc.txt

Next, sort by last name, run:

Sort-r linuxidc.txt

Want to sort in reverse order? Try:

Sort + 2 linuxidc.txt

You can eliminate any duplicate entries in the file when sorting the file, and run:

Sort + 2-u linuxidc.txt sort-u linuxidc.txt

Without any option, sort compares the entire lines in the file and outputs them in ASCII order. You can use options to control the output.

How to use the uniq command to delete duplicate lines on a Linux

Use the following files:

Linuxidc@linuxidc:~/www.linuxidc.com$ cat-n linuxidc.txt

Sample output:

1 www.linuxidc.com 2 Linux Commune 3 linuxidc 4 linuxidc.com 5-6 5566778899 7 5566778899 8 5566778899

The uniq command removes line 8 from the file and places the result in a file named linuxidc.com.txt:

Uniq linuxidc.txt linuxidc.com.txt

Verify it:

Cat-n linuxidc.com.txt

How to use the uniq command to delete duplicate lines

How to delete duplicate lines in a .txt file and save the results to a new file

Try any of the following syntax:

Sort shuru_wenjian | uniq > shuchu_wenjian sort shuru_wenjian | uniq-u | tee shuchu_wenjian

Summary

The sort command is used to sort the lines of a text file, and uniq filters copy adjacent lines from the text file. There are many useful options for these commands. I recommend that you read the man page by typing the following man command:

Man sort man uniq after reading the above, have you mastered the method of how to delete duplicate lines of text in Linux? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.