Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to use the uniq command". Many people will encounter such a dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The uniq command itself is used to remove duplicate lines from a text file, similar to the sort command, but always different. Text files in Linux can be either txt or any other custom format.

First, let's create a file with some duplicate lines:

Vi ostechnix.txtwelcome to ostechnixwelcome to ostechnixLinus is the creator of Linux.Linux is secure by defaultLinus is the creator of Linux.Top 500 super computers are powered by Linux

As you can see in the above file, we have some duplicate lines (the first and second lines, the third and fifth lines are duplicated).

1. Use the uniq command to delete consecutive duplicate lines in the file

If you use the uniq command without any arguments, it will delete all consecutive duplicate lines and display only unique lines.

Uniq ostechnix.txt

Sample output:

As you can see, the uniq command removes all consecutive duplicate lines from a given file. You may also notice that the output above is still duplicated by the second and fourth lines. This is because the uniq command deletes duplicate lines only if they are adjacent to each other. Of course, we can also delete non-contiguous duplicate lines. Please look at the second example below.

2. Delete all duplicate lines sort ostechnix.txt | uniq

Sample output:

You see that? There are no duplicate lines. In other words, the above command will display lines that appear only once in the ostechnix.txt. We use the sort command in conjunction with the uniq command because, as I mentioned, uniq does not delete duplicate lines unless they are adjacent.

3. Show only the unique line in the file

To display only the unique line in the file, you can do this:

Sort ostechnix.txt | uniq-u

Sample output:

Linux is secure by defaultTop 500 super computers are powered by Linux

As you can see, only two lines are unique in a given file.

4. Show only duplicate lines

Similarly, we can also display duplicate lines in the file, like this:

Sort ostechnix.txt | uniq-d

Sample output:

Linus is the creator of Linux.welcome to ostechnix

These two lines are duplicate lines in the ostechnix.txt file. Note that-d (lowercase d) will print only duplicate lines, one for each group. Print all duplicate lines, using-D (uppercase D), as follows:

Sort ostechnix.txt | uniq-D

See the difference between the two options in the screenshot below:

5. Show the number of occurrences of each line in the file

For some reason, you may want to check the number of repetitions of each line in a given file. To do this, use the-c option, as follows:

Sort ostechnix.txt | uniq-c

Sample output:

Linus is the creator of Linux.Linux is secure by defaultTop 500 super computers are powered by Linuxwelcome to ostechnix

We can also sort by the number of occurrences of each row, and then display it, as follows:

Sort ostechnix.txt | uniq-c | sort-nr

Sample output:

Welcome to ostechnixLinus is the creator of Linux.Top 500 super computers are powered by LinuxLinux is secure by default6, limit the comparison to N characters

We can use the-w option to limit the comparison of a specific number of characters in the file. For example, let's compare the first four characters in the file and display duplicate lines, as follows:

Uniq-d-w 4 ostechnix.txt7, ignore N characters specified by comparison

Just like comparing the first N characters of the lines in a file, we can also use the-s option to ignore the first N characters of the comparison.

The following command ignores the first four characters of each line in the file:

Uniq-d-s 4 ostechnix.txt

To ignore comparing the first N fields instead of characters, use the-f option in the above command.

For more details, please refer to the help section:

Uniq-help

You can also use the man command to view:

This is the end of man uniq's "how to use the uniq command". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.