Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "the example introduction of deleting duplicate line command uniq under linux". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First, what does uniq do?

The duplicate lines in the text are basically not what we want, so we need to remove them. There are other commands under linux to remove duplicate lines, but I think uniq is a more convenient one. Pay attention to the following two points when using uniq

1. When working with text, it is usually used in combination with the sort command, because uniq does not check for duplicate lines unless they are adjacent lines. If you want to sort the input first, use sort-u.

2. When operating on text, if the field is a first blank character (usually including spaces and tabs), and then a non-empty character, the empty character before the character in the field will be skipped

Second, uniq parameter description

The code is as follows:

[zhangy@BlackGhost] $uniq-- help

Usage: uniq [option]. [file]

Filter adjacent matching lines from the input file or standard input and write to the output file or standard output.

When no options are attached, the matching rows will be merged at the first occurrence.

The parameters that must be used for long options are also required for short options.

-c,-- count / / precede each line with a prefix number indicating the number of occurrences of the corresponding line header

-d,-- repeated / / output only duplicate lines

-D,-- all-repeated / / outputs only duplicate lines, but several lines output several lines

-f,-- skip-fields=N / /-the number of segments ignored by f,-f 1 ignores the first paragraph

-I,-- ignore-case / / case-insensitive

-s,-- skip-chars=N / / root-f is a bit like, but-s is ignored. The following characters-s 5 will ignore the next 5 characters.

-u,-- unique / / after removing the duplicates, all are displayed, and the distinct function of the root mysql is a bit like

-z,-- zero-terminated end lines with 0 byte, not newline

-w,-- check-chars=N / / A pair of contents after the nth character of each line are not compared.

-- help / / displays this help and exits

-- version / / displays version information and exits

I don't know what the use of-z is.

Third, test the text file uniqtest

The code is as follows:

This is a test

This is a test

This is a test

I am tank

I love tank

I love tank

This is a test

Whom have a try

WhoM have a try

You have a try

I want to abroad

Those are good men

We are good men

Fourth, detailed explanation of examples.

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-c uniqtest

3 this is a test

1 i am tank

2 i love tank

1 this is a test / / and the first line are duplicated

1 whom have a try

1 WhoM have a try

1 you have a try

1 i want to abroad

1 those are good men

1 we are good men

As we can see from the example above, a feature of uniq is that when duplicate rows are checked, only adjacent rows are checked. Duplicate data, there must be a lot of not adjacent to each other.

The code is as follows:

[zhangy@BlackGhost mytest] $sort uniqtest | uniq-c

1 WhoM have a try

1 i am tank

2 i love tank

1 i want to abroad

4 this is a test

1 those are good men

1 we are good men

1 whom have a try

1 you have a try

This can solve the problem mentioned in the previous example.

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-d-c uniqtest

3 this is a test

2 i love tank

Uniq-d displays only duplicate lines

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-D uniqtest

This is a test

This is a test

This is a test

I love tank

I love tank

Uniq-D displays only the duplicate lines, and displays all the repeated lines. He can't use it with-c

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-f 1-c uniqtest

3 this is a test

1 i am tank

2 i love tank

1 this is a test

2 whom have a try

1 you have a try

1 i want to abroad

2 those are good men / / there is only one line, showing two lines

In this case, the those has only one line, but it is repeated because-f 1 ignores the first column and checks for repetition starting with the second field.

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-I-c uniqtest

3 this is a test

1 i am tank

2 i love tank

1 this is a test

2 whom have a try / / one uppercase and one lowercase

1 you have a try

1 i want to abroad

1 those are good men

1 we are good men

When checking, it is not case-sensitive

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-s 4-c uniqtest

3 this is a test

1 i am tank

2 i love tank

1 this is a test

3 whom have a try / / what is the difference between the previous example

1 i want to abroad

1 those are good men

1 we are good men

When checking, ignore the first four characters, so whom have a try is the same as you have a try.

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-u uniqtest

I am tank

This is a test

Whom have a try

WhoM have a try

You have a try

I want to abroad

Those are good men

We are good men

Repeat the items, and then display them all

The code is as follows:

[zhangy@BlackGhost mytest] $uniq-w 2-c uniqtest

3 this is a test

3 i am tank

1 this is a test

1 whom have a try

1 WhoM have a try

1 you have a try

1 i want to abroad

1 those are good men

1 we are good men

The content after the second character of each line is not checked, so the i am tank root i love tank is the same.

This is the end of the introduction of the example of deleting duplicate line command uniq under linux. Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.