Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

How to delete duplicate lines in Linux

2025-04-01 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)06/01 Report--

This article introduces you how to delete duplicate lines in Linux, the content is very detailed, interested friends can refer to, hope to be helpful to you.

First, use sort+uniq. Note that uniq alone will not work.

Sort-n test.txt | uniq

Second, use the sort+awk command. Note that awk alone will not work either, for the same reason as above.

Sort-n $file | awk'{if ($0,000lines) print; line=$0}'

Third, with the sort+sed command, you also need the sort command to sort first.

Sort-n $file | sed'$! n; / ^ .∗\ n\ 1 $/! P; D'

Shell script

#! / bin/shfile='test.txt'sort-n $file | uniqsort-n $file | awk'{if ($0Secretline) print; line=$0} 'sort-n $file | sed' $! n; / ^\ (. *\)\ n\ 1 $/! P; D'

Test file:

Yanggang@barry$ cat test.txt

Aaa

Bbbbb

Ccccc

one hundred and twenty three

Aaaaa

one hundred and twenty three

Bbb

Aaa

Execution result:

Yanggang@barry$. / diffRow.sh

Aaa

Aaaaa

Bbb

Bbbbb

Ccccc

one hundred and twenty three

On how to delete duplicate lines in Linux to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 267

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report