Shulou(Shulou.com)06/02 Report--

The old employer monitors the slow query log of Redis, uses Crontab to sample the slow query log at regular intervals, and then writes a text file.

So there are many duplicate slow query logs in the text file.

How to deduplicate the whole paragraph of repeated slow query logs?

The way to use the Shell command is the simplest and most straightforward.

Awk'/ ^ 1\) / {print sscatter = ""}; {s "" $0)}; END {print s} 'b.txt | sort | uniq | sed's / /\ n / g' | awk'/ ^ 1\)\ (integer\) / {rc = rc+1;rowdata=rc ")\ t" $1 "" $2 "$3"\ n\ t "4" $5 "$6" $7 "\ n\ t" $8 "$9" $10 " Print rowdata}; / ^ / {print "\ t" $0} END {print "No duplicate logs Total:" rc}'

The core is to merge the entire paragraph of redis logs into one line, remove the duplicates, and restore the format.

Here are the attachments for testing

RedisSlowLog.txt

With Wang Gong's description and explanation of the demand for this problem.

Https://blog.csdn.net/wzy0623/article/details/83652539?tdsourcetag=s_pcqq_aiomsg

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.