Shulou(Shulou.com)06/01 Report--

Today, I would like to share with you the relevant knowledge points of practical examples of AWK, which are detailed in content and clear in logic. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article.

1. Analyze the access log (Nginx as an example) log format:'$remote_addr-$remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer"$http_user_agent"$http_x_forwarded_for" 'Statistical visits to IP: # awk' {a [$1] + +} END {for (v in a) print v A [v]} 'access.log Statistical visit IP:# awk' {a [$1] + +} END {for (v ina) {if (a [v] > 100) print v [v]}} 'access.log counts the number of visits to IP and ranks the first 10 END {for (v ina) print v A [v] | "sort-K2-nr | head-10"} 'access.log counts the most visited IP:# awk'$4 during the time period > = "[02/Jan/2017:00:02:00" & & $4 counts the number of visits in the last minute: # date=$ (date-d'-1 minute'+%d/%d/%Y:%H:%M) # awk-vdate=$date'$4~date {C++} END {printc} 'access.log counts the top 10 pages visited: # awk'{a [$7] +} END {for (vin a) print v A [v] | "sort-K1-nr | head-n10"} 'access.log counts the number of each URL and the total size of returned content: # awk' {a [$7] + Size [$7] + = $10} END {for (v ina) print a [v], vMagazine size [v]} 'access.log counts the number of status codes per IP: # awk' {a [$1 "" $9] + +} END {for (v ina) print v [v]} 'access.log Statistics the number of visits to IP is 404 status times: # awk' {if ($9 impulse 404 /) a [$1 "$9] + +} END {for (I ina) print v A [v]} 'access.log2, compare the contents of the two files: # seq 1 5 > a # seq 3 7 > b find the same record of the b file in the a file: method 1 seq # awk' FNR==NR {a [$0] Next} {if ($0 ina) print $0}'a b34 percent awk 'FNR==NR {a [$0]; next} {if ($0 ina) print FILENAME,$0}' a bb 3b 4b 'awk' FNR==NR {a [$0]} NR > FNR {if ($0 ina) print $0}'a b34 percent awk 'FNR==NR {a [$0] = 1 nextNext} (a [$0] = 1)' b # a [$0] is obtained from each line of b file, if 1 indicates that there are # awk 'FNR==NR {a [$0] = 1 Next} {if (a [$0] = = 1) print}'a b345 method 2FILENAME== # awk 'FILENAME== "a" {a [$0]} FILENAME== "b" {if ($0 in a) print $0}' a b345 method 3V # awk 'ARGIND==1 {a [$0] = 1} ARGIND==2 & & a [$0] = = 1' a b345 find out the different records of b file in a file: method 1V # awk 'FNR==NR {a [$0]; next}! ($0 in a)' a b6 records awk 'FNR==NR {a [$0] = 1 Next} (a [$0]! = 1)'a b # awk'FNR==NR {a [$0] = 1 Next} {if (a [$0]! = 1) print}'a b67 method 2 FILENAME== # awk'FILENAME== "a" {a [$0] = 1} FILENAME== "b" & & a [$0] = 1} ARGIND==2 & & a [$0]! = 1'a b3, merge two file contents: # cat azhangsan 20lisi 23wangwu 29# cat bzhangsan manlisi womanwangwu man merge a file into b file: method 1Rod # awk' FNR==NR {a [$1] = $0 Next} {print a [$1], $2}'a bzhangsan 20 manlisi 23 womanwangwu 29 man method 2bzhangsan # awk 'FNR==NR {a [$1] = $0} NR > FNR {print a [$1], $2}' a bzhangsan 20 manlisi 23 womanwangwu 29 man merge the service names of a file with the same IP: # cat a192.168.1.1: httpd192.168.1.1: httpd192.168.1.2: postfix192.168.1.3: mysqld192.168.1.4: httpd# awk 'BEGIN {FS= ":" OFS= ":"} {a [$1] = a [$1] $2} END {for (v in a) print vmina [v]} a192.168.1.4: httpd192.168.1.1: httpd tomcat192.168.1.2: httpd postfix192.168.1.3: mysqld interpretation: array a storage is $1yoga [$1] $2, the first a [$1] is subscript to the first field, and the value is a [$1] $2, that is, $1yoga [$1] $2. The a [$1] of the value uses the first field as the subscript to get the corresponding value, but the first array a has no elements, then a [$1] is null, and the array storage is 192.168.1.1=httpd. When 192.168.1.1 is encountered, a [$1] obtains the httpd of the superscript through the first field subscript, and puts the second field of the currently processed line after the value of the previous subscript. As the new value of the subscript 192.168.1.1. At this point, the array store is 192.168.1.1=httpd tomcat. Each time the same subscript (the first field) is encountered, it gets the value corresponding to the last subscript and the new value of the current field as the subscript. 4. Merge the first column into a row # cat file1 2 34 5 67 8 "awk'{for (i4 472 5 83 69 9) interpretation: the for loop is a field that traverses each row, NF equals 3, and loops 3 times. When reading the first row: the first field: a [1] = a [1] 1 "value a [1] has not yet defined an array, and the subscript cannot get the corresponding value, so it is empty, so a [1] = 1. The second field: a [2] = a [2] 2 "" value a [2] array a has been defined, but there is no subscript 2, and the corresponding value cannot be obtained, so a [2] = 2. The third field: a [3] = a [3] 3 "" value a [2] as above, empty, a [3] = 3. When reading the second line: the first field: a [1] = a [1] 4 "" value a [2] gets the 2 of the array an as the corresponding value of the subscript, there is already this subscript above, and the corresponding value is 1, so the second field a [1] = 1 4: a [2] = a [2] 5 "ditto, a [2] = 25 the third field: a [3] = a [3] 6" ditto A [2] = 36 when reading the third row, the processing method is the same as above, and there are three subscripts at the end of the array, namely, 1, 1, 4, 4, 7, 7, 2, 5, 8, 3, 3, 36. Finally, the for loop outputs all the subscript values. 5. String splitting: method 1: awk # echo "hello" | awk-F''{for | awk'{split ($0echo a, "''"); for (v in a) print a [v]} 'lohel6; count the number of occurrences of each letter in the string: # echo "a.b.cparc.d.e" | awk-F' [.]' {for (item1) Ia 1b 1c 2d 1e 17, cost statistics: # cat azhangsan 8000 1zhangsan 5000 1lisi 1000 1lisi 2000 1wangwu 1500 1zhaoliu 6000 1zhaoliu 2000 1zhaoliu 3000 travel awk'{name [$1] +; cost [$1] + = $2 Number [$1] + = $3} END {for (v in name) print v costing [v], number [v]} 'azhangsan 5000 1lisi 3000 2wangwu 1500 1zhaoliu 11000 38, get the maximum number of a column # cat aa b 1c d 2e f 3G h 3i j 2 get the third field maximum: # awk' BEGIN {max=0} {if ($3 > max) max=$3} END {print max} 'a3 print the third field maximum line: # awk 'BEGIN {max=0} {a [$0] = $3 If ($3 > max) max=$3} END {for (v in a) if (a [v] = = max) print v}'ag h 3e f 39, remove the first and last lines of text # seq 5 | awk'NR > 2 {print s} {slots 0} '234 interpretation: read the first line, NR=1, do not perform print sbook 1 to read the second line, NR=2, do not perform print sbook 2 (greater than true) to read the third line, NR=3, execute print s, at this time s is the last line of p assignment content 2 Execute print s, print the penultimate line, s = last line 10, get the back-end IP and port # cat aupstream example-servers1 {server 127.0.0.1 IP 80 weight=1 max_fails=2fail_timeout=30s in the Nginx upstream block } upstream example-servers2 {server 127.0.0.1 example-servers1/,/ 80 weight=1 max_fails=2fail_timeout=30s; server 127.0.0.1 weight=1 max_fails=2fail_timeout=30s; server 82 backup;} # awk'/ example-servers1/,/} / {if (NR > 2) {print s} {slots 2}} a127.0.1 example-servers1/,/} / {if (I > 1) print slots +'a # awk'/ example-servers1/,/} / {if (I > 1) {print s} {slots 2 Read the first line, the initial value of I is 0, 0 > 1 is false, do not execute prints, the second line is read, the second line is read, the second line is printed, the last line is prints, the penultimate line is printed. The third line is read by prints,s=127.0.0.1:80,i=2. S = the last line. This is the same as above, except that iTunes + is used as a counter. These are all the contents of the article "what are the practical examples of AWK?" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.