Shulou(Shulou.com)06/02 Report--

This article mainly summarizes some common commands about the statistical analysis of Nginx logs and shares them for your reference and study. Let's take a look at the detailed introduction:

I. related statistics of IP

Statistics of IP visits

Awk'{print $1} 'access.log | sort-n | uniq | wc-l

Check the number of IP visits in a certain period of time (4-5 o'clock)

Grep "07/Apr/2017:0 [4-5]" access.log | awk'{print $1}'| sort | uniq-c | sort-nr | wc-l

Check out the top 100 most frequently visited IP

Awk'{print $1} 'access.log | sort-n | uniq-c | sort-rn | head-n 100

View IP that have been visited more than 100 times

Awk'{print $1} 'access.log | sort-n | uniq-c | awk' {if ($1 > 100) print $0}'| sort-rn

Query the detailed access of an IP, sorted by access frequency

Grep '104.217.108.66' access.log | awk' {print $7}'| sort | uniq-c | sort-rn | head-n 100

Second, page visit statistics

View the most visited page (TOP100)

Awk'{print $7} 'access.log | sort | uniq-c | sort-rn | head-n 100

View the most frequently visited pages ([exclude php pages] (TOP100)

Grep-v ".php" access.log | awk'{print $7}'| sort | uniq-c | sort-rn | head-n 100

View pages that have been visited more than 100 times

Cat access.log | cut-d''- f 7 | sort | uniq-c | awk'{if ($1 > 100) print $0}'| less

View the most visited page with the last 1000 records

Tail-1000 access.log | awk'{print $7}'| sort | uniq-c | sort-nr | less

Statistics of requests per second

Count the number of requests per second and the time point of top100 (accurate to seconds)

Awk'{print $4} 'access.log | cut-c 14-21 | sort | uniq-c | sort-nr | head-n 100

Statistics of requests per minute

Count the number of requests per minute and the time point of top100 (accurate to minutes)

Awk'{print $4} 'access.log | cut-c 14-18 | sort | uniq-c | sort-nr | head-n 100

Hourly request statistics

Count the number of requests per hour and the time point of top100 (accurate to hours)

Awk'{print $4} 'access.log | cut-c 14-15 | sort | uniq-c | sort-nr | head-n 100

Third, performance analysis

Add $request_time to the last field in nginx log

List the pages with a transmission time of more than 3 seconds, and display the first 20

Cat access.log | awk'($NF > 3) {print $7}'| sort-n | uniq-c | sort-nr | head-20

List the pages with a request time of more than 3 seconds for php pages, and count the number of times they appear, showing the first 100 entries

Cat access.log | awk'($NF > 1 & & $7 dollars /\ .php /) {print $7}'| sort-n | uniq-c | sort-nr | head-100

IV. Spider capture statistics

Statistics on the number of spiders crawling

Grep 'Baiduspider' access.log | wc-l

Count the number of spiders crawling 404

Grep 'Baiduspider' access.log | grep' 404' | wc-l

Fifth, TCP connection statistics

View current TCP connections

Netstat-tan | grep "ESTABLISHED" | grep ": 80" | wc-l

Use tcpdump to sniff the access to port 80 to see who is the highest.

Tcpdump-I eth0-tnn dst port 80-c 1000 | awk-F "."'{print $1 "." $2 "." $3 "." $4}'| sort | uniq-c | sort-nr

VI. Example script

Get the number of nginx access log entries in the last minute

#! / bin/bash export LANG=Cexport PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binTIME=$ (date-d "1 minute ago" + "% d/%h/%Y:%H:%M") grep "$TIME" / var/log/nginx/access.log | wc-l

Get the number of nginx error log entries in the previous minute

#! / bin/bash export LANG=Cexport PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binTIME=$ (date-d "1 minute ago" + "% Y-%m-%d% HRV% M") grep "$TIME" / var/log/nginx/error.log | wc-l

Summary

The above is the whole content of this article, I hope that the content of this article can bring some help to your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.