Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to achieve Web log analysis, many people may not know much about it. In order to make you understand better, the editor summed up the following content for you. I hope you can get something according to this article.

1. Web log

The Web access log records all kinds of raw information such as Web server receiving and processing requests and run-time errors. Through the security analysis of the WEB log, it can not only help us to locate the attacker, but also help us to restore the attack path, find the security loopholes in the website and repair them.

Let's take a look at an Apache access log:

127.0.0.1-[11/Jun/2018:12:47:22 + 0800] "GET / login.html HTTP/1.1" 200786 "-" Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 "

Through this Web access log, we can clearly know which page of your website users visited under the circumstances of what IP, when, what operating system, what browser, and whether the visit was successful.

This paper introduces the ideas and techniques commonly used in the security analysis of Web logs.

2. Log analysis skills

In the security analysis of WEB logs, it can be carried out in accordance with two ideas, gradually in-depth, restore the whole attack process.

The first is to determine the time range of the invasion, take this as a clue, find suspicious logs within this time range, further investigate, and finally identify the attacker and restore the attack process.

The second kind: after invading the website, the attacker will usually leave the back door to maintain permissions to facilitate access again. We can find the file and use it as a clue to carry out analysis.

Common analysis tools:

Under Window, it is recommended to use EmEditor for log analysis, which supports large text, and the search efficiency is good.

Under Linux, use the Shell command to combine query analysis.

Shell+Linux command to achieve log analysis, generally combined with grep, awk and other commands to achieve several commonly used log analysis statistical skills.

Apache log analysis skills:

1. List the IP commands with the most visits on the day: cut-d-f 1 log_file | uniq-c | sort-rn | head-202. check the number of IP visits on the day: awk'{print $1} 'log_file | sort | uniq | wc-L3, view the number of times a page has been visited: grep "/ index.php" log_file | wc-L4, see how many pages each IP has visited: awk' {+ + S [$1]} END {for (an in S) print a S [a]} 'log_file5, sort the number of pages visited by each IP from small to large: awk' {+ + S [$1]} END {for (an in S) print S [a], a} 'log_file | sort-N6, see which pages have been visited by an IP: grep ^ 111.111.111.111 log_file | awk' {print $1p 7}'7, remove the pages of the day of search engine statistics: awk'{print $12 $1} 'log_file | grep ^\ "Mozilla | awk' {print $2}'| sort | uniq | wc-l8 to see how many IP visits were made during the 14:00 hour on June 21, 2018: awk'{print $4 last month 1} 'log_file | grep 21/Jun/2018:14 | awk' {print $2}'| sort | uniq | wc-L3, log analysis case

Web log analysis example: it is forwarded to a private network server through nginx agent, and several pictures Trojans are uploaded under a site directory of the private network server. Although it cannot be parsed under II7, we still want to find out who uploaded it through which path.

Here, we encounter a problem: because proxy forwarding is set, only the ip of the proxy server is recorded, not the visitor IP? At this point, how to identify different visitors and sources of attack?

This is a problem with improper configuration of administrator logs, but the good news is that we can use browser fingerprints to locate different access sources and restore attack paths.

1. Locate the source of attack

First of all, only one record of the image Trojan was found. Since all the access logs only record the proxy IP, the attack path cannot be restored through IP. In this case, the browser fingerprint can be used to locate it.

Browser fingerprint:

Mozilla/4.0+ (compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+.NET4.0C;+.NET4.0E)

2. Search related log records

By filtering the log records related to the browser's fingerprint, you can clearly see the attack path of the attacker.

3. To interpret the access log found, the attacker's approximate access path is as follows:

A, the attacker visited the home page and login page B, the attacker visited MsgSjlb.aspx and MsgSebd.aspxC, the attacker visited Xzuser.aspxD, the attacker POST several times (suspected of flaws in the upload module through this page) E, and the attacker visited the picture Trojan

Open the website, visit Xzuser.aspx, confirm that the attacker uploaded the picture Trojan through the file on this page, at the same time, found that there is a vulnerability of ultra vires access to the website, the attacker can visit a specific URL and enter the background interface without logging in. Find the location of the loophole in the website through log analysis and fix it.

4. Skills of log statistical analysis

Statistical crawler:

Grep-E 'Googlebot | Baiduspider' / www/logs/access.2019-02-23.log | awk' {print $1}'| sort | uniq

Statistics browser:

Cat / www/logs/access.2019-02-23.log | grep-v-E 'MSIE | Firefox | Chrome | Opera | Safari | Gecko | sort | uniq-c | sort-r-n | head-n 100

IP Statistics:

Grep '23MayCharger 2019' / www/logs/access.2019-02-23.log | awk' {print $1}'| awk-fumble.'{print $1 "." $2 "." $3 "." $4}'| sort | uniq-c | sort-r-n | head-n 219.136.134.134.1497 182.34.15.2481431 211.140.100119.145.149.101427 61.183.15.1791427218.8.1891422 124.232. 150.1711421 106.187.47.2241420 61.160.220.2521418 114.80.201.18

Statistical network segment:

Cat / www/logs/access.2019-02-23.log | awk'{print $1}'| awk-Found.' {print $1 "." $2 "." $3 ".0"}'| sort | uniq-c | sort-r-n | head-n 200

Statistical domain name:

Cat / www/logs/access.2019-02-23.log | awk'{print $2}'| sort | uniq-c | sort-rn | more

HTTP Status:

Cat / www/logs/access.2019-02-23.log | awk'{print $9}'| sort | uniq-c | sort-rn | more5056585 3041125579 2007602 4005 301

URL Statistics:

Cat / www/logs/access.2019-02-23.log | awk'{print $7}'| sort | uniq-c | sort-rn | more

File traffic statistics:

Cat / www/logs/access.2019-02-23.log | awk'{sum [$7] + = $10} END {for (i in sum) {print sum [I], I}'| sort-rn | moregrep '200' / www/logs/access.2019-02-23.log | awk' {sum [$7] + = $10} END {for (i in sum) {print sum [I], I}'| sort-rn | more

URL traffic statistics:

Cat / www/logs/access.2019-02-23.log | awk'{print $7}'| egrep'\? | &'| sort | uniq-c | sort-rn | more

Script running speed:

Find out the slowest script

Grep-v 0 $/ www/logs/access.2019-02-23.log | awk-F'\ "'{print $4"$1} 'web.log | awk' {print $1"$8}'| sort-n-k 1-r | uniq > / tmp/slow_url.txt

IP, URL extraction:

# tail-f / www/logs/access.2019-02-23.log | grep'/ test.html' | awk'{print $1 "" $7} 'after reading the above, do you have any further understanding of how to implement Web log analysis? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.