Shulou(Shulou.com)06/01 Report--

Linux log analysis of several commonly used commands, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Sometimes we need to do some analysis and simple statistical work on the online application log. If we are familiar with several commands of text processing under Linux, we may have unexpected gains:

More / less view file contents

Head/tail looks at the contents of the lines at the beginning or end of the file

Grep looks in the file

Awk text processing

Sort sorting

Sed text processing

Here are two examples to illustrate the simple application of these commands.

1. Count the interface calls from Alipay to Taobao and from Taobao to Alipay today.

The first step is to find the log file path and determine the log format

Because the API call is TC, first ssh to a tc server, go to the log directory, ll, and see several log files named alipay. I feel that the log is called by the print API.

More alipay-notify-success.log

Output:

2009-06-29 00 INFO alipay-notify-success-

* alipay notify callback*

Out_trade_no=T200P2062628786,trade_status=WAIT_BUYER_PAY,notify_action_type=createPartnerTradeAction

Input:

As can be seen from the log, this is the log of Alipay callback Taobao API. The format includes: time, order number, status, notification type and so on.

More alipay.log

Output:

2009-06-29 00 INFO core.SignedTbClientInvoker 0000072 []

* Payway Request and Response*

Service Name:

Trade_create

Request URL:

Http://aligw.alipay.com

From the log, this is the log of Taobao calling Alipay. The format includes: time, interface type, request content and so on.

Step two, find the feature text

The number of calls to the interface needs to be counted by type, so the feature string is the type text of the interface, using the grep command

Grep', notify_action_type' alipay-notify-success.log

The second log is special, with a single line of characteristic text and no fixed prefix or suffix, while the first line is a fixed "Service Name:". You can find the previous line, and then output more than one line when you output grep:

Grep 'Service Name:'-A1 alipay.log

The grep command has many optional parameters, such as ignoring case, output before (- B before), after (- An after) line text, and so on.

The third step is text sorting.

Usually, a line of text in a log file consists of several columns, with a separated string in the middle, and our goal is to find the needed columns and do related calculations and statistics. Here we need to use the awk command.

For the alipay-notify-success.log log file, we use the following command:

Grep', notify_action_type' alipay-notify-success.log | awk-Flying print'{a [$3] + +} END {for (i in a) print I "," a [I]}'

The grep command finds the matching line as input to the awk command, followed by a specified delimiter followed by an expression. First, an array a (which can also be understood as a map) is defined with a delimited third column as the subscript (key), the value is added one at a time, and the END is followed by the last executed statement, looping out the array.

The awk command itself is powerful, so you can take a comprehensive look at its help

Step 4, sort.

In the third step, the calculation and statistics have been completed. Finally, we sort them according to the number of calls, which is convenient for viewing.

Use the sort command here

Grep', notify_action_type' alipay-notify-success.log | awk-Fleming print'{a [$3] + +} END {for (i in a) print I "," a [I]}'| sort-t,-K2-n-r

-t is similar to ark's-F function in that it divides a line of text into columns,-k specifies the columns to be sorted,-n for numerical sorting, and-r for reverse order

Finally, we see the output:

Notify_action_type=createPartnerTradeAction,52641

Notify_action_type=payByAccountAction,44807

Notify_action_type=sellerSendGoodsAction,43848

Notify_action_type=confirmReceiveAction,40705

Notify_action_type=modifyTradeAction,25733

Notify_action_type=allowRefundAction,10407

Notify_action_type=autoFinishTradeAction,8351

Notify_action_type=closeTradeAction,8030

Notify_action_type=applyRefundiiiAction,2653

Notify_action_type=refundDisburseAction,2330

Notify_action_type=confirmDisburseAction,401

Notify_action_type=extendTimeoutLimitAction,368

Notify_action_type=modifyRefundiiiAction,280

Notify_action_type=cancelRefundiiiAction,52

Notify_action_type=null,20

Notify_action_type=unfreezeTradeAction,1

Notify_action_type=refundVoucherCheckPassAction,1

Notify_action_type=freezeTradeAction,1

Of course, this is a single machine, according to the number of machines applied, you can roughly assess the overall situation.

Similarly, for log alipay.log

Grep 'Service Name:'-A1 alipay.log | sed' / Service Name:/'d | sed'/-/ 'd | awk-F''{a [$2] +} END {for (i in a) print I "," a [I]}'| sort-t,-K2-n-r

Output:

Trade_create,51326

Send_goods_confirm_by_platform,40716

ConfirmReceiveGoods,39351

ModifyTradeFee,25261

Cae_charge_agent,10074

Close_trade,3871

ExtendTimeout,378

Calculate_service_fee,52

Union_data_prepare,15

Logistic_sign_in,4

Next, let's look at another example.

Requirements: count the number of requests and average response time of my Taobao homepage on the denali machine.

Or in a few steps:

1. Find the log and log format first

Apache access log, / home/admin/cai/logs/cronlog/2009/06/2002009-06-29-taobao-access_log

More 2002009-06-29-taobao-access_log

Output:

58.208.1.15 148452 3251 [29/Jun/2009:00:00:04 + 0800] "GET http://my.taobao.com/mytaobao/home/my_taobao.jhtml" 14147" http://my.t

Aobao.com/mytaobao/home/my_taobao.jhtml "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; QQDownload 551; User-agent)

: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;); SLCC1; .NET CLR 2.0.50727; MDDC; .NET CLR 3.5.30729; .NET CLR 3.0.30618)

2. Find the feature text

Grep 'GET http://my.taobao.com/mytaobao/home/my_taobao.jhtml' 2009-06-29-taobao-access_log

3. Text categorization

Grep 'GET http://my.taobao.com/mytaobao/home/my_taobao.jhtml' 2009-06-29-taobao-access_log | awk-F'' {iatrophid2} END {print NR "," i/NR/1000} "

After reading the above, have you mastered the methods of several common commands for log analysis under Linux? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.