Shulou(Shulou.com)05/31 Report--

Website operation and maintenance tool IIS log example analysis, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

For a website that needs long-term maintenance, how to make the website run stably for a long time is a very meaningful thing. It is also normal that some problems that are not exposed in the development phase are likely to occur in the operation and maintenance phase. There are also times when we want to constantly optimize the website to make it respond to user requests more quickly, all of which happen in the operation and maintenance phase after development.

Different from the development phase, the operation and maintenance phase can not allow you to debug the program and find all kinds of problems, we can only analyze the health of the website through a variety of system logs, for the website deployed on IIS, the IIS log provides the most valuable information, we can use it to analyze the response of the site, to determine whether the site has performance problems, or what needs to be improved.

What information is contained in the IIS log

I mentioned earlier that the IIS log provides the most valuable information. What is this information? Take a look at this screenshot:

It says:

1. When did the request take place?

two。 Which client IP accesses which port of the server IP

3. What is the type and version of the client tool?

4. The URL of the request and what are the query string parameters

5. Is the method of request GET or POST

6. What is the processing result of the request: the HTTP status code and the underlying status code of the operating system

7. During the request, how much data was uploaded by the client and how much data was sent by the server

8. How long does the request take up the server, and so on.

What is the use of this information in analysis? I'll talk about it later. Just make an impression of it first.

Configuration of IIS logs

By default, IIS generates log files, but there are some parameters that deserve our attention. The setting interface of IIS is as follows (this article takes the interface of IIS 8 as an example).

In IIS Manager, select a website and double-click the "Log" icon. Please refer to the following figure:

At this point (the main part) the interface is as follows:

In the screenshot, the log is created by generating a new file every day and generating the file name by date (this is the default).

Description: IIS uses UTC time, so I checked the bottom check box and told IIS to use local time to generate the file name.

Click the "Select Field" button, and the following dialog box appears:

Note: [number of fields sent] and [number of bytes received] are not selected by default. It is recommended to check them.

As for other fields, you can decide whether to check them or not as needed.

How to analyze IIS logs?

If you set the IIS log parameters as I described earlier, IIS will generate an IIS log after processing the request (after a while).

We can click "View Log File" in the right area of "Log Interface" to quickly locate the root directory of the IIS log, and then go to the directory to find the corresponding log file (by default, the directory will be distinguished according to the serial number of the application pool).

For example, I found the log I needed:

This file is full of characters. How can I analyze it now?

There is a tool called Log Parser that specializes in parsing IIS logs, which we can use to view the information in the logs.

For example, I can run the following command line (note: I wrap the command text in order not to affect the width of the page):

"C:\ Program Files\ LogParser 2.2\ LogParser.exe"-i:IISW3C-o:DATAGRID "SELECT CPHI IPP CSM Methodist SSHUE Portal CsMUIHUM StemMJ SCRICHUTALUS status, sc-bytes,cs-bytes,time-taken FROM u_ex130615.log"

You can now read the IIS log in tabular form:

Explanation: I do not recommend using this method to analyze IIS logs for two reasons:

1. Slow: when the log file is slightly larger, it is a waste of time to use it for analysis (especially if multiple statistics are required).

two。 Inconvenient: it does not support rich query syntax and is not as comprehensive as SQL Server query for data tables.

Recommended IIS log analysis method

Although Log Parser supports making parsed IIS logs available for reading in tabular form, sometimes when we need to do some detailed analysis, we may query in different ways [multiple times]. For this requirement, if you run Log Parser directly for each query, you will waste a lot of time. Fortunately, Log Parser supports the export of parsing results in multiple formats (the following is a screenshot of the help document):

Here, I recommend that the output format be SQL.

Note: SQL here does not refer to SQLSERVER, but to all databases that provide ODBC access.

I can import the IIS log into SQLSERVER using the following command (note: I wrap the command text so as not to affect the page width):

"C:\ Program Files\ Log Parser 2.2\ logparser.exe"SELECT * FROM'D:\ Temp\ uperex130615.log' to MyMVC_WebLog"-i:IISW3C-o:SQL-oConnString: "Driver= {SQL Server}; server=localhost\ sqlexpress;database=MyTestDb;Integrated Security=SSPI"-createtable:ON

After the import is complete, we can use the familiar SQLSERVER to do various queries and statistical analysis, such as the following query:

SELECT cip,csmethod,sport,csuristem,scstatus,scwin32status,scbytes,csbytes,timetaken FROM dbo.MyMVC_WebLog

If the following:

Note:

1. When the IIS log exports the result to SQLSERVER, the characters in the field name that do not conform to the identifier specification will be deleted.

For example, c-ip becomes cip and s-port becomes sport.

2. The time recorded in the IIS log is UTC time, and the date and time are separated. When you export to SQLSERVER, two fields are generated:

The date and time fields look uncomfortable, don't they?

I am also disgusted with this result. Here are two solutions:

1. Add a column to the SQLSERVER, and then change the UTC time to the local time zone. The T-SQL script is as follows:

Alter table MyMVC_WebLog add RequestTime datetime go update MyMVC_WebLog set RequestTime=dateadd (hh,8,convert (varchar (10), date,120) +'+ convert (varchar (13), time,114))

two。 Change the time directly when exporting the IIS log, and modify the command at this time:

"C:\ Program Files\ Log Parser 2.2\ logparser.exe"SELECT TO_LOCALTIME (TO_TIMESTAMP (ADD (TO_STRING (date, 'yyyy-MM-dd'), TO_STRING (time, 'hh:mm:ss')),' yyyy-MM-dd hh:mm:ss')) AS RequestTime, * FROM'D:\ Temp\ upex130615.log' to MyMVC_WebLog2"-i:IISW3C-o:SQL-oConnString: "Driver= {SQL Server}; server=localhost\ sqlexpress;database=MyTestDb Integrated Security=SSPI "- createtable:ON

Look at these three columns:

Select RequestTime, date, time from MyMVC_WebLog2

After dealing with this, you can delete the date and time columns directly (you can also ignore them when exporting IIS logs, but specify each field name).

So much for the problem of UTC time in the IIS log. I hope everyone understands.

Exception logging in the IIS log

The information for each request is recorded in the IIS log, including normal response requests and abnormal requests.

The [exception] mentioned here has nothing to do with the exception in. Net framework.

For an ASP.NET program, if an uncaught exception is thrown, it will be logged in the IIS log, but the exception I'm talking about is not limited to that.

The exception mentioned in this article can be divided into four parts:

1. (ASP.NET) an uncaught exception thrown by the program causes the server to produce a response output of 500.

2. There are no errors in requested resources such as 404.

3. Server errors greater than 500, for example: 502503

4. System error or network transmission error.

The first three types of exceptions can be obtained with the following query:

Select scStatus, count (*) AS count, sum (timetaken * 1.0) / 1000.0 AS sum_timetaken_second from MyMVC_WebLog with (nolock) group by scStatus order by 3 desc

There is a column in the IIS log: sc-win32-status, which records system-level errors, such as network transmission errors, that occur during request processing.

Normally, 0 indicates normal, and a non-zero value means that an error has occurred. We can count such errors as follows:

Declare @ recCount bigint; select @ recCount = count (*) from MyMVC_WebLog with (nolock) select scWin32Status, count (*) AS count, (count (*) * 100.0 / @ recCount) AS [percent] from MyMVC_WebLog with (nolock) where scWin32Status > 0 group by scWin32Status order by 2 desc

The following table lists the more common network-related errors and explanations:

All status codes can be explained by the following command:

The network name specified by D:\ Temp > net helpmsg 64 is no longer available.

With regard to scwin32status and scStatus, I would like to add that they are not related.

For example, request this address: http://www.abc.com/test.aspx

It is possible to scStatus=200, but scwin32status=64, which means that ASP.NET has successfully processed the request, but when IIS sends the response result, the client is disconnected.

Another case is: scStatus=500, but scwin32status=0, which means that an uncaught exception occurred during the processing of the request, but the exception result was successfully sent to the client.

Talk about scwin32status=64 again

I remember that I didn't understand when I saw scStatus=200,scwin32status=64 in the past, so I searched the Internet and found all kinds of answers, and some even said it was related to web crawlers. In order to verify all kinds of answers, I did an experiment. I write an ashx file and use it to simulate a long network transmission. The code is as follows:

Public class Test_IIS_time_taken: IHttpHandler {public void ProcessRequest (HttpContext context) {context.Response.ContentType = "text/plain"; System.Threading.Thread.Sleep (1000 * 2); context.Response.Write (string.Format ("{0}, {1}\ r\ n", "Start", DateTime.Now)); context.Response.Flush (); System.Threading.Thread.Sleep (1000 * 2); for (int I = 0; I < 20) ) {context.Response.Write (string.Format ("{0}, {1}\ r\ n", I, DateTime.Now)); context.Response.Flush (); System.Threading.Thread.Sleep (1000 * 1);} context.Response.Write ("End");}

This code is very simple, I do not want to explain too much, just want to say: I use Thread.Sleep and Response.Flush these two methods to simulate a long continuous sending process.

We can see this output in the browser (I took a screenshot when the display was not completely over)

I did this test eight times, only 2 times the display was completed, and the other 6 times I closed the browser window ahead of time.

Then, let's look at the contents of the IIS log:

According to the IIS log and combined with my own operations, I can find:

1. When I close the browser window in advance, I will see scStatus=200,scwin32status=64

two。 If the request is fully displayed, I will see the scStatus=200,scwin32status=0

From this experiment, we can also find that timeTaken includes network transmission time.

Based on the results of this experiment, have you ever thought about a question:

If there is a large number of scStatus=200,scwin32status=64 in your site's IIS log, and the request is initiated by the user's browser.

What is the cause of this?

My guess is that users are no longer willing to wait when they visit the site, and they close the browser window.

In other words: you can see whether the response speed of the site can satisfy users from the statistical results of scwin32status=64.

Look for performance issues

There is a column in the IIS log called timeTaken, which shows what it means in the IIS interface: all time.

This elapsed time is defined as the time from the * bytes of the request received by the server to the time all the response content is sent.

This field is described on Microsoft's website: http://support.microsoft.com/kb/944884

Once we know the definition of timeTaken, we can use it to analyze the processing time of some requests, that is, performance analysis.

For example, I want to see the load of the slowest 20 pages, which can be queried like this:

Select top 20 csuristem,scstatus,scwin32status,scbytes,csbytes,timetaken from dbo.MyMVC_WebLog with (nolock) where csUriStem like'/ Pages/%' order by timeTaken desc

Or I want to look at the responses of the 20 slowest AJAX situations, which can be queried like this:

Select top 20 csuristem,scstatus,scwin32status,scbytes,csbytes,timetaken from dbo.MyMVC_WebLog with (nolock) where csUriStem like'/ ajax/%' order by timeTaken desc

In summary, the way to find a performance problem is to select the timeTaken field in the query and use it to sort in descending order.

Note: the two fields of scbytes,csbytes are also worthy of our attention:

1. If the csbytes is too large, we need to analyze whether the form contains too much useless data and whether the form can be split.

There is also a possibility that the csbytes will get bigger: the Cookie is too large, but it will show that many requests have large csbytes, so it is easy to distinguish.

2. If the scbytes is too large, we need to check whether the page is not paged, or we can consider implementing it by loading on demand.

Typically, when you use ViewState heavily, both values become larger. So we can find the abuse of ViewState through the IIS log.

There is also a special case: uploading and downloading files will also cause these two values to become larger, and I will not explain the reasons.

Scbytes,csbytes, no matter which number is very large, will occupy the network transmission time, for users, it will need a longer waiting time.

I said three fields at once, which one should I refer to when looking for performance problems?

I think: priority should be given to timeTaken, because its value directly reflects the user's wait time (excluding front-end rendering time).

If the timeTaken is too large, it is necessary to check whether the scbytes,csbytes is also too large

If the latter two are too large, then the direction of optimization is to reduce the amount of data transmission, otherwise it means that the program processing takes up a lot of time, and optimization of the program code should be considered.

Find goals that can be improved

In addition to finding performance problems in the IIS log, you can also use it to find targets for improvement.

For example:

1. Is there a 404 error?

two。 Is there a large number of 304 requests?

3. Are there a large number of duplicate requests?

When a 404 response is found, we should analyze the cause of 404:

1. Is the user entering the wrong URL address?

two。 Or does the developer reference a resource file that does not exist?

If it is the latter, invalid references should be removed as soon as possible, because the 404 response is also a page response, and they also take up network transmission time, especially if such requests cannot be cached, it will always appear, wasting network resources.

If you want to easily find 404 errors in the development phase, you can refer to my blog: some errors that the program should find before it is released

If you find a large number of 304 requests, you should also analyze them carefully:

1. Is it a 304request due to the ASP.NET cache response?

two。 Or is it a 304 request generated when a static resource file is requested?

If it is the latter, it may have something to do with the browser settings or IIS settings.

IIS has a [enable content expiration] feature, which can be used to set cache headers when outputting responses to reduce the number of requests.

This feature is useful for static files, and the results of ASP.NET processing are not affected.

Specific setting methods can be referred to: some methods that can optimize the performance of ASP.NET websites without modifying the code

We can use this query to analyze the loading frequency of the page:

Select top 20 csUriStem, count (*) AS [count], avg (timeTaken) AS avg_timeTaken, max (timeTaken) AS max_timeTaken from MyMVC_WebLog with (nolock) where csUriStem like'/ Pages/%' group by csUriStem order by 2 desc

If you find a large number of duplicate requests, you also need to analyze them carefully:

1. Does the response content of the request vary with different parameters?

two。 The URL of the request is fixed, and the response content is rarely changed.

If it is the latter, consider using the page caching feature. For example: OutputCache of ASP.NET

Some of the ways my blog can optimize the performance of ASP.NET sites without changing the code introduces a feature that caches requests without changing the code, and try it if you need to.

The influence of Program Architecture on IIS Log Analysis process

Earlier, I introduced some methods for analyzing IIS logs, all of which are inseparable from queries. Most of the time, we need to output URL information (cs-uri-stem) in the query and make statistics according to their groups. therefore, a reasonable design of URL will bring convenience to the later statistics and get more accurate statistical results. An extreme counterexample is that with the default development method of WebForms, where the page loads and the submission of each button is the same URL, you will find it difficult to count how long each operation takes for the user.

What kind of URL design can meet the statistical needs?

I think: every user action (page display or submission) should have a URL corresponding to it, and different URL can reflect different actions.

It is also recommended that different user actions can be clearly distinguished in URL, so that it is convenient to do more statistics (for example, page loading, AJAX requests, report display).

Although we can use timeTaken for performance statistics, when you use frameset or iframe heavily in your program, it will be difficult to count how long it takes for a page (a page containing iframe) to load. Because the entire page is divided into multiple requests, they are not contiguous in the IIS log, and you cannot accurately count by user requests. For example, a1.aspx embeds b1.aspx, b2.aspx, and b3.aspx in the way of iframe. When you count the loading time of a1.aspx, the result you get is always different from what users feel, because the timeTaken of a1.aspx does not contain the timeTaken of b1.aspx, b2.aspx, and b3.aspx!

So if you want to use IIS logs to analyze program performance, then iframe should no longer be used.

After reading the above, have you mastered the method of example analysis of the website operation and maintenance tool IIS log? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.