Shulou(Shulou.com)06/03 Report--

Typically, log files are in text format, where the content is an unstructured text string. This makes us query log information, generally can only use the search function of text editing software, after entering keywords, rely on eyesight to detect each matching result. When the log is small, or just check it occasionally, it doesn't hurt to do so. However, no matter how simple things are, they are afraid to repeat them many times. If frequent queries are required, quantitative changes may lead to qualitative changes. If you have to rely on manual search every time, then no matter how good your eyesight is, there will be times of dizziness. Therefore, if you want to easily query the log, you must find the right tool, and with the right tool, you can hit enter while drinking coffee.

In the tool, the first thing that comes to mind is the use of various computer development languages, plus relational databases. However, the development process of such tools is tedious, and many working environments need to be prepared, including configuring the language development environment, installing database services, installing database query applications, and so on.

For such a "heavy" plan, we resolutely put aside. Because today we are going to introduce a lightweight and convenient tool-the aggregator, which can turn text logs into structured data, and then we can use the familiar SQL-style queries.

Here, we take advantage of the two advantages of Structured Process Language (SPL for short):

The log content is structured into a data table structure. SPL is much simpler, easier to use and intuitive than common development languages.

SPL supports SQL queries on structured files directly, eliminating the need to install and configure third-party database software.

The following is the specific implementation process.

1. Log structure analysis

Different log files have a variety of content formats, each of which looks disorganized. But for a particular log, it must have its own structure. After getting the log file, the first thing to do is to analyze the log content, refine the data structure, and summarize the fields that can be structured.

As an example, we use a startup log under Tencent Video software to do a case. If you have also used Tencent Video, you can use the following code to experience and learn to analyze your usage behavior. This log file is located under the AppData path of the current user and starts with QQLive.exe [Main]. On my machine, this file is:

C:\ Users\ [Joancy]\ AppData\ Roaming\ Tencent\ QQLive\ Log\ QQLive.exe [main] [2018-8-3 21-5-35-557] [12164] .log

[Joancy] is my Windows login user name in the above path, and it will be your user name on your machine. There are many log files at the beginning of QQLive.exe [Main]. Just take one.

Here are two lines in the log file:

[18-07-19 14:35:06] [9416]-[31ms] [QQLiveMainModule.dll] [CQQLiveModule::ParsCommandLine] cmd= "C:\ Program Files (x86)\ Tencent\ QQLive\ QQLive.exe"-system_startup

[18-07-19 14:35:08] [9416]-[2266ms] [HttpModule.dll] [CDownloadMgr::AddTask] keyid = 1pm url = http://182.254.116.117/d?dn=vv.video.qq.com.&ttl=1

As you can see, the content of this log is relatively regular, one record at a time. The content in a pair of parentheses in each line is a section, corresponding to a field. It's just that the last two sections are a little special, in which the penultimate section can be omitted, while the last section is not enclosed in parentheses. In this way, we can sort out the data structure of the log table as follows, and take the first row as a corresponding example:

Field name type section content 1 recording time DateTime [18-07-19 14:35:06] 2 thread number Integer [9416] 3 load time Integer [31ms] 4 load module String [QQLiveMainModule.dll] 5 load function String [CQQLiveModule::ParsCommandLine] 6 day log content Stringcmd= "C:\ Program Files (x86)\ Tencent\ QQLive\ QQLive.exe"-system_startup

Table (1)

When parsing each field, you need to pay attention to:

1) recording time: since there are only two digits in the year, you need to specify a matching date format when converting to the date-time type, otherwise 18 will be regarded as AD 18 instead of 2018. The specific operation method is to open the options in the aggregator menu, click on the environment page in the pop-up window, and set the property 'date-time format' to 'yy-MM-dd HH'.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.