Shulou(Shulou.com)05/31 Report--

This article is a detailed introduction to "what are the two new functions of Logtail in the log processing stage". The content is detailed, the steps are clear, and the details are properly handled. I hope this article "what are the two new functions of Logtail in the log processing stage" can help you solve your doubts. Let's go deeper and learn new knowledge together with the ideas of the small editor.

Log Collection Process

For log-collecting clients, the work pipeline usually consists of three processes: Input, Process, and Output.

Input: Adapts to various log access sources. Logtail currently supports data writing in two forms: text file and Syslog (TCP streaming).

Process: custom log processing logic, common examples include: log segmentation, log encoding conversion, log structured parsing, log filtering, etc.

Output: Defines log output, e.g. Logtail writes data to log service in HTTP protocol.

Log transcoding

The log service limits the character encoding of data to UTF-8, which is also Logtail's requirement for character encoding during the data sending phase.

However, some older application components may print GBK encoded data to log files when processing Chinese.

In this case, you can select the log file encoding "GBK" in the advanced options of Logtail configuration. Then, when Logtail collects logs, it will first perform GBK to UTF-8 encoding conversion on the log contents, and then perform subsequent processing.

Logtail currently supports UTF-8 and GBK file encoding formats. For GBK format, Logtail uses the iconv API of Linux system, which consumes extra machine computing resources during transcoding.

Q: How do I know if my GBK log files can be collected via Logtail?

A: Under Linux Shell, use iconv command for transcoding test. Assuming log file name is gbk.log, execute command:

iconv -f GBK -t UTF-8 gbk.log -o gbk_to_utf8.log

If the execution succeeds, it means that the file encoding is GBK; if the execution fails (similar to iconv: illegal input sequence at position 2743 error), it means that the file is not legal GBK encoding, and cannot be encoded by Logtail. Please try to adjust the log file encoding format of the application output to UTF-8.

log filtering

As an example of a web server, Nginx receives a large number of requests at all times and records these requests in access.log:

10.200.98.220 - - [25/May/2016:14:55:42 +0800] "HEAD sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 200 37 "-" "aliyun-sdk-java" 1360510.200.98.220 - - [25/May/2016:14:55:42 +0800] "POST sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 404 37 "-" "ali-log-logtail" 1360810.200.98.220 - - [25/May/2016:14:55:42 +0800] "PUT sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 401 37 "-" "aliyun-sdk-java" 1360910.200.98.220 - - [25/May/2016:14:55:42 +0800] "PUT sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 502 37 "-" "aliyun-sdk-java" 1361010.200.98.220 - - [25/May/2016:14:55:42 +0800] "GET sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 401 37 "-" "cpp-sdk-0.6" 1361210.200.98.220 - - [25/May/2016:14:55:42 +0800] "PUT sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 400 37 "-" "cpp-sdk-0.6" 1361410.200.98.220 - - [25/May/2016:14:55:42 +0800] "POST sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 400 37 "-" "ali-log-logtail" 1361510.200.98.220 - - [25/May/2016:14:55:42 +0800] "HEAD sls-pub.alibaba-inc.com/projects/ali-cn-hangzhou-sls-admin/logstores HTTP/1.1" 0.024 18204 200 37 "-" "cpp-sdk-0.6" 13616

For the problem investigation scenario, the log volume of HTTP 200 requests is usually huge. If we want to reduce the cost of log storage and upload only the request logs with exceptions, what should we do?

Today, you can turn on advanced options for Logtail configuration and set filters to solve data filtering problems.

As shown in the figure above, two filters are set for url field and status field respectively. Logs where the specified field Key exists and Value matches the regular expression are retained.

When multiple filters are defined, the judgment condition is the relationship of "and," and the log satisfying all filter settings is legal, otherwise it is discarded.

For a log, when the url field matches "(POSTs.)| (GETs.) "When the match is successful and the status field matches"[345]d+"successfully (only logs with POST and GET requests and status codes other than 200 are collected), Logtail uploads the log to the log service, as shown in the following figure:

If the name of the field that sets the filter is not found in the log, then the log is illegal and needs to be discarded. By default, all log data read and parsed successfully by Logtail is written to the log service without any filter settings by the user.

Example Configuration:

Collect only logs that match topic action or plugin

Read here, this article "Logtail in the log processing stage of the two new functions are what" article has been introduced, want to master the knowledge points of this article also need to practice to understand, if you want to know more related content of the article, welcome to pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.