Shulou(Shulou.com)05/31 Report--

What is the working principle of Logtail, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

The principle of Logtail data acquisition is introduced, including file acquisition principle and plug-in acquisition principle.

Principle of document collection

The process of Logtail file collection includes six steps: file monitoring, file reading, log processing, log filtering, log aggregation and data sending. The following are described separately:

Note: this section only describes the file collection principle of Logtail in normal operation mode, which does not support the collection of history files. If you need to collect history files, please refer to the collection history files.

File monitoring

When Logtail acquires the collection configuration, it recursively scans the log directories and files in the directory that conform to the file name rules according to the configured log path, file name and maximum monitoring directory depth.

To ensure the timeliness and stability of log collection, Logtail monitors the registration events of the collection directory (Inotify under Linux and ReadDirectoryChangesW under Windows) and polls regularly.

When the configuration is applied for the first time, the log files stored in the directory will not be collected until a modification event occurs after the configuration application.

When the file is monitored for modification, it will enter the file reading process.

File reading

Each Logtail read starts at the offset where the file was last read.

If the file is read for the first time, the file size will be checked. If the file is less than 1MB, it will be read from the header, otherwise it will be read from the tail 1MB.

The maximum amount of 512KB data is read at a time, so a log supports a maximum of 512KB.

Log processing

For the read data block, the line is split according to the line header configuration and is divided into multiple logs.

Perform corresponding parsing for each log content, such as regularity, delimiter, JSON, and so on.

If the time field is not configured, the log time is the current resolution time; if the time extraction field is configured, the time is extracted from the resolved log field. If the time is more than 12 hours from the current time, the log is discarded and the error message is uploaded.

If the log can be parsed correctly, enter the log filtering process.

If the log parsing fails and the discard parsing failure log in the advanced configuration is enabled, the log will be discarded directly and the error message of parsing failure will be reported.

If the log parsing fails, but discarding the failed log in the advanced configuration is not enabled, the original log that failed in parsing will be uploaded, where Key is _ _ raw_log__ and Value is the log content.

Log filtering

If the user does not set the filter configuration in the advanced configuration, skip the log filtering.

If the user has set the filter configuration, all fields in each log will be traversed and verified.

Only if all the fields configured in the filter appear in the log, and all the corresponding fields match the filter configuration, the log will be collected, otherwise the log will be discarded.

Log aggregation

In order to reduce the number of network requests, after log processing and filtering, it will be cached in Logtail for a period of time before sending.

There are three cache rules. If any one of them is satisfied, it will trigger sending:

Log aggregation time exceeds 3 seconds

The number of log aggregates exceeds 4096

The total size of log aggregation exceeds 1MB

Log sending

Logs are compressed before they are sent. Currently, Logtail uses the LZ4 compression algorithm.

Log sending is limited by max_bytes_per_sec and send_request_concurrency. Logtail ensures that the sending rate and concurrency do not exceed the configuration value. For more information, please see startup parameters configuration.

If the data transmission fails, choose whether to retry or discard the data according to the error message:

401 error, indicating that you do not have permission to collect data, so it is discarded directly.

404 error, indicating that project or logstore does not exist and is discarded directly.

Error 403, Quota out of limit, wait 3 seconds and try again.

500 error, wait 3 seconds and try again.

The network timed out. Wait 3 seconds and try again.

Principle of plug-in acquisition

The plug-in collection process of Logtail mainly includes the following steps: plug-in data collection, data processing, log aggregation and log sending.

Plug-in data acquisition

The principle of plug-in data collection is described in the documentation of each plug-in, see the help documentation of each plug-in for details.

Data processing.

Plug-in data processing logic, please refer to the plug-in-data processing.

Log aggregation

The log aggregation logic of the plug-in is consistent with the log aggregation logic of file collection.

Log sending

The log sending logic of the plug-in is consistent with that of file collection.

Resource restriction

Logtail will work according to the resource limit in the configuration file. If the resource takes longer (5 minutes) than the limit, Logtail will restart it forcefully. A certain amount of data duplication may occur after reboot.

Reliability of data acquisition

When collecting data, Logtail will save the collected CheckPoint information locally on a regular basis. In case of outage, Crash and other anomalies, Logtail will start collecting data from the last recorded location again to ensure that the data is not lost as much as possible.

Logtail uses many mechanisms to improve the reliability of log collection, but there is no guarantee that logs will not be lost. The following conditions may result in log loss:

Logtail is not running and the log is rotated multiple times.

The speed of log rotation is extremely fast, for example, one rotation per second.

The speed of log collection can not reach the speed of log generation for a long time.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.