Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Fluentd for simple flow processing", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use Fluentd for simple flow processing" bar!

In some log collection scenarios, we need to convert the data flow. For example, we may need to extract some fields from the log record for error alarm, or insert new fields into the log record for subsequent analysis.

This article briefly introduces the technical details of data manipulation using Fluentd.

Filtering events based on the values of log fields when it comes to filtering, we usually think of regular expressions, and in linux we usually use grep for text lookup and filtering. Fluentd has a built-in filter_grep filtering plug-in that allows regular filtering of data streams.

Suppose we are using a web service, such as Apache, and we need to monitor its access log. The events generated by the input plug-in are similar to the following structure:

{"host": "192.168.1.1", "method": "GET", "path": "/ index.html", "code": 200, "size": 2344, "referer": null}

The code field indicates the status of the user's request. We may not care much about the request with the status of 2xx, so we can filter out such events and specifically deal with the exceptions that may occur in the user's request.

We can implement event filtering by adding the following configuration to Fluentd.

@ type grep key code pattern ^ 2\ d\ d$

Using the grep filtering plug-in, specify the code field as the filter field through key, match events with a code value of 2xx through pattern, and exclude these events.

Filter_grep can also filter multiple fields. For example, keep events with a status code of 5xx, but filter out requests in url that start with / test/. As follows:

@ type grep key code pattern ^ 5\ d\ d $key path pattern ^ / test/

Insert custom fields into the event We can insert some fields into the log record at some stage of processing for later use. This can be achieved through Fluentd's built-in filter_record_transformer filter plug-in. Assuming that we are deploying the web service in a clustered manner, we may need to mark which server the user's request is processed by. Such requirements can be achieved by configuring the following in Fluentd: @ type record_transformer server "${hostname}"

Here, the record_transformer plug-in inserts a server field into the event record, whose value is the hostname of the web server. The new log record is updated to the following format:

{"host": "192.168.1.1", "method": "GET", "path": "/ index.html", "code": 200, "size": 2344, "referer": null, "server": "app1"}

In addition to inserting predefined variables, such as ${hostname}, filter_record_transformer can also insert other variables or use ruby expressions to evaluate field values. For details, please refer to the instructions for the use of this plug-in, which we will introduce in the subsequent plug-in series.

Thank you for reading, the above is the content of "how to use Fluentd for simple flow processing". After the study of this article, I believe you have a deeper understanding of how to use Fluentd for simple flow processing, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.