Shulou(Shulou.com)06/02 Report--

What this article shares to you is about the life cycle of Fluentd events, which the editor thinks is very practical, so I share it with you to learn. I hope you can get something after reading this article.

What is an event?

Event is the data structure used by the internal processing flow of Fluentd. Once the log record enters the Fluentd, it is encapsulated into an event. Event consists of three parts: tag, time and record.

Tag identifies the source, or type, of the event for internal message routing, that is, which plug-in will handle it later

Time is the time when the event occurred

Record is the actual content of the log, which is a JSON object.

The Input plug-in is responsible for encapsulating the source data as event, such as the in_tail plug-in generating event from text. For the following line of text:

192.168.0.1-[28/Feb/2013:12:00:00 + 0900] "GET / HTTP/1.1" 200777

The following event object will be generated:

Tag: apache.access # is set according to the plug-in's tag parameter

Time: 1362020400 # 28/Feb/2013:12:00:00 + 0900

Record: {"user": "-", "method": "GET", "code": 200," size ": 777," host": "192.168.0.1", "path": "/"} # decide how to parse the single-line log record according to the parse entry in the in_tail plug-in and generate the corresponding JSON object

Below we explain the event handling process through a specific configuration.

This example uses a very basic configuration snippet to describe how the plug-ins are related, including how to define the input source (or listener) and how to set general matching rules to route the event to the output.

We use in_http and out_stdout plug-ins to describe the loop process of event.

@ type http port 8888 bind 0.0.0.0

The above configuration uses the in_http plug-in to define a HTTP server with a listening port of 8888. Then we define a Match rule according to which the event routing engine dispatches the http request to the output. The output here is stdout, which simply prints the http request to the screen.

@ type stdout

The role of Match is to set a matching rule test.cycle, for each event that enters Fluentd, if its tag value is equal to test.cycle (or matches, because match can use wildcards. The tag here is generated by the in_http plug-in. Then the event will enter the output plug-in defined by the match, and the output plug-in in this case is out_stdout.

So far, we have defined three basic items: Input, Match, and Output, although only two configuration segments are used. This is a collection configuration that can be used, which can be tested with the following command:

Curl-I-X POST-d 'json= {"action": "login", "user": 2} 'http://localhost:8888/test.cycle

You will see the following output:

HTTP/1.1 200 OKContent-Type: text/plainConnection: Keep-AliveContent-Length: 0

In / var/log/td-agent.log, you will have the following output:

2020-03-05 144168913 + 0800 test.cycle: {"action": "login", "user": 2}

Let's take a look at how events are handled and changed.

When you have a collection configuration in place, Fluentd generates various rules for processing input data. Log events go through a series of processing processes, which determines the cycle of events.

Filter (Filters)

The filter is used to filter events to determine whether to receive or discard events. We can add a filter to the example above.

@ type http port 8888 bind 0.0.0.0

@ type grep key action pattern ^ logout$

@ type stdout

After adding a filter, events must be processed by the filter before being routed to match. The filter decides whether to accept the event based on the type of event and filtering rules.

In our example, we use the grep filter, which filters events such as test.cycle and excludes events with an action value of logout in the http request.

So, if you try to send the following request, you won't see any output in td-agent.log.

Curl-I-X POST-d 'json= {"action": "logout", "user": 2} 'http://localhost:8888/test.cycle

As you can see from the example, events are processed from top to bottom according to the configuration order. We can configure as many filters as needed, so that the configuration file becomes very long and complex. Fluentd provides tags to resolve this issue.

Label (Labels)

The purpose of the tag is to define a set of configuration items that can be referenced by other configuration items to enable event routing redirection. Similar to the function of goto in programming languages.

Again in the above example, let's define a tag to see the effect.

@ type http bind 0.0.0.0 port 8888 @ label @ STAGING

@ type grep key action pattern ^ login$

@ type grep key action pattern ^ logout$

@ type stdout

This STARTING tag encapsulates the previous filter and match together and then references them in source. In this way, after the event is generated by the input plug-in, it skips the separate filter and goes directly into the processing flow defined by STARTING.

Cache (Buffers)

We saw the event generated from input, filtered by filter, and finally reached output. In the example above, we are using the stdout plug-in to output directly to the console without caching.

In practical application, the data is generally cached first, and then flush to the target storage after reaching certain conditions. This can improve the reliability of the system, and it is also very important to stabilize the system throughput. You can learn more about caching plug-ins in future articles.

In general, events continue to flow between plug-ins until they reach output, ending the entire lifecycle. As shown below:

These are the life cycles of Fluentd events, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.