Shulou(Shulou.com)06/03 Report--

The most powerful feature of Logstash is the rich filter plug-in. This filter provides not only the function of filtering, but also complex logical processing of the original data entering the filter. Even add unique events to subsequent processes.

1. The basic syntax of logstash

Logstash is mainly composed of three parts: input, filter and output. And filter is the filter plug-in, this component can not be used, but this does not reflect the powerful filtering capabilities of logtash.

Input {input plug-in} filter {filter plug-in} outer {output plug-in}

Let's introduce each plug-in in turn.

1.1.The logstash input plug-in (input)

The input plug-in of Logstash is mainly used to receive data. Logstash supports a variety of data sources, such as reading files, standard input, reading syslog logs, reading network data, and so on. Here are the configuration methods of each receiving data source.

Let's first look at an example:

Input {file {path = > ["/ var/log/messages"] type = > "system" start_position = > "timestamp"}} output {stdout {codec = > rubydebug}}

At this point, we have a few questions:

1. How does the input of logstash receive logs?

Logstash uses a ruby gem library called filewatch to listen for changes in files and to record the progress (timestamp) of reading monitored log files through a database file called .accounedb. The default path for this sincedb data file is under / plugins/inputs/file, and the file name is similar to .uploedb _ 452905a167cf4509fd08acb964fdb20c. It represents the logstash plug-in storage directory, and the default is LOGSTASH_HOME/data. (the rpm package is installed in the / usr/share/logstash directory).

2. Which input technology of logstash is similar to that of Linux?

Input is just like the content on the right side of the Linux pipe, in the Linux pipeline, the output on the left side is passed to the right receiving mechanism of the pipe. This is the case with input. When he listens to a file, he fetches the data line by line in the form of a tail-f command. However, input can specify where to start reading (a certain timestamp), and start_position is the specified timestamp. If not specified, read from scratch. Type is used to mark the time type.

Here's an example:

1.2, standard input

Stdin gets information from standard input. Here we look at a slightly more complicated example:

Input {stdin {add_field = > {"key" = > "apple"} tags = > ["add1"] type = > "test1"}} output {stdout {codec = > rubydebug}}

Then we start the terminal, you should note that I do not specify the input file here, but specify stdin, which means standard input, which means interactive input data.

[root@::172.31.22.29 / etc/logstash/conf.d] # / usr/share/logstash/bin/logstash-f / etc/logstash/conf.d/l1.confSending Logstash logs to / var/log/logstash which is now configured via log4j2.propertieshello word # manually enter this information {"type" = > "test1", "@ version" = > "1", "message" = > "hello word" "@ timestamp" = > 2019-01-22T05:42:08.340Z, "tags" = > [[0] "add1"], "host" = > "ip-172-31-22-29.ec2.internal", "key" = > "apple"}

Type and tags are two special fields of logstash. Type is generally placed in input to mark event types. Tags is mainly used to add tags to events for use in subsequent processing processes, mainly for filter or output stages.

2. Logstash coding plug-in (Codec)

In the previous example, we have already used the coding plug-in codec, which is this rubydebug, which is a codec.

Coding plug-in (codec) can handle different types of data during logstash input or output, at the same time, it can better and more easily coexist with other custom data products, such as fluent, netflow, collectd and other common data format products. Therefore, logstash is not only an input-- > filter-- > output data flow, but also an input-- > decode-- > filter-- > encode-- > output data flow.

The common encoding formats supported by codec are plain, json, json_lines and so on. The following is described in turn:

1. Plain of the codec plug-in

Plain is the simplest coding plug-in. Whatever information you enter, you will return it, such as timestamp and type in the above example without:

Modify the configuration file:

Input {stdin {}} output {stdout {codec = > plain}}

Now let's start:

[root@::172.31.22.29 / etc/logstash/conf.d] # / usr/share/logstash/bin/logstash-f / etc/logstash/conf.d/l1.confSending Logstash logs to / var/log/logstash which is now configured via log4j2.properties2019-01-22T06:10:14.161Z ip-172-31-22-29.ec2.internal hello word # enter information 2019-01-22T06:10:19.382Z ip-172-31-22-29.ec2.internal hello word here

2. Json and json_lines of codec plug-in

Sometimes the logs collected by logstash are in JSON format, so we can add codec = > json to the input field to parse, so that the fields can be generated according to the specific content, which is convenient for analysis and storage. If you want the logstash output in json format, you can add codec= > json to the output field. The following is a time profile that contains json encoding:

Input {stdin {}} output {stdout {codec = > json}}

Start it up:

[root@:172.31.22.29 / etc/logstash/conf.d] # / usr/share/logstash/bin/logstash-f / etc/logstash/conf.d/l1.confSending Logstash logs to / var/log/logstash which is now configured via log4j2.propertieshello word # enter this line of information {"host": "ip-172-31-22-29.ec2.internal", "@ version": "1", "message": "hello word" "@ timestamp": "2019-01-22T06:27:27.191Z"}

You will find that when the plug-in is json. He will bring some information, such as host, @ version, @ timestamp. Each field in json is in key:value format, and multiple fields are separated by commas. This output is relatively long, so it is slightly better to use the json_lines encoding format.

2.1.Let 's demonstrate how to input the json format in input.

Input {stdin {codec = > json}} output {stdout {}}

Next, start the service:

[root@::172.31.22.29 / etc/logstash/conf.d] # / usr/share/logstash/bin/logstash-f / etc/logstash/conf.d/l1.confSending Logstash logs to / var/log/logstash which is now configured via log4j2.properties {"message": "hello word", "@ version": "1", "@ timestamp": "2019-01-22T06:32:56.973Z" "host": "ip-172-31-22-29.ec2.internal"} {"host" = > "ip-172-31-22-29.ec2.internal", "@ timestamp" = > 2019-01-22T06:32:56.973Z, "@ version" = > "1", "message" = > "hello word"}

This means that when I type the contents of the red box, I enter a line of JSON format here, and the system will return the information to me.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.