Shulou(Shulou.com)06/03 Report--

This article focuses on "how to use logstash multi pipeline". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn how to use logstash multi pipeline.

I. background

Suppose we need to collect data from multiple sources, such as: 1, collecting from files. 2. Collect from tcp. The data collected from different channels need to be processed differently. So how should we deal with it at this time?

Solution 1, solution 1:

Use the same logstash instance, and then receive multiple inputs in a pipeline, such as:

Input {file {} file {} tcp {}}

Although this can be achieved, but in the filter and output stages, if different inputs have different filtering, output, etc., then the file will be very complex, need to make a variety of judgments, difficult to maintain.

2. Option 2:

Multiple logstash instances are used, and each logstash instance handles different inputs. This is possible, but logstash is a bit performance-intensive and undesirable.

3. Option 3:

In the same logstash instance, multiple pipeline are used, and each pipeline handles different input,filter and out. That is, the configuration is scattered across multiple configuration files.

III. Implementation steps

> solution 3 above is used to implement this.

1. Write pipeline files. 1. Collect from files and output to the console

Vim file-pipeline.conf

Input {file {path = > ["/ Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipeline/file-pipeline.log"] start_position = > "end" sincedb_path = > "/ Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipeline/sincedb.db" sincedb_write_interval = > "15 seconds" mode = > "tail" type = > "file-pipeline"} filter {} output {stdout {codec = > rubydebug {metadata = > true}} 2, Collect from socket Output to console

Vim tcp-pipeline.conf

# enable a tcp snooping on port 9092. It is recommended to set the value of # id to a unique value, which is very useful when using monitoring api when using multiple tcp input. Input {tcp {port = > 9202 host = > "127.0.0.1" mode = > "server" type = > "tcp-pipeline" id = > "console-tcp"}} filter {} output {stdout {codec = > line {charset = > "UTF-8"}

Note ⚠️:

1. The value of id in tcp is recommended to be set to a unique value, which is very useful when we use monitoring api when we have multiple tcp inputs.

2. Modify pipelines.yml configuration file

Vim LS_HOME/config/pipelines.yml

-pipeline.id: file-pipeline path.config: "/ Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipeline/file-pipeline.conf"-pipeline.id: tcp-pipeline queue.type: persisted path.config: "/ Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipeline/tcp-pipeline.conf"

Note ⚠️:

1. If the configuration file above uses a pipeline, such as deleting the tcp-pipeline below, change the path.config value of file-pipeline to... / * .conf

Then the output will be shared at this time, and data duplication will be found.

That is, it is modified to:

# in this way, multiple configuration files are shared by one pipeline,filter\ output and so on. -pipeline.id: file-pipeline path.config: "/ Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipeline/*.conf" 3. Start logstashbin/logstash

Note ⚠️:

1. The startup command here cannot be followed by-e or-f. If it does, the default config/pipelines.yml will not be used.

4. Test 1, test file collection cd / Users/huan/soft/elastic-stack/logstash/logstash/pipeline.conf/multi-pipelineecho "observation console output" > > file-pipeline.log2, test collection from tcp nc 127.0.0.1 9202123456-> observation console output 3, results

At this point, I believe you have a deeper understanding of "how to use logstash multi pipeline". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.