How to build a log collection system using Fluentd and MongoDB

How to use Fluentd and MongoDB to build a log collection system, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail, people with this need can come to learn, I hope you can gain something.

Installation

In order to complete the relevant tests, the following components need to be installed:

Fluentd with MongoDB Plugin

MongoDB

Apache (with the Combined Log Format)

The MongoDB plug-in is already included in Fluentd's latest installation package, and you can also use the command

Gem install fluent-plugin-mongo

To install

Debian Package

RPM Package

MongoDB Downloads

Configuration

If you installed Fluentd using the deb/rpm package above, the configuration file is located at: / etc/td-agent/td-agent.conf, otherwise it should be at: / etc/fluentd/fluentd.conf

First, we edit the source in the configuration file to set the log source

Source >

Type tail

Format apache

Path / var/log/apache2/access_log

Tag mongo.apache

/ source >

How to build a log collection system using Fluentd and MongoDB

Where:

① type tail: tail is a built-in input method for Fluentd, which works by constantly fetching new logs from the source file.

② format apache: specifies the use of Fluentd's built-in Apache log parser.

③ path / var/log/apache2/access_log: specify the log file location.

④ tag mongo.apache: specifies that tag,tag is used to classify different logs

Next, edit the output configuration, configure the log collection and store it in MongoDB.

Match mongo.** >

# plugin type

Type mongo

# mongodb db + collection

Database apache

Collection access

# mongodb host + port

Host localhost

Port 27017

# interval

Flush_interval 10s

/ match >

The match tag can be followed by a regular expression to match the tag we specified. Only the logs corresponding to the matching tag will apply the configuration inside. Other items in the configuration are easy to understand, just look at the comments, where flush_interval is used to control how often the log is written to MongoDB.

test

Access the Apache with the ab tool to generate the corresponding access log for collection

$ab-n 100-c 10 http://localhost/

Then we can see the collected logs in MongoDB.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.