Shulou(Shulou.com)06/02 Report--

This article introduces you how to use Fluentd+MongoDB to collect Apache logs, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Today we introduce a combination: Fluentd+MongoDB, which collects semi-structured data in real time.

After the background knowledge log is connected to Fluentd, it will be routed inside the Fluentd in json format. This determines that the way Fluentd handles logs is very flexible. It treats logs as semi-structured data and can easily modify its structure. Accordingly, the final storage database of logs should also be good at dealing with such semi-structured or unstructured data. In this way, the whole system can be more coordinated and efficient.

MongoDB also happens to deal with internal data in a json-like way, which is very suitable for the target storage of Fluentd.

Implementation mechanism We usually combine Fluentd+MongoDB as a pair of CP with the following architecture.

In this combination, the responsibility of Fluentd is to continuously "tail" the Apache access log to parse the Apache log text into meaningful fields (such as ip, path, etc.), and cache it to write the cached log to MongoDB periodically

Install deployment install Apache, MongoDB install Fluentd install the MongoDB plug-in in Fluentd (the latest version of Fluentd is built-in) fluent-gem install fluent-plugin-mongo

Then comes the configuration.

Configuration instructions first configure the input @ type tail path / var/log/apache2/access_log pos_file / var/log/td-agent/apache2.access_log.pos @ type apache2 tag mongo.apache.access to use tail to track Apache's log file access_log, and use Fluentd's built-in Apache log parser apache2 to parse the log. The log event tag is mongo.apache.access.

Then configure the output # plugin type @ type mongo

# mongodb db + collection database apache collection access

# mongodb host + port host localhost port 27017

# interval flush_interval 10s

# make sure to include the time key time_key time matches all tag at the beginning of mongo, using out_mongo as the output plug-in. Configure the database and collection, MongoDB address, and port in which the logs are stored in MongoDB. Set the flush interval to 10 seconds, and write cached logs to MongoDB every 10 seconds.

Test verification to ensure the normal operation of each service. We use ping Apache to create some test data. $ab-n 100-c 10 http://localhost/

Then, you can see these logs in MongoDB. $mongo > use apache > db ["access"] .findOne () {"_ id": ObjectId ("4ed1ed3a340765ce73000001"), "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200"," size ":" 44 "," time ": ISODate (" 2011-11-27T07:56:27Z ")} {" _ id ": ObjectId (" 4ed1ed3a340765ce73000002 ") "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200"," size ":" 44 "," time ": ISODate (" 2011-11-27T07:56:34Z ")} {" _ id ": ObjectId (" 4ed1ed3a340765ce73000003 ")," host ":" 127.0.0.1 " "user": "-", "method": "GET", "path": "/", "code": "200"," size ":" 44 "," time ": ISODate (" 2011-11-27T07:56:34Z ")}

On how to use Fluentd+MongoDB to collect Apache logs to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.