Shulou(Shulou.com)06/01 Report--

Fluentd is a log collection system, its characteristic is that its parts are customizable, you can collect logs to different places through simple configuration. At present, the open source community has contributed the following storage plug-ins: MongoDB, Redis, CouchDB,Amazon S3, Amazon SQS, Scribe, 0MQ, AMQP, Delayed, Growl and so on. The purpose of this article is to introduce MongoDB support that is already built into the latest version of Fluentd. It is mainly illustrated by an example of collecting Apache logs: mechanism diagram.

Installation to complete the relevant tests, you need to install the following components: Fluentd with MongoDB PluginMongoDBApache (with the Combined Log Format) has included the MongoDB plug-in in the latest installation package of Fluentd. You can also use the command gem install fluent-plugin-mongo to install the Debian PackageRPM PackageMongoDB Downloads configuration. If you installed Fluentd using the deb/rpm package above, the configuration file is located at: / etc/td-agent/td-agent.conf. Otherwise, the location should be: / etc/fluentd/fluentd.conf first we edit the source in the configuration file to set the log source type tail format apache path / var/log/apache2/access_log tag mongo.apache where: type tail: tail is the built-in input mode of Fluentd, which is based on the principle of constantly getting new logs from the source file. Format apache: specifies the use of Fluentd's built-in Apache log parser. Path / var/log/apache2/access_log: specify the log file location. Tag mongo.apache: specify that the tag,tag is used to classify different logs and then edit the output configuration. After the logs are collected and stored in the MongoDB, the # plugin type type mongo # mongodb db + collection database apache collection access # mongodb host + port host localhost port 27017 # interval flush_interval 10smatch tag can be followed by a regular expression to match the specified tag. Only the logs corresponding to the matching tag will use the configuration. Other items in the configuration are easy to understand, just look at the comments, where flush_interval is used to control how often the log is written to MongoDB. The test uses ab tool to access Apache. To generate corresponding access logs for collecting $ab-n 100-c 10 http://localhost/, and then we can see the collected logs in MongoDB: $mongo > use apache > db.access.find () {"_ id": ObjectId ("4ed1ed3a340765ce73000001"), "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/" "code": "200,200", "size": "44", "time": ISODate ("2011-11-27T07:56:27Z")} {"_ id": ObjectId ("4ed1ed3a340765ce73000002"), "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "44" "time": ISODate ("2011-11-27T07:56:34Z")} {"_ id": ObjectId ("4ed1ed3a340765ce73000003"), "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200"," size ":" 44 " "time": ISODate ("2011-11-27T07:56:34Z")} Source: blog.treasure-data.com

Translation: http://blog.nosqlfan.com/html/3521.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.