Shulou(Shulou.com)06/01 Report--

Preface

Log is a very important part of the system, through which the problems existing in the system can be found in time, and can also provide clues for repairing problems. Docker provides a variety of plug-in ways to manage logs, this article will record the process of using MongoDB to store Docker logs.

Data stream

The date generated by Docker is sent to FluentD, and then FluentD is transferred to MongoDB.

Configure Fluentd

Prerequisite: operating system is CentOS7, MongoDB3.4 is installed

1. Install Fluentd

The CentOS 7 software library does not include an installation package for Fluentd, so it needs to be installed manually. The software td-agent used in CentOS.

# curl-L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

The above command creates an installation library for td-agent, td.repo, in the / etct/yum.repo.d/ directory, and then automatically installs td-agent. Set to start automatically after installation.

# systemctl start td-agent# systemctl enable td-agent

If there is a problem during startup, you can view the log file td-agent.log under / var/log/td-agent/

two。 Configure docker log rollover to MongoDB

Transferring to MongoDB requires plug-in support, but td-agent does not install the plug-in by default and needs to be installed.

# yum-y install td-agent-gemt# td-agent-gem install fluent-plugin-mongo

Edit / etc/td-agent/td-agent.conf, add

# # match tag=docker.* and dump to mongodb## when docker starts the container, the tag of the log should be in the form of docker.* @ type mongo # database address host 127.0.0.1 # database port port 22017 # database and dataset name database docker collection log for storing logs

Restart td-agent.

Start the container

The command to start the container is changed to:

$docker run-mount type=bind,source=/home/demo/app,destination=/app-log-driver=fluentd-log-opt tag= "docker.log" python3:latest python3 / app/test.py

After running, you can view the log in MongoDB.

Follow up

However, in the actual use process, it is found that the logs generated by Docker are not immediately written to MongoDB, with a delay of about 1 minute.

Check the Fluentd documentation to see that there is a flush parameter that controls how often the log is written, and the default is 60s. Change the flush_interval parameter to increase the write frequency, as follows:

# # match tag=docker.* and dump to console @ type mongo host 127.0.0.1 port 22017 database docker collection log flush_mode interval flush_interval 1s

I didn't understand the document very well at first, but after adding the flush_interval parameter, it didn't work, and when I first changed it, I didn't put flush_mode and flush_interval in the buffer structure. That is:

# # match tag=docker.* and dump to console @ type mongo host 127.0.0.1 port 22017 database docker collection log flush_mode interval flush_interval 1s

You can also see from the log that the above configuration did not work:

2018-01-19 14:33:32 + 0800 [warn]: parameter'* * flush_mode**' in @ type mongo host "1270.0.0.1" port 22017 database "docker" collection "log" flush_mode interval flush_interval 1s buffer_chunk_limit 8m time_key time time_key time time_format% Y-%m-%d% H:%M:%S.%L is not used.

It says flush_mode didn't work.

Summary

The above is the whole content of this article, I hope that the content of this article has a certain reference and learning value for your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Reference documentation:

Https://docs.treasuredata.com/articles/td-agenthttps://docs.docker.com/engine/admin/logging/fluentd/https://docs.fluentd.org/v1.0/articles/out_mongo

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.