Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to make a fluentd image with kafka plug-ins and es plug-ins. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Preface

Fluentd is an open source data collector for unified logging layer. It is the sixth CNCF graduation project after Kubernetes, Prometheus, Envoy, CoreDNS and containerd. It is often used to compare elastic's logstash. Relatively speaking, fluentd is more lightweight and flexible. Now the development is very fast and the community is very active. When writing this blog, github's star is 8.8k fork is 1k.

Premise

Docker

Dockerfile file writing

Dockerfile

FROM fluent/fluentd:v1.3.2

ADD fluent.conf / etc/fluent/

RUN echo "source 'https://mirrors.tuna.tsinghua.edu.cn/rubygems/'" > Gemfile & & gem install bundler

RUN gem install fluent-plugin-kafka-v 0.12.3-- no-document

RUN gem install fluent-plugin-elasticsearch-v 4.0.3-- no-document

CMD ["fluentd"]

Fluent.conf

@ type kafka

Brokers kafka:9092

Format json

Topic kafeidou

@ type elasticsearch

Host elasticsearch

Port 9200

Index_name fluentd

Type_name fluentd

Based on the fluentd image of version 1.3.2, because the processing layer extension of fluentd is extended as a plug-in, it is necessary to install the corresponding kafka plug-in and elasticsearch plug-in when making this image.

Here the fluentd plug-in version of kafka is 0.12.3 and the fluentd plug-in version of ElasticSearch is 4.0.3.

Dockerfile and fluent.conf are ready to execute the command to make an image.

Docker build-t fluentd-es-kafka:v1.3.2.

In this way, the fluentd image containing the es plug-in and the kafka plug-in is finished.

It only takes a docker command to run such a fluentd.

Docker run-it-d fluentd-es-kafka:v1.3.2

After startup, the container starts to listen for kafka messages with host of kafka and transfer data to elasticsearch nodes with host of elasticsearch.

If the node address of es is different from that of kafka, you need to hang the default configuration file in the volume overlay container.

Docker run-it-v {directory where fluent.conf is stored}: / etc/fluent-d fluentd-es-kafka:v1.3.2 about how to make fluentd images with kafka plug-ins and es plug-ins is shared here. I hope the above content can be of some help and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.