Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build a log system in Filebeat5+Kafka+ELK Docker. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Preparatory work

All installation, configuration, or documentation are official, such as docker.io, elastic.co, kafka.apache.org, etc.

If the old articles from Baidu can't solve the problem, the consequences will be self-imposed.

Prepare three virtual machines with different ip (explained in A B C below), do clusters, standard!

Do a good job in psychological expectations, Google science search can not be less, monster problems will emerge one after another, must not be scared!

Why does the name version number Filebeat5.0.0-alpha1Kafka0.9.0.1Zookeeper3.4.8Elasticsearch2.4.0Logstash2.4.0Kibana4.6.0 use Docker

Purely for personal hobbies, all kinds of technologies like the Docker mirror version of Docker as long as they are close to it. In this article, except that the binary version of filebeat agent is installed directly on the application machine and has nothing to do with docker, all other cluster installations are based on the docker image version.

Why use Kafka

Distributed publish and subscribe-based messaging system Kafka, which can well glue the business application side (client) and log analysis server (server) together, and plays a buffering role, and provides many excellent features such as asynchronism, decoupling, persistence, sequencing and so on. And Kafka can be integrated with many open source components such as Storm, Spark, etc., which will be of great help to the future expansion of this layer.

Why choose Filebeat5?

It is true that Flume is the first choice at the beginning, and there are two things to do: ① uploads the log files on the client side to Kafka; ② to consume Kafka queue messages to be stored in ElasticSearch.

At that time, the latest release of Flume version is 1.6.0, while it supports up to 1.7.5 es version, does not support 2.x version, in the middle of the es did a variety of downgrades and even have to cooperate with jdk8 and so on, and finally gave up.

The only choice is logstash-forwarder and filebeat, and the latter is actually the upgrade + replacement of the former, so there is no doubt that you should choose filebeat directly.

Monster line again, the latest stable version of filebeat is 1.3.0, and it does not support output to kafka. That is, the ① thing was stuck. Fortunately, the alpha1 beta version of Beats5 was released. Although it is unstable, the log has not been found in the test, so use it first.

Before I lost my voice, alpha2 released it again.

Https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha2.html

The first ② thing is achieved through Logstash (because the docker image is the hihahiha of elk)

Opening and finishing

After asking 100,000 why, we can finally open the whole (in fact, the front # why # is also the history of my blood and tears.)

1. Installation of Zookeeper

Introduce a good docker image directly, and use it directly under pull.

Docker pull jeygeethan/zookeeper-cluster

Start the commands at the three points of the cluster, and customize the virtual volumes.

Docker run-name docker-zk-d-restart=always\

-net= "host"\

-p 2181 purl 2181\

-p 2888 purl 2888\

-p 3888 purl 3888\

-v ~ / dockerdata/zookeeper/lib:/var/lib/zookeeper\

-v ~ / dockerdata/zookeeper/log:/var/log/zookeeper\

-v / etc/localtime:/etc/localtime:ro\

Jeygeethan/zookeeper-cluster 192.168.0.1192.168.0.2192.168.0.3 1 {1-2-3: three nodes are set}

After all three nodes are started successfully, enter node A

Run docker exec-it docker-zk bash

The / usr/share/zookeeper directory will be entered by default.

Run bin/zkCli.sh

Entered the client command line of zk

Create node create / nicholas "nicholas"

Check the node get / nicholas to show that it has been created successfully

Execute the get operation on virtual machines B and C to check whether the new nodes are synchronized, so it is successful.

# II. Installation of Kafka

Similarly, pull image first.

Docker pull jeygeethan/kafka-cluster

Start on the same three nodes, notice that I use the same three virtual machines here for kafka and zk.

Docker run-- name docker-kafka-d-p 9092

-e KAFKA_HOST=192.168.0.1

-e KAFKA_PORT=9092

-e ZOOKEEPER_CONNECT=192.168.0.1:2181192.168.0.2:2181192.168.0.3:2181

-e KAFKA_ID=0 {0amp 1amp 2: three nodes are set separately, starting with 0}

-v ~ / dockerdata/kafka/logs:/tmp/kafka-logs

Jeygeethan/kafka-cluster

Enter virtual machine An and virtual machine B at the same time

Enter docker

Docker exec-it docker-kafka bash

Conversion directory

Cd / opt/kafka_2.11-0.9.0.1/bin

Create Topic

. / kafka-topics.sh-- create-- topic TP_NIC-- partitions 4-- replication-factor 2

-- zookeeper 192.168.0.1 virtual 2181192.168.0.2purl 2181192.168.0.3purl

View Topic

. / kafka-topics.sh-describe-topic TP_NIC

-- zookeeper 192.168.0.1 virtual 2181192.168.0.2purl 2181192.168.0.3purl

Produce messages on broker0 (virtual machine A)

. / kafka-console-producer.sh-- topic=TP_NIC\

-- broker-list=192.168.0.1:9092192.168.0.2:9092192.168.0.3:9092

Consume messages on broker1 (virtual machine B)

. / kafka-console-consumer.sh-- topic=TP_NIC\

-- zookeeper 192.168.0.1 virtual 2181192.168.0.2purl 2181192.168.0.3purl

At this point, virtual machines An and B have been associated with TP_NIC, on the A command line, enter all kinds of characters at will, and the same characters can be seen on B means that the consumption is successful.

III. Installation of ELK

The latest version 5 of the 2.4 version of pull image is already available.

Docker pull sebp/elk:es240_l240_k460

Modify the directory permissions of the virtual volume to be mapped. Note that 991992993 here corresponds to three independent users of ELK. If you take a look at docker file's build script, you can see that in order for docker to run successfully, let's first configure the permissions.

Chown-R 991 / dockerdata/es & & chown-R 992 / dockerdata/logstash & & chown-R 993 / dockerdata/kibana

Enter the corresponding directory and set the configuration first.

Note: these configuration files are cp from docker. If the source file is not available, please start the original image by docker run and then copy it by docker cp.

Elasticsearch configuration

Vi ~ / dockerdata/es/config/elasticsearch.yml

The editing content is as follows

Cluster.name: mm-cluster

Node.name: mm-node-01

Node.master: false

Node.data: true

# restrict outside access

Network.host: 192.168.0.11

Transport.tcp.port: 9300

Http.port: 9200

Path.data: / etc/elasticsearch/data

Path.work: / etc/elasticsearch/work

Path.logs: / etc/elasticsearch/logs

Path.plugins: / etc/elasticsearch/plugins

Bootstrap.mlockall: true

Discovery.zen.ping.multicat.enabled: false

Discovery.zen.fd.ping_timeout: 100s

# discovery.zen.fd.ping_retries: 6

# discovery.zen.fd.ping_interval: 30s

Discovery.zen.ping.timeout: 100s

Discovery.zen.minimum_master_nodes: 1

Discovery.zen.ping.unicast.hosts: ["192.168.0.11", "192.168.0.12", "192.168.0.13"]

Gateway.recover_after_nodes: 2

# action.auto_create_index: false

Index.number_of_replicas: 0

Index.number_of_shards: 2

Kibana configuration

Vi ~ / dockerdata/kibana/config/kibana.yml

Check elasticsearch.url: "just match the http://localhost:9200"."

Logstash configuration

Add input configuration file for Kafka:

Vi ~ / dockerdata/logstash/config/03-kafka-input.conf

-

Input {

Kafka {

Zk_connect = > "192.168.0.1 virtual 2181192.168.0.2purl 2181192.168.0.3 purl 2181"

# group_id = > ""

Topic_id = > "syslog"

Codec = > "plain"

Reset_beginning = > false

Consumer_threads = > 5

Decorate_events = > true

Add_field = > {"[@ metadata] [type]" = > "syslog"}

}

}

Modify the log resolution filtering configuration file:

Vi ~ / dockerdata/logstash/config/10-syslog.conf

-

Filter {

If [@ metadata] [type] in ["syslog", "accesslog"] {

Ruby {

Code = > "event ['mlogsendts'] = event [' @ timestamp']"

}

Mutate {

Add_field = > ["mlogsendts_string", "% {@ timestamp}"]

}

Json {

Source = > "message"

Add_field = > {

"mlogmsg" = > "% {message}"

}

Remove_field = > ["message"]

}

Grok {

Patterns_dir = > ["/ opt/logstash/patterns"]

Match = > {"mlogmsg" = > "[% {MMLOGTS:mlogts}]\ s [% {MMLOGWORDEXT:mlogcell}]\ s [% {MMLOGWORDEXT:mlognode}]\ s [% {MMLOGWORDEXT:mlogthread}]\ s [% {MMLOGWORD:mloglevel}]\ s [% {MMLOGWORDEXT:mlogclass}]\ s% {GREEDYDATA}"}

}

Grok {

Match = > {"source" = > "% {GREEDYDATA} /% {GREEDYDATA:mlogfilename} .log"}

}

Syslog_pri {}

Date {

Match = > ["mlogts", "yyyy-MM-dd HH:mm:ss.SSS"]

Timezone = > "Asia/Shanghai"

Target = > "@ timestamp"

}

}

}

The most complicated thing here is actually two things.

① replaces the system @ timstamp with the timestamp in the log (see configuration)

The ② grok expression parses the variables in the log in segments (find an online grok verification tool to verify your own regularity, which is very laborious! )

Grok expressions support regular format variables in various formats by default. Search for them on your official website. Here I am some custom regular variables, which are stored in:

Vi ~ / dockerdata/logstash/patterns/mmlog

.

The logstash file under patterns will be scanned automatically by default, so the file name can be defined casually, as long as the rule is correct.

The content is:

MMLOGTS {4} -\ d {2} -\ d {2}\ s\ d {2}\:\ d {2}\:\ d {2}.\ d {3}

MMLOGWORD\ w

MMLOGWORDEXT [^]] +

MMLOGTHREAD\ w (\ w)\:\ w -\ w

MMLOGCLASS [\ w.] +\:\ w\ s

Now you can start docker. For both Elasticsearch and Logstash in ELK, there need to be three points in the cluster, while Kibana just shows the data and a single point can be used. So the startup scripts are:

Kibana single point virtual machine, all three of ELK are enabled. EBay's heap size configuration is based on the fact that the virtual machine is 2c4g (relatively low configuration)

Docker run-- privileged=true-I-d-p 5601-p 9200-p 9200-p 9300-p 5044-50005 000\

-net= "host"\

-v ~ / dockerdata/es/config/:/etc/elasticsearch/:rw\

-v ~ / dockerdata/es/plugins/:/etc/elasticsearch/plugins/:rw\

-v ~ / dockerdata/es/data/:/etc/elasticsearch/data/:rw\

-v ~ / dockerdata/es/work/:/etc/elasticsearch/work/:rw\

-v ~ / dockerdata/es/logs/:/var/log/elasticsearch/:rw\

-v ~ / dockerdata/logstash/config/:/etc/logstash/conf.d/:rw\

-v ~ / dockerdata/logstash/patterns/:/opt/logstash/patterns/:rw\

-v ~ / dockerdata/logstash/logs/:/var/log/logstash/:rw\

-v ~ / dockerdata/kibana/config/:/opt/kibana/config/:rw\

-v ~ / dockerdata/kibana/logs/:/var/log/kibana/:rw\

-v / etc/localtime:/etc/localtime:ro\

-e ES_JAVA_OPTS= "- Duser.timezone=Asia/Shanghai"\

-e ES_HEAP_SIZE= "2g"\

-e LS_HEAP_SIZE= "1g"\

-e ELASTICSEARCH_START=1-e LOGSTASH_START=1-e KIBANA_START=1\

-- name docker-elk sebp/elk:es240_l240_k460\

& & docker logs-f docker-elk

The startup and shutdown Kibana (KIBANA_START=0) of the other two points, and the virtual machine configuration is 4c8g (relatively high configuration).

Docker run-- privileged=true-I-d-p 5601-p 9200-p 9200-p 9300-p 5044-50005 000\

-net= "host"\

-v ~ / dockerdata/es/config/:/etc/elasticsearch/:rw\

-v ~ / dockerdata/es/plugins/:/etc/elasticsearch/plugins/:rw\

-v ~ / dockerdata/es/data/:/etc/elasticsearch/data/:rw\

-v ~ / dockerdata/es/work/:/etc/elasticsearch/work/:rw\

-v ~ / dockerdata/es/logs/:/var/log/elasticsearch/:rw\

-v ~ / dockerdata/logstash/config/:/etc/logstash/conf.d/:rw\

-v ~ / dockerdata/logstash/patterns/:/opt/logstash/patterns/:rw\

-v ~ / dockerdata/logstash/logs/:/var/log/logstash/:rw\

-v ~ / dockerdata/kibana/config/:/opt/kibana/config/:rw\

-v ~ / dockerdata/kibana/logs/:/var/log/kibana/:rw\

-v / etc/localtime:/etc/localtime:ro\

-e ES_JAVA_OPTS= "- Duser.timezone=Asia/Shanghai"\

-e ES_HEAP_SIZE= "4G"\

-e LS_HEAP_SIZE= "2g"\

-e ELASTICSEARCH_START=1-e LOGSTASH_START=1-e KIBANA_START=0\

-- name docker-elk sebp/elk:es240_l240_k460\

& & docker logs-f docker-elk

Open the following URL to verify whether the installation is successful:

Kibana Web: http://:5601

Elasticsearch Json: http://:9200/

To install the plug-in for ES, please enter docker and download it. Head, hq and other plug-ins are easier to use.

IV. Installation of Filebeat5

This is the easiest. Download the filebeat-5.0.0-rc1-linux-x86_64.tar.gz decompression and installation on the official website.

Vi / usr/local/src/filebeat5/filebeat.yml

The editing content is as follows:

# Filebeat Configuration Example #

# # Filebeat # #

Filebeat.prospectors:

# Each-is a prospector. Below are the prospector specific configurations

-input_type: log

Paths: ["/ usr/local/src/logs/$ {appname}-${cellname}-${nodename} / sys-.log"]

Encoding: utf-8

Exclude_files: ['.\ d {4} -\ d {2} -\ d {2}.. log']

Document_type: syslog

Fields:

Mlogapp: ${appname}

Fields_under_root: true

Scan_frequency: 1s

Ignore_older: 30m

# must set ignore_older to be greater than close_inactive.

Close_inactive: 5m

Close_removed: true

Clean_removed: true

Multiline:

Pattern: ^ [: digit:]] {4}-[: digit:] {2}-[[: digit:]] {2} [[: blank:]] [[: digit:]] {2}: [: digit:]] {2}: [: digit:] {2}. [: digit:] {3}]

Negate: true

Match: after

Max_lines: 500

Output.kafka:

# initial brokers for reading cluster metadata

Hosts: ["192.168.0.1", "192.168.0.2", "192.168.0.3"]

# message topic selection + partitioning

Topic:'% {[type]}'

Partition.round_robin:

Reachable_only: false

Required_acks: 1

Compression: gzip

Max_message_bytes: 1000000

# # Logging # #

Logging.level: info

Logging.to_files: true

Logging.to_syslog: false

Logging.files:

Path: / usr/local/logs/filebeat

Name: filebeat.log

Keepfiles: 7

.

.

.

--

* start Filebeat5:**

Export appname= "uss-web" & & export cellname= "cell01" & & export nodename= "node01"\

& & cd / usr/local/src/filebeat/\

& & nohup. / filebeat-e > / usr/local/src/logs/filebeat/nohup.out 2 > & 1 & that's all about how Filebeat5+Kafka+ELK Docker builds the log system. I hope the above can help you and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.