Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use Elastic+logstash+filebeat to do Nginx log analysis. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

1. Elasticserach installation

1. Installation (elastic version 6.3.2 depends on java JDK8)

Download the appropriate version:

Curl-L-O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz

Tar-xvf elasticsearch-6.4.0.tar.gz

Cd elasticsearch-6.4.0/bin

. / elasticsearch

2. Cluster health check

Curl-XGET http://127.0.0.1:9200/_cat/health?v Cluster Health View

Curl-XGET http://127.0.0.1:9200/_cat/nodes?v node status view

Curl-XGET http://127.0.0.1:9200/_cat/indices?v View Index

Curl-XPUT http://127.0.0.1:9200/customer?pretty add customer index

Curl-XDELETE http://127.0.0.1:9200/customer?pretty delete index

3 、 Config

Cluster.name: cluster name

Node.name: node name

Path.data: data storage path

Path.logs: log storage path

Network.host: listening address

Http.port: listening port

JVM configuration, logging configuration refer to the official documentation

Important parameters:

Path settings

Cluster name

Node name

Network host

Discovery settings

Heap size

Heap dump path

GC logging

Temp directory

Important system parameters:

Disable swapping

Increase file descriptors

Ensure sufficient virtual memory

Ensure sufficient threads

JVM DNS cache settings

4 、 Running as daemon

. / bin/elasticsearch-d-p pid start

Kill cat pid stop

5 、 Set up X-Pack

Refer to the x-pack cracking method

II. Installation and configuration of kibana

Wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

Tar-xzf kibana-6.4.0-linux-x86_64.tar.gz

Cd kibana-6.4.0-linux-x86_64/

1. Start. / bin/kibana 2, config:

Server.port: 5601

Server.host: "192.168.12.81"

Kibana.index: ".kibana"

Elasticsearch.username: "elastic"

Elasticsearch.password: "dinpay"

III. Installation and configuration of logstash

Wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz

Configure filebeat to collect Nginx logs: (Nginx.conf)

Input {

Beats {

Port = > 5044

Codec = > "json"

}

}

Output {

Elasticsearch {

Hosts = > ["127.0.0.1 9200"]

Index = > "test1-nginx-access-% {+ YYYY.MM.dd}"

Template_overwrite = > true

User = > elastic

Password = > dinpay

}

}

Configure logstash.yml elastic installed locally (no https configured)

Xpack.monitoring.enabled: true

Xpack.monitoring.elasticsearch.username: elastic

Xpack.monitoring.elasticsearch.password: dinpay

Xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]

IV. Installation and configuration of filebeat

Wget https://artifacts.elastic.co/downloads/beats/filebeat/https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz

Tar-xf filebeat-6.4.0-linux-x86_64.tar.gz

Cd filebeat-6.4.0-linux-x86_64

Configure filebeat.yml

Type: log

Paths:

/ var/log/nginx/access.log

Output.logstash:

Hosts: ["192.168.12.81 purl 5044"]

Xpack.monitoring:

Enabled: true

Elasticsearch:

Hosts: ["http://192.168.12.81:9200"," http://192.168.12.81:9200"]

Username: elastic

Password: dinpay

5. Configuration of Nginx log format:

Log_format json'{"@ timestamp": "$time_iso8601",'

'"host": "$server_addr",'

'"clientip": "$clientRealIp",'

'"remote_user": "$remote_user",'

'"request": "$request",'

'"http_user_agent": "$http_user_agent",'

'"size": $body_bytes_sent,'

'"responsetime": $request_time,'

'"upstreamtime": "$upstream_response_time",'

'"upstreamhost": "$upstream_addr",'

'"http_host": "$host",'

'"url": "$uri",'

'"domain": "$host",'

'"xff": "$http_x_forwarded_for",'

'"referer": "$http_referer",'

'"status": "$status",'

'"ss": "$upstream_status"}'

Access_log / var/log/nginx/access.log json;map $http_x_forwarded_for $clientRealIp {"" $remote_addr;~ ^ (? P [0-9\.] +),?. * $firstAddr } this is the end of the article on "how to use Elastic+logstash+filebeat for Nginx log analysis". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.