Shulou(Shulou.com)06/03 Report--

Filebeat 6.4.3 collecting nginx logs

Nginx log format specification:

Add a log configuration combination to the nginx main configuration file

Log_format eslog'$remote_addr-$remote_user [$time_local] "$request"'$status $body_bytes_sent "$http_referer"'"$http_user_agent"$http_x_forwarded_for"; ES cluster installs plug-ins

Ingest-user-agent and ingest-geoip need to be installed to collect nginx logs in the cluster. After the installation, you need to restart the es service, otherwise the data cannot be entered into the es cluster normally.

/ usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip/usr/share/elasticsearch/bin/elasticsearch-plugin list upgrade java to 1.8yum install java-1.8-y install filebeat Import key

Rpm-- import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create a repo file

More lostash.repo

[elastic-6.x]

Name=Elastic repository for 6.x packages

Baseurl= https://artifacts.elastic.co/packages/6.x/yum

Gpgcheck=1

Gpgkey= https://artifacts.elastic.co/GPG-KEY-elasticsearch

Enabled=1

Autorefresh=1

Type=rpm-md

Installation

Yum install filebeat-y

Modify the filebeat main configuration file

More / etc/filebeat/filebeat.yml

Filebeat.config:

Prospectors:

Path: ${path.config} / prospectors.d/.yml

Reload.enabled: false

Modules:

Path: / etc/filebeat/modules.d/.yml

Reload.enabled: false

Output.elasticsearch:

Hosts: ['10.2.3.30mer 9200']

Setup.kibana:

Host: "10.2.3.13 5601"

Configure the filebeat nginx module Access log: / data/nginxlog/eslog/es-access.*Error log: / data/nginxlog/eslog/es-error.*more / etc/filebeat/modules.d/nginx.yml-module: nginx # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. Var.paths: ["/ data/nginxlog/eslog/es-access.*"] # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. Var.paths: ["/ data/nginxlog/eslog/es-error.*"] add launch systemctl enable filebeat to open the nginx module cd / etc/filebeatfilebeat modules enable nginxfilebeat modules list initialization environment

This automatically imports the filebeat template and nginx dashboard into the es cluster:

Set up the initial environment:Loaded index templateLoading dashboards (Kibana must be running and reachable) Loaded dashboardsLoaded machine learning job configurations./filebeat setup-e startup service systemctl start filebeat startup service error

The contents are as follows:

Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory / usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory / usr/share/filebeat/kibana/6/index-pattern: error loading / usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file:. Response: {"statusCode": 400, "error": "Bad Request" "message": "Request Timeout after 30000ms"} 2018-10-31T16:35:45.659+0900 INFO kibana/client.go:113 Kibana url: http://10.2.3.13:56012018-10-31T16:37:15.664+0900 ERROR instance/beat.go:743 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory / usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory / usr/share/filebeat/kibana/6/index -pattern: error loading / usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response: handling method:

Log in to kibana and enter Dev tools to delete.

Delete old template # DELETE _ template/filebeat-6.X.X delete old data # DELETE filebeat-6.4.2-* re-import template: cd / etc/filebeat/filebeat setupLoaded index templateLoading dashboards (Kibana must be running and reachable) Loaded dashboardsLoaded machine learning job configurations restart service: systemctl restart filebeatDashboard effect:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.