Shulou(Shulou.com)06/03 Report--

Xiaosheng blog: http://xsboke.blog.51cto.com

-Thank you for your reference. If you have any questions, you are welcome to communicate.

Catalogue

Demand environment Grok official introduction Web side configuration part of the effect shows the need to use filebeat to collect nginx access logs, output to logstash,logstash use grok filter plugin to structure the acquired nginx logs to output the structured nginx logs to the elasticsearch environment

Only grok-related configurations are listed here. For other configurations, refer to the article ELK7.4- Quick start for data collection.

Official introduction of web 172.16.100.251 nignx/filebeat/logstash elasticsearch 172.16.100.252 elasticsearch/kibanaGrok

Parse any text and structure it.

Grok is a good way to parse unstructured log data into structured and queryable content.

The tool is ideal for syslog logs, apache and other Web server logs, mysql logs, and any log format that is commonly used by humans rather than computers.

Grok syntax:% {SYNTAX:SEMANTIC}

SYNTAX: matching syntax, that is, Grok syntax. Attachment: click to view: Grok PatternsSEMANTIC: used to identify the matching string, that is, field.

An officially recommended web tool for verifying Grok syntax, using the method, write the string that needs to be parsed into the first text input box, write the written Grok expression into the second input box, and then check Named Captures Only.

Example:

Nginx Log Format:$request_time | $host | $remote_addr | [$time_local] Nginx Log:0.123 | baidu.com | 192.168.0.1 | [18/Oct/2019:11:22:14 + 0800]

-

Hint, in order to avoid the escape of "|", use "\" to disable its escape. Web configuration 1. Nginx Log Format# the log format here is more complex, in order to better display Groklog_format access'$request_time | $host | $remote_addr | $remote_user | [$time_local] | $request | $status | $upstream_status | $upstream_response_time | $upstream_addr | $body_bytes_sent | $request_body | $http_referer | $http_user_agent | $http_x_forwarded_for | $http_x_forwarded_path,$server_addr | $upstream_cache_status' 2. Filebeatvim / etc/filebeat/filebeat.ymlfilebeat.inputs:- type: log enabled: true paths:-/ var/log/nginx/jpg_access.log tags: ["nginx_access"] 3. Logstashvim / etc/logstash/conf.d/nginx.confinput {beats {port = > 5044}} filter {if "nginx_access" in [tags] {grok {match = > {"message" > "% {NUMBER" : request_time}\ |% {IPORHOST:host}\ |% {IPORHOST:remote_addr}\ |% {USERNAME:remote_user}\ | [% {HTTPDATE:time_local}\]\ |% {NOTSPACE:request_method}% {NOTSPACE:request} (?: HTTP/% {NUMBER:http_version})\ |% {NUMBER:status}\ |% {NUMBER:upstream_status}\ |% {NUMBER:upstream_response_time}\ |% {NOTSPACE:upstream_addr}\ | |% {NUMBER:body_bytes_sent}\ |% {NOTSPACE:request_body}\ |% {NOTSPACE:http_referer}\ |% {GREEDYDATA:http_user_agent}\ |% {NOTSPACE:http_x_forwarded_for}\ |% {NOTSPACE:http_x_forwarded_path}\ |% {NOTSPACE:upstream_cache_status} "}} geoip {source = >" http_x_forwarded_for "# through geoip | Library query IP attribution}} output {if "nginx_access" in [tags] {elasticsearch {hosts = > ["172.16.100.252"] index = > "nginx_access-% {+ YYYY.MM.dd}"}} effect

Then add the index on the Kibana, and you will see more custom fields on the Discover page.

This is more conducive to later data analysis, and in the Discover page can be more intuitive filter or view data.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.