Python rewrites Logstash to import Nginx Access Log into Elastic DB after cleaning

Step1. Modify log format of Nginx (change to JSON format)

Change the log_format of nginx to the following parameters (modify / etc/nginx/nginx.conf):

Log_format main'{"@ timestamp": "$time_iso8601", "host": "$server_addr", "clientip": "$remote_addr", "size": $body_bytes_sent, "responsetime": $request_time, "upstreamtime": "$upstream_response_time", "upstreamhost": "$upstream_addr", "http_host": "$host", "url": "$uri", "xff": "$http_x_forwarded_for", "referer": "$http_referer", "agent": "$http_user_agent" "status": "$status"}

After reload nginx, you see that the format of access.log is as follows:

{"@ timestamp": "2017-12-13T17:29:49+08:00", "host": "120.76.XX.XX", "clientip": "120.76.XX.XXX", "size": 26963, "responsetime": 0.000, "upstreamtime": "0.000", "upstreamhost": "127.0.0.1 120.76.XX.XX", "http_host": "weixin.XXX.com", "url": "/ XXXXXXX/haowanyihao/thumb.png", "xff": "111.22.65.171" "referer": "-", "agent": "WeChat/6.6.0.32 CFNetwork/811.4.18 Darwin/16.5.0", "status": "200"}

Step2. Write python programs

#-coding: utf-8 -''By Willson Luo at 2017-11-23 v1.0'''import pandas as pdimport json,time,datetime,iso8601from elasticsearch import Elasticsearchfrom geoip import geolite2# connect to elasticsearch databasees = Elasticsearch ("localhost:9200") es = Elasticsearch (hosts= [{'host':' localhost', 'port':' 9200'}], httpauth= ('elastic',' xxxxx')) # nginx column name#title = ['@ timestamp','host','clientip','size' 'responsetime','upstreamtime','upstreamhost','httphost','url','xff','referer','agent' 'status'] # nginx access logngxlog =' access.log'ngxdata = open (ngxlog). Readlines () # nginx data (json format) ngxjson = {} for A1 in range (len (ngxdata)): step1 = ngxdata [A1] .strip (). Split ("\") abc = iso8601.parsedate (step1 [3]) bcd = abc.strftime ('% Ymuri% mmi% DT% HRV% MRU% S% Z') cde = abc.strftime ('% Y% Y%) M% d') ngxindex = 'logstash-weixin-nginx-access-'+ cde ngxjson [' @ timestamp'] = bcd ngxjson ['host'] = step1 [7] ngxjson [' size'] = step1 [14] .replace (":" "). Replace (", ") ngxjson ['responsetime'] = step1 [16] .replace (": ",") .replace ("," ") ngxjson ['upstreamtime'] = step1 [19] ngxjson [' upstreamhost'] = step1 [23] if step1 [35] ="-": ngxjson ['clientip'] = step1 [11] ngxjson [' httphost'] = step1 [27] ipaddr = step1 [11] else: ngxjson ['clientip'] = step1 [35] .split (" ") [0] ngxjson ['httphost'] = step1 [39] ipaddr = step1 [35] .split (" ) [0] if "Apple" in step1 [43]: ngxjson ['agent'] = "Apple" elif "WeChat" in step1 [43]: ngxjson [' agent'] = "WeChat" elif "curl" in step1 [43]: ngxjson ['agent'] = "Linux" elif "Alibaba" in step1 [43]: ngxjson [' agent'] = "Aliyun" elif "Android" in step1 [43] : ngxjson ['agent'] = "Android" elif "MSIE" in step1 [43]: ngxjson [' agent'] = "IE" elif "Firefox" in step1 [43]: ngxjson ['agent'] = "Firefox" elif "Windows" in step1 [43]: ngxjson [' agent'] = "Windows" elif "Apache-Http" in step1 [43]: ngxjson ['agent'] = "Apache" "else: ngxjson ['agent'] = step1 [43] ngxjson [' status'] = step1 [47] location = geolite2.lookup (ipaddr). Location match = geolite2.lookup (ipaddr). Getinfodict () location = [] location.append (match ['location'] [' longitude']) location.append (match ['location'] [' latitude']) geoip = {} geoip ['location'] = location if match. Haskey ('city'): city = match [' city'] ['names'] [' en'] else: city = "-" if match.haskey ('country'): country = match [' country'] ['names'] [' en'] else: country = "-" if match.haskey ('subdivisions'): subdivisions = match [' subdivisions'] [0] ['names'] ] ['en'] else: subdivisions = "-" ngxjson [' geoip'] = geoip ngxjson ['country'] = country ngxjson [' subdivisions'] = subdivisions ngxjson ['city'] = city ngxjson [' possition'] = country+ "-" + subdivisions+ "-" + city print A1 Ngxjson es.index (index=ngxindex, doctype= "logs", body=ngxjson)

Step3. Rendering and processing of data through Kibana

1 > Import index into Kibana first (usually let you create this thing in the first step) Kibana-- > Management-- > Kibana (Index Patterns)

2 > build available views Kibana-- > Visualize (this thing is more obvious)

Step4. Build Dashboard (that is, drag the contents of Visualize)

Write Blog for the first time, it is estimated that there are a lot of mistakes and omissions, please correct, thank you