Shulou(Shulou.com)06/02 Report--

ELK Log Analysis system introduction Log Server improves security centralized storage of logs defects analysis of logs difficult ELK log analysis system ElasticsearchLogstashKibana log processing steps centralized management of logs formatting (Logstash) and output to Elasticsearch for formatted data rope bow | and storage (Elasticsearch) front-end data presentation (Kibana) Elasticsearch introduction to Elasticsearch provides a distributed multi-user overview Ability of full-text search bow | the concept of Elasticsearch is close to real-time cluster node index (library)-> type (table)-> document (record) shards and replicas Logstash introduction Logstash introduces a powerful data processing tool Can achieve data transmission, format processing, format output data input, data processing (such as filtering (rewrite, etc.) and data output LogStash main components ShipperIndexerBrokerSearch and StorageWeb InterfaceKibana introduction Kibana introduces an open source analysis and visualization platform for Elasticsearch to search and view the data stored in Elasticsearch cable bow | to conduct advanced data analysis through various charts and display the main functions of Kibana Elasticsearch seamlessly integrate data, complex data analysis benefits more team members with flexible interfaces Sharing is easier to configure, visual multi-data source simple data export deployment ELK log analysis system experimental environment node1 node server IP address: 192.168.80.128node2 node server IP address: 192.168.80.129apache server IP address: 192.168.80.800 in node1 Install Elasticsearch [root @ node1 ~] # vim / etc/hosts / / configure resolution name 192.168.80.128 node1192.168.80.129 node2 [root@node1 ~] # java-version / / check whether [root@node1] # mount.cifs / / 192.168.80.2/LNMP-C7 / mnt/Password for root@//192.168.80.2/LNMP-C7 is installed on Java: [root@node1 mnt] # cd / mnt/elk/ [root@node1 elk] # rpm-ivh elasticsearch-5.5.0.rpm / / install [root@node1 elk] # systemctl daemon-reload / / reload daemon [root@node1 elk] # systemctl enable elasticsearch.service / / Boot Auto start [root@node1 elk] # cd / etc/elasticsearch/ [root@node1 elasticsearch] # cp elasticsearch.yml elasticsearch.yml.bak / / backup [root@node1 elasticsearch ] # vim elasticsearch.yml / / modify the configuration file cluster.name: my-elk-cluster / / Cluster name node.name: node1 / / Node name The second node is node2path.data: / data/elk_data / / data storage location path.logs: / var/log/elasticsearch/ log storage location bootstrap.memory_lock: false / / do not lock memory at startup network.host: 0.0.0.0 / / provide the IP address of the service binding For all addresses http.port: 9200 # # Port number is 9200discovery.zen.ping.unicast.hosts: ["node1" "node2"] / / Cluster discovery implements [root@node1 elasticsearch] # mkdir-p / data/elk_data/ / create data store point [root@node1 elasticsearch] # chown elasticsearch.elasticsearch / data/elk_data/ gives permission to [root@node1 elasticsearch] # systemctl start elasticsearch.service / / enable service [root@node1 elasticsearch] # netstat-ntap | grep 9200 / / check See if tcp6 00: 9200: * LISTEN 2166/java is enabled

In node1 Install the node component dependency package on node2 [root@node1 elasticsearch] # yum install gcc gcc-c++ make-y / / install the compilation tool [root@node1 elasticsearch] # cd / mnt/elk/ [root@node1 elk] # tar zxvf node-v8.2.1.tar.gz-C / opt/ extract the plug-in [root@node1 elk] # cd / opt/node-v8.2.1/ [root@node1 node-v8.2.1] # . / configure / / configure [root@node1 node-v8.2.1] # make & & make install / / compile and install in node1 Install the phantomjs front-end framework on node2 [root@node1 elk] # tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2-C / usr/local/src/ extract to / usr/local/src [root@node1 elk] # cd / usr/local/src/phantomjs-2.1.1-linux-x86_64/bin/ [root @ node1 bin] # cp phantomjs / usr/local/bin/ compiler system identified in node1 Install elasticsearch-head data Visualization on node2 [root@node1 bin] # cd / mnt/elk/ [root@node1 elk] # tar zxvf elasticsearch-head.tar.gz-C / usr/local/src/ extract [root@node1 elk] # cd / usr/local/src/elasticsearch-head/ [root@node1 elasticsearch-head] # npm install / / install and modify configuration file [root@node1 elasticsearch-head] # vim / etc/elasticsearch/ Elasticsearch.yml / / Last line join http.cors.enabled: true / / enable cross-domain access support The default is falsehttp.cors.allow-origin: "*" / the domain address allowed for cross-domain access [root@node1 elasticsearch-head] # systemctl restart elasticsearch.service / / restart [root@node1 elasticsearch-head] # cd / usr/local/src/elasticsearch-head/ [root@node1 elasticsearch-head] # npm run start & / / run data visualization service [1] 82515 at the background [root@node1 elasticsearch-head] # netstat-ntap | grep 9100tcp 00 0.0.0.0 grep 9100tcp 9100 0.0.0.0 LISTEN 82525/grunt [root@node1 elasticsearch-head] # netstat-ntap | grep 9200tcp6 00:: 9200: * LISTEN 82981/java check the health value status on the browser

Create an index on node1

Create index information [root@node2 ~] # curl-XPUT 'localhost:9200/index-demo/test/1?pretty&pretty'-H' content-Type: application/json'-d'{"user": "zhangsan", "mesg": "hello world"} 'view in the browser

Install logstash on the Apache server Multi-elasticsearch docking [root@apache ~] # yum install httpd-y / / installation service [root@apache ~] # systemctl start httpd.service / / start service [root@apache ~] # java-version [root@apache ~] # mount.cifs / / 192.168.100.8/LNMP-C7 / mnt/ mount Password for root@//192.168.100.8/LNMP-C7: [root@apache ~] # cd / mnt/elk/ [root@apache elk] # rpm-ivh logstash-5.5.1.rpm / / install logstash [root @ apache elk] # systemctl start logstash.service [root@apache elk] # systemctl enable logstash.service / / set boot self-boot [root@apache elk] # ln-s / usr/share/logstash/bin/logstash / usr/local/bin/ make it easy for the system to identify [root@apache elk] # logstash- e 'input {stdin {} } output {stdout {}}'/ / Standard input and output The stdin plugin is now waiting for input:16:58:11.145 [Api Webserver] INFO logstash.agent-Successfully started Logstash API endpoint {: port= > 9600} www.baidu.com / / input 2019-12-19T08:58:35.707Z apache www.baidu.comwww.sina.com.cn / / input 2019-12-19T08:58:42.092Z apache www.sina.com.cn [root @ apache elk] # logstash-e 'input {stdin {}} output {stdout {codec= > rubydebug}' / / use rubydebug to display detailed output Codec is a codec The stdin plugin is now waiting for input:17:03:08.226 [Api Webserver] INFO logstash.agent-Successfully started Logstash API endpoint {: port= > 9600} www.baidu.com / / formatting {"@ timestamp" = > 2019-12-19T09:03:80.267Z, "@ version" = > "1", "host" = > "apache" "message" = > "www.baidu.com"} [root@apache elk] # logstash-e 'input {stdin {} output {elasticsearch {hosts= > ["192.168.80.129root@apache elk"]}}' # # use logstach to write information to elasticsearch The stdin plugin is now waiting for input:17:06:46.846 [Api Webserver] INFO logstash.agent-Successfully started Logstash API endpoint {: port= > 9600} www.baidu.com / / Enter the information www.sina.com.cn to view the information in the browser

Output Syslog files to Elasticsearch [root @ apache elk] # chmod vim / var/log/messages / / give other users read permissions [root@apache elk] # vim / etc/logstash/conf.d/system.conf / / create a file input {file {search = > "/ var/log/messages" / / output directory type = > "system" start_position = > "beginning"}} output {elasticsearch enter address pointing to node1 node hosts = > ["192.168.80.129beginning 9200"] index = > "system-% {+ YYYY.MM.dd}"} [root@apache elk] # Systemctl restart logstash.service / / restart the service to view details with data browsing

Install kibana data Visualization [root@node1] # cd / mnt/elk/ [root@node1 elk] # rpm-ivh kibana-5.5.1-x86_64.rpm / / install [root@node1 elk] # cd / etc/kibana/ [root@node1 kibana] # cp kibana.yml kibana.yml.bak / / backup [root@node1 kibana] # vim kibana.yml on the node1 server / / modify the configuration file server.port: 5601 / / Port number server.host: "0.0.0.0" / / listen on any network segment elasticsearch.url: "http://192.168.80.129:9200" / / Native node address kibana.index:" .kibana " / / Index name [root@node1 kibana] # systemctl start kibana.service / / enable the service [root@node1 kibana] # systemctl enable kibana.service browser to access kibana

Docking apache log files in apache server Make statistics [root@apache elk] # vim / etc/logstash/conf.d/apache_log.conf / / create the configuration file input {file {path = > "/ etc/httpd/logs/access_log" / / input information type = > "access" start_position = > "beginning"} File {path = > "/ etc/httpd/logs/error_log" type = > "error" start_position = > "beginning"}} output {if [type] = = "access" {/ / output information elasticsearch {is judged according to the condition Hosts = > ["192.168.80.129 YYYY.MM.dd 9200"] index = > "apache_access-% {+ YYYY.MM.dd}"} if [type] = = "error" {elasticsearch {hosts = > ["192.168.80.129 YYYY.MM.dd 9200"] index = > "apache _ error-% {+ YYYY.MM.dd} "}} [root@apache elk] # logstash-f / etc/logstash/conf.d/apache_log.conf / / configure logstach access web page information according to the configuration file View kibana statistics

Select management > Index Patterns > create index patterns; to create the information of two apache logs

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.