Shulou(Shulou.com)06/01 Report--

This article introduces you to use elk to collect network equipment example analysis, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Brief introduction

With the increase of servers and network devices in the computer room, log management and query have become a headache for system administrators.

Common problems encountered by system administrators are as follows:

It is impossible to log in to every server and device to view logs during daily maintenance.

The storage space on the network device is limited, so it is impossible to store logs with too long a date, and the problems with the system may be caused by some operations that occurred a long time ago.

In some cases of illegal intrusion, the intruder will generally clear the local log and remove the trace of the intrusion.

Zabbix and other monitoring systems can not replace log management, and can not monitor items such as system login, scheduled task execution and so on.

Based on the above reasons, it is very necessary to build a Rsyslog log server for centralized log management in the current network environment.

The advantages of Rsyslog services are as follows:

Rsyslog server can be supported by most network devices, and most of the system device options of network devices have the configuration option of remote log service. Just fill in the IP address and port (most devices already default to 514), and then confirm it.

The Linux server can send logs to the log server by adding a simple line to the local Rsyslog service configuration. It is very easy to deploy and configure.

Deploy Architectur

Deploy Architectur

Rsyslog configuration system environment and software version: CentOS Linux release 7.5.1804 (Core) Elasticserch-6.8.4Kibana-6.8.4Logstash-6.8.4Filebeat-6.8.4Rsyslog-8.24.0 sets SELINUX to disabled# setenforce "sed-I" SELINUX configuration for cingingSELINUX configuration ledgeg' / etc/selinux/config firewall configuration firewall-cmd-- add-service=syslog-- permanentfirewall-cmd-- reload checks whether rsyslog software is installed

# CentOS7 installs rsyslog by default

[root@ZABBIX-Server ~] # rpm-qa | grep rsyslogrsyslog-8.24.0-16.el7.x86_64 edit rsyslog configuration file

Vim / etc/rsyslog.conf # make changes as follows

[root@ZABBIX-Server mnt] # egrep-v "* # | ^ $" / etc/rsyslog.conf$ModLoad imudp$UDPServerRun 514$ ModLoad imtcp$InputTCPServerRun 514$ WorkDirectory / var/lib/rsyslog$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat$IncludeConfig / etc/rsyslog.d/*.conf$OmitLocalLogging on$IMJournalStateFile imjournal.state*.info;mail.none;authpriv.none;cron.none;local6.none;local5.none Local4.none / var/log/messages$template h4c, "/ mnt/h4c/%FROMHOST-IP%.log" local6.*? h4c$template huawei, "/ mnt/huawei/%FROMHOST-IP%.log" local5.*? huawei$template cisco, "/ mnt/cisco/%FROMHOST-IP%.log" local4.*? cisco

$ModLoad imudp # immark is the module name, supports tcp protocol $ModLoad imudp # imupd is the module name, supports udp protocol $InputTCPServerRun 514$ UDPServerRun 514 # allows port 514 to receive logs forwarded using UDP and TCP protocols

Note:

* .info;mail.none;authpriv.none;cron.none;local6.none;local5.none;local4.none / var/log/messages does not add a local6.none;local5.none;local4.none command by default, and the network log will be written to / var/log/messages to check the rsyslog service when the corresponding file is written.

Restart the rsyslog service systemctl restart rsyslog.service log directory

Network devices point logs to syslog servers. Note that devices from different manufacturers have different local. The corresponding relationship is as follows:

/ mnt/huawei-local6/mnt/h4c-local5/mnt/cisco-local4 network device configuration Huawei:info-center loghost source Vlanif99info-center loghost 192.168.99.50 facility local5H3C:info-center loghost source Vlan-interface99info-center loghost 192.168.99.50 facility local6CISCO: (config) # logging on (config) # logging 192.168.99.50 (config) # logging facility local4 (config) # logging source-interface e0Ruijie:logging buffered warningslogging source interface VLAN 99logging facility local6logging server 192.168.99.50

Note: 192.168.99.50 is the IP of the rsyslog server

Edit filebeat profile

Collect log files under rsyslog to logstash

[root@ZABBIX-Server mnt] # egrep-v "^ # | ^ $" / etc/filebeat/filebeat.ymlfilebeat.inputs:- type: log enabled: true paths:-/ mnt/huawei/* tags: ["huawei"] include_lines: ['Failed','failed','error','ERROR','\ bDOWN\ bDOWN' bdown\ b'] drop_fields: fields: ["beat", "input_type" "source", "offset", "prospector"]-type: log paths:-/ mnt/h4c/* tags: ["H4c"] include_lines: ['Failed','failed','error','ERROR','\ bDOWN\ baud bup'] include_lines: ['Failed','failed','error','ERROR','\ bDOWN\ baud'\ bdown\ b'] drop_fields: fields: ["beat", "input_type", "source", "offset" "prospector"] setup.template.settings: index.number_of_shards: 3output.logstash: hosts: ["192.168.99.185 3output.logstash 5044"] processors:-add_host_metadata: ~-add_cloud_metadata: ~ Edit the logstash configuration file

The logs sent from filebeat are processed separately according to different tags, and the processed log data are transferred to es for storage, and further visual display is made on kibana.

[root@elk-node1 ~] # egrep-v "^ # | ^ $" / etc/logstash/conf.d/networklog.confinput {beats {port = > 5044}} filter {if "huawei" in [tags] {grok {match = > {"message" > "% {SYSLOGTIMESTAMP:time}% {DATA:hostname}% {GREEDYDATA:info}"}} else if "h4c" in [tags] {grok {match = > {"message" = > "% {SYSLOGTIMESTAMP:time}% {YEAR:year}% {DATA:hostname}% {GREEDYDATA:info}"}} mutate {remove_field = > ["message" "time", "year", "offset", "tags", "path", "host", "@ version", "[log]", "[prospector]", "[beat]", "[input] [type]", "[source]"} output {stdout {codec = > rubydebug} elasticsearch {index = > "networklogs-% {+ YYYY.MM.dd}" hosts = > ["192.168.99.185tags 9200"] sniffing = > false} make visual configuration on kibana

Create an index pattern matching stored network device log index

Create a datasheet

The data table of kibana can be exported to a CSV file

Create a pie chart

This is the end of the sample analysis of using elk to collect network devices. I hope the above content can be of some help and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.