Shulou(Shulou.com)06/03 Report--

I. introduction

The Elastic Stack-it is not a software, but a collection of Elasticsearch,Logstash,Kibana open source software, externally as an open source solution for a log management system. It can search logs from any source, any format, analyze and obtain data, and display it in real time. Like Shield (Security), Guardian (alarm) and Marvel (Monitoring), provide more possibilities for your product.

Elasticsearch: search, providing distributed full-text search engines

Logstash: log collection, management, storage

Kibana: web display of log filtering

Filebeat: monitoring log files, forwarding

II. Test environment roadmap

Environment: ip, hostname as planned above, the system has been update. All hosts are at the same time. The firewall test environment is closed. The following is the deployment and installation of this elk learning

Objective: to collect and monitor the system logs and online application service logs of the main server through the elk host.

III. Installation of Elasticsearch+Logstash+Kibana (operating on elk.test.com)

3.1. Basic environmental inspection

[root@elk ~] # hostnameelk.test.com [root@elk ~] # cat / etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.30.67 elk.test.com192.168.30.99 rsyslog.test.com192.168.30.64 nginx.test.com

3.2. Software package

[root@elk ~] # cd elk/ [root@elk elk] # wget-c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm[root@elk elk] # wget-c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm[root@elk elk] # wget https://download.elastic.co/kibana/ Kibana/kibana-4.5.1-1.x8664.rpm [root @ elk elk] # wget-c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm

3.3. Check

[root@elk elk] # lselasticsearch-2.3.3.rpm filebeat-1.2.3-x86_64.rpm kibana-4.5.1-1.x86_64.rpm logstash-2.3.2-1.noarch.rpm

The server only needs to install e, l, k, and the client only needs to install filebeat.

3.4. To install elasticsearch, installing jdk,elk server first requires the support of the java development environment. Because filebeat software is used on the client, it does not depend on the java environment, so it does not need to be installed.

[root@elk elk] # yum install java-1.8.0-openjdk-y

Install es

[root@elk elk] # localinstall elasticsearch-..rpm-y. Installing: elasticsearch-.-.noarch / # NOT starting on installation Please execute the following statements to configure elasticsearch service to start automatically using systemd systemctl daemon-reload systemctl enable elasticsearch.service### You can start elasticsearch service by executing systemctl start elasticsearch.service Verifying: elasticsearch-.-.noarch / Installed: elasticsearch.noarch:.-

Reload systemd, scan for new or changed units; boot and add boot self-boot

[root@elk elk] # systemctl daemon-reload [root@elk elk] # systemctl enable elasticsearchCreated symlink from / etc/systemd/system/multi-user.target.wants/elasticsearch.service to / usr/lib/systemd/system/elasticsearch.service. [root@elk elk] # systemctl start elasticsearch [root@elk elk] # systemctl status elasticsearch ● elasticsearch.service-Elasticsearch Loaded: loaded (/ usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2016-05-20 15:38:35 CST 12s ago Docs: http://www.elastic.co Process: 10428 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited Status=0/SUCCESS) Main PID: 10430 (java) CGroup: / system.slice/elasticsearch.service └─ 10430 / bin/java-Xms256m-Xmx1g-Djava.awt.headless=true-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancy...May 20 15:38:38 elk.test.com elasticsearch [10430]: [2016-05-20 15 ghans 3838 279] [INFO] [env] [James Howlett] heap. [true] May 20 15:38:38 elk.test.com elasticsearch [10430]: [2016-05-20 15 env] [James Howlett] max .65536] May 20 15:38:41 elk.test.com elasticsearch [10430]: [INFO] [node] [James Howlett] initializedMay 20 15:38:41 elk.test.com Elasticsearch [10430]: [2016-05-20 15 James Howlett 38 elasticsearch 41726] [INFO] [node] [James Howlett] starting... May 20 15:38:41 elk.test.com elasticsearch [10430]: [2016-05-20 15 15 James Howlett 38 May 41915] [INFO] [transport] [James Howlett] publ...:9300} May 20 15:38:41 elk.test.com elasticsearch [10430]: [2016- 05-20 15 elk.test.com elasticsearch 38 James Howlett 41920] [INFO] [discovery] [James Howlett] elas...xx35hwMay 20 15:38:45 elk.test.com elasticsearch [10430]: [2016-05-20 15 elas...xx35hwMay 3838 elk.test.com elasticsearch 45099] [INFO] [cluster.service] [James Howlett] new_...eived) May 20 15:38:45 elk.test.com elasticsearch [10430]: [2016-05-20 15Switzerland 3845164] [INFO] [gateway] [James Howlett] reco..._stateMay 20 15:38:45 elk.test.com elasticsearch [10430]: [2016-05-20 15 reco..._stateMay 38 elk.test.com elasticsearch 45185] [INFO] [http] [James Howlett] publ...:9200} May 20 15:38:45 elk.test.com elasticsearch [10430]: [2016-05-20 15 INFO 38 Swiss 45185] [INFO] [node ] [James Howlett] startedHint: Some lines were ellipsized Use-l to show in full.

Inspection service

[root@elk elk] # rpm-qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf [root@elk elk] # netstat-nltp | grep javatcp6 00 127.0.0.1 qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf 9200:: * LISTEN 10430/javatcp6 00:: 1LISTEN 10430/javatcp6 9200: * LISTEN 10430/javatcp6 00 127.0.0.1 LISTEN 10430/javatcp6 9300: * LISTEN 10430/java

Modify the firewall to open ports 9200 and 9300 to the public

[root@elk elk] # firewall-cmd-- permanent-- add-port= {9200xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

3.5 install kibana

[root@elk elk] # yum localinstall kibana-4.5.1-1.x86_64.rpm-y [root@elk elk] # systemctl enable kibanaCreated symlink from / etc/systemd/system/multi-user.target.wants/kibana.service to / usr/lib/systemd/system/kibana.service. [root@elk elk] # systemctl start kibana [root@elk elk] # systemctl status kibana ● kibana.service-no description given Loaded: loaded (/ usr/lib/systemd/system/kibana.service; enabled Vendor preset: disabled) Active: active (running) since Fri 2016-05-20 15:49:02 CST 20s ago Main PID: 11260 (node) CGroup: / system.slice/kibana.service └─ 11260 / opt/kibana/bin/../node/bin/node / opt/kibana/bin/../src/cliMay 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["status" "plugin:elasticsearch...May 20 15:49:05 elk.test.com kibana [11260]: {" type ":" log "," @ timestamp ":" 2016-05-20T07:49:05+00:00 "," tags ": [" status "," plugin:kbn_vi...lized "} May 20 15:49:05 elk.test.com kibana [11260]: {" type ":" log "," @ timestamp ":" 2016-05-20T07:49:05+00:00 "," tags ": [" status " Plugin:markdo...lized} May 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["status", "plugin:metric...lized"} May 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["status" Plugin:spyMod...lized} May 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["status", "plugin:status...lized"} May 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["status" "plugin:table_...lized"} May 20 15:49:05 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:05+00:00", "tags": ["listening", "info"], "pi...:5601"} May 20 15:49:10 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:10+00:00" "tags": [status "," plugin:elasticsearch...May 20 15:49:14 elk.test.com kibana [11260]: {"type": "log", "@ timestamp": "2016-05-20T07:49:14+00:00", "tags": ["status", "plugin:elasti...found"} Hint: Some lines were ellipsized, use-l to show in full.

Check that the kibana service is running (Kibana default process name: node, port 5601)

[root@elk elk] # netstat-nltpActive Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0. 0 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1 of LISTEN 909/sshdtcp 0 0. 0. 0. 0. 1 of the 25 0. 0. 0. 0. LISTEN 1595/mastertcp 0 0 0.0.0.0 5601 0.0.0.015 * LISTEN 11260/node

Modify the firewall and open tcp/5601 to the outside world

[root@elk elk] # firewall-cmd-permanent-add-port=5601/tcpSuccess [root@elk elk] # firewall-cmd-reloadsuccess [root@elk elk] # firewall-cmd-list-allpublic (default, active) interfaces: eno16777984 eno33557248 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5601/tcp masquerade: no forward-ports: icmp-blocks: rich rules:

At this point, we can open a browser, test access to the kibana server http://192.168.30.67:5601/, and make sure there is no problem, as shown below:

Here, we can modify the firewall to forward the user's access to port 80 connection to 5601, so that we can directly enter the URL without specifying the port, as follows:

[root@elk elk] # firewall-cmd-permanent-add-forward-port=port=80:proto=tcp:toport=5601 [root@elk elk] # firewall-cmd-reload [root@elk elk] # firewall-cmd-list-allpublic (default, active) interfaces: eno16777984 eno33557248 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5601/tcp masquerade: no forward-ports: port=80:proto=tcp:toport=5601:toaddr= icmp-blocks: rich rules:

3.6 install logstash and add configuration files

[root@elk elk] # yum localinstall logstash-2.3.2-1.noarch.rpm-y

Generate a certificate

[root@elk elk] # cd / etc/pki/tls/ [root@elk tls] # lscert.pem certs misc openssl.cnf private [root@elk tls] # openssl req-subj'/ CN=elk.test.com/'-x509-days 3650-batch-nodes-newkey rsa:2048-keyout private/logstash-forwarder.key-out certs/logstash-forwarder.crtGenerating a 2048 bit RSA private key.. .. +... + writing new private key to 'private/logstash-forwarder.key'-

Then create a configuration file for logstash. As follows:

View Code

Start logstash and check the port. In the configuration file, we wrote port 5000.

[root@elk conf.d] # systemctl start logstash [root@elk elk] # / sbin/chkconfig logstash on [root@elk conf.d] # netstat-ntlpActive Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0 root@elk elk 22 0.0.0.0 * LISTEN 909/sshdtcp 0 0 127.0.0.1 LISTEN 1595/mastertcp 25 0.0.0.0 * LISTEN 11260/nodetcp 0 0 0.0.0.0 514 0.0.0.0 * LISTEN 618/rsyslogdtcp6 00: 5000: * LISTEN 12819/javatcp6 00: 3306: * LISTEN 1270/mysqldtcp6 00 127.0.1 LISTEN 1270/mysqldtcp6 9200: * LISTEN 10430/javatcp6 00:: 1LISTEN 10430/javatcp6 9200: * LISTEN 10430/javatcp6 00 127.0.0.1 LISTEN 10430/javatcp6 9300: * LISTEN 10430/javatcp6 0 0: 22: * LISTEN 909/sshdtcp6 0 0:: 1:25: * LISTEN 1595/mastertcp6 0 0: 514:: * LISTEN 618/rsyslogd

Modify the firewall to open port 5000 to the public.

[root@elk ~] # firewall-cmd-- permanent-- add-port=5000/tcpsuccess [root@elk ~] # firewall-cmd-- reloadsuccess [root@elk ~] # firewall-cmd-list-allpublic (default, active) interfaces: eno16777984 eno33557248 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5000/tcp 5601/tcp masquerade: no forward-ports: port=80:proto=tcp:toport=5601:toaddr= icmp-blocks: rich rules:

3.7Modification of elasticsearch configuration file

View the directory and create a folder es-01 (the name is not required). Logging.yml is included and elasticsearch.yml is the created file, as shown below:

[root@elk ~] # cd / etc/elasticsearch/ [root@elk elasticsearch] # tree. ├── es-01 │ ├── elasticsearch.yml │ └── logging.yml └── scripts

[root@elk elasticsearch] # cat es-01/elasticsearch.yml-http: port: 9200network: host: elk.test.comnode: name: elk.test.compath: data: / etc/elasticsearch/data/es-01

Restart elasticsearch and logstash services.

3.9 copy the fiebeat installation package to the rsyslog, nginx client

[root@elk elk] # scp filebeat-1.2.3-x86_64.rpm root@rsyslog.test.com:/root/elk [root@elk elk] # scp filebeat-1.2.3-x86_64.rpm root@nginx.test.com:/root/elk [root@elk elk] # scp / etc/pki/tls/certs/logstash-forwarder.crt rsyslog.test.com:/root/elk [root@elk elk] # scp / etc/pki/tls/certs/logstash-forwarder.crt nginx.test.com:/root/elk

4. Deploy filebeat on the client side (operate on rsyslog and nginx clients)

The filebeat client is a lightweight tool that collects log resources from files on the server, which are forwarded to the Logstash server for processing. The Filebeat client communicates with the Logstash instance using the secure Beats protocol. The lumberjack protocol is designed for reliability and low latency. Filebeat uses the computing resources of the computer that hosts the source data, and the Beats input plug-in minimizes the need for Logstash resources.

4. (node1) install filebeat, copy certificates, and create a log collection profile

[root@rsyslog elk] # yum localinstall filebeat-1.2.3-x86_64.rpm-y # copy the certificate to the local specified directory [root@rsyslog elk] # cp logstash-forwarder.crt / etc/pki/tls/certs/. [root@rsyslog elk] # cd / etc/filebeat/ [root@rsyslog filebeat] # tree. ├── conf.d │ ├── authlogs.yml │ └── syslogs.yml ├── filebeat.template.json └── filebeat.yml1 directory, 4 files

There are three modified files, filebeat.yml, which define the configuration to connect to the logstash server. The two configuration files in the conf.d directory are custom monitoring logs. Let's take a look at their respective contents:

Filebeat.yml

View Code

Authlogs.yml & syslogs.yml

View Code

After the modification is complete, start the filebeat service

[root@rsyslog filebeat] # service filebeat startStarting filebeat: [OK] [root@rsyslog filebeat] # chkconfig filebeat on [root@rsyslog filebeat] # netstat-altpActive Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:25151 *: * LISTEN 6230/python2 tcp 0 0 *: ssh *: * LISTEN 5509/sshd tcp 0 0 localhost:ipp *: * LISTEN 1053/cupsd tcp 0 0 localhost: Smtp *: * LISTEN 1188/master tcp 0 0 rsyslog.test.com:51155 elk.test.com:commplex-main ESTABLISHED 7443/filebeat tcp 0 52 rsyslog.test.com:ssh 192.168.30.65 rsyslog.test.com:51155 elk.test.com:commplex-main ESTABLISHED 7443/filebeat tcp 10580 ESTABLISHED 7164/sshd tcp 0 *: ssh *: * LISTEN 5509/sshd tcp 0 0 localhost:ipp *: * LISTEN 1053/cupsd tcp 0 0 localhost:smtp *: * LISTEN 1188/master

If the connection is not available and the state is abnormal, check the client's firewall.

4.2. (node2) install filebeat, copy certificates, and create collection log configuration files

[root@nginx elk] # yum localinstall filebeat-1.2.3-x86_64.rpm-y [root@nginx elk] # cp logstash-forwarder.crt / etc/pki/tls/certs/. [root@nginx elk] # cd / etc/filebeat/ [root@nginx filebeat] # tree. ├── conf.d │ ├── nginx.yml │ └── syslogs.yml ├── filebeat.template.json └── filebeat.yml1 directory, 4 files

Modify the filebeat.yml as follows:

View Code

Syslogs.yml & nginx.yml

View Code

After the modification is complete, start the filebeat service and check the filebeat process

[root@nginx filebeat] # service filebeat startStarting filebeat: [OK] [root@nginx filebeat] # chkconfig filebeat on [root@nginx filebeat] # netstat-aulptActive Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 *: ssh *: * LISTEN 1076/sshd tcp 0 0 localhost:smtp *: * LISTEN 1155/master tcp 0 0 *: http *: * LISTEN 1446/nginx tcp 0 52 nginx.test.com:ssh 192.168.30.65:11690 ESTABLISHED 1313/sshd tcp 00 nginx.test.com:49500 elk.test.com:commplex-main ESTABLISHED 1515/filebeat tcp 00 nginx.test.com:ssh 192.168.30.65:6215 ESTABLISHED 1196/sshd tcp 00 Nginx.test.com:ssh 192.168.30.65 ssh 6216 ESTABLISHED 1200/sshd tcp 00 *: ssh *: * LISTEN 1076/sshd

As you can see from the above, the client filebeat process has connected to the elk server. Let's verify it.

5. Verify and visit kibana http://192.168.30.67

5.1 set up

Check the system logs of the next two machines: node1's

Node2's nginx access log

VI. Experience

After learning rsyslog + LogAnalyzer before, and then learning this, I found that elk is good in terms of the overall system and experience, and it is updated quickly. Later, we will continue to learn and update the relevant monitoring and filtering log methods, log analysis, and the architecture that uses kafka for storage.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.