Shulou(Shulou.com)06/02 Report--

Some time ago, I used ossec to collect some system logs (syslog, secure, maillog, etc.). After taking a look at the architecture of elk, I found that it is very suitable for ossec and very fun.

1. Introduction:

Elk official website https://www.elastic.co/downloads

Elk consists of three open source tools, elasticsearch, logstash and kiabana.

2. Ossec+redis+elk architecture diagram:

1. The function of each application:

Ossec: event source, alert source

Redis: used to process queues to prevent data loss. Buffer data.

Logstash: it is used to collect, divide and centralize logs.

Elasticsearch: an open source distributed search engine that provides search capabilities and is used to store the final data

Kibana: web page display, supporting a variety of queries, statistics and displays

2. Workflow:

(1) ossec client sends logs to ossec server (stored in / var/logs/ossec/alerts/alerts.log) through port 1514. Logstash-shipper divides all logs of ossec server and sends the divided log contents to redis.

(2) redis is used as a buffer between ossec server and logstash indexer to improve system performance and reliability. when logstash fails to extract data, the data is saved in redis and will not be lost.

(3) logstash indexer extracts the logs of redis and collects them together (responsible for summarizing data).

(4) logstash indexer then gives the data to elasticsearch,elasticsearch to store the final data and provide search function.

(5) finally, the web interface of log analysis is provided through kibana.

3. Install elk:

1. Elk package

Elk updates quickly, and there are many versions, so if you choose a different version, you may not be able to use it.

If you install the latest version of elk,logstash3.x to change the configuration, if you use the configuration of logstash2.52, an error will be reported.

There are three ways to install elk. I choose the tar.gz package here to install it.

Logstash-1.5.2.tar.gz

Elasticsearch-1.6.0.tar.gz

Kibana-4.1.1-linux-x64.tar.gz

Redis-3.0.6.tar.gz

2. Server IP

Ossec client:192.168.153.187

Ossec server:192.168.153.172 (install ossec server and logstash and think of this server as the client of logstash (that is, logstash-shipper)

Elk+redis:192.168.153.200 (this logstash is server, or indexer)

3. Installation process

(1), 192.168.153.187

Install ossec client, see previous blog for installation

(2), 192.168.153.172

Install ossec server, see previous blog for installation

Install logstash

If logstash depends on jdk, install jdk

[root@elk-redis ~] # yum install java-1.8.0-openjdk

[root@elk-redis ~] # java-version

Openjdk version "1.8.0,91"

[root@ossec-server ~] # wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

[root@ossec-server] # tar-xf logstash-1.5.2.tar.gz-C / usr/local/

Running logstash in the background

[root@ossec-server] # / usr/local/logstash-1.5.2/bin/logstash-f / usr/local/logstash-1.5.2/logstash-200.conf &

Logstash startup completed

{

"@ timestamp" = > "2016-05-19T02:03:22.746Z"

"@ version" = > "1"

"ossec_group" > "pam,syslog,"

"reporting_ip" = > "192.168.153.187"

"reporting_source" = > "/ var/log/secure"

"rule_number" = > "5502"

"severity" = > 3

"signature" = > "Login session closed."

"@ message" = > "May 19 10:03:57 localhost sshd [4623]: pam_unix (sshd:session): session closed for user root"

"@ fields.hostname" = > "agent15"

"@ fields.product" = > "ossec"

"raw_message" = > "* * Alert 1463623401.3764:-pam,syslog,\ n2016 May 19 10:03:21 (agent15) 192.168.153.187-

> / var/log/secure\ nRule: 5502 (level 3)-> 'Login session closed.'\ nMay 19 10:03:57 localhost sshd [4623]: pam_unix

(sshd:session): session closed for user root "

"ossec_server" = > "ossec-server"

}

{

"@ timestamp" = > "2016-05-19T02:03:58.846Z"

"@ version" = > "1"

"ossec_group" > "syslog,sshd,authentication_success,"

"reporting_source" = > "192.168.153.172"

"rule_number" = > "5715"

"severity" = > 3

"signature" = > "SSHD authentication success."

"src_ip" = > "192.168.153.1"

"acct" = > "root"

"@ message" = > "May 19 10:03:57 ossec-server sshd [22805]: Accepted password for root from 192.168.153.1 port 31490

Ssh3 "

"@ fields.hostname" = > "ossec-server"

"@ fields.product" = > "ossec"

"raw_message" = > "* * Alert 1463623437.4008:-syslog,sshd,authentication_success,\ n2016 May 19 10:03:57 ossec-server-

> 192.168.153.172\ nRule: 5715 (level 3)-> 'SSHD authentication success.'\ nSrc IP: 192.168.153.1\ nUser: root\ nMay 19 10:03:57

Ossec-server sshd [22805]: Accepted password for root from 192.168.153.1 port 31490 ssh3 "

"ossec_server" = > "ossec-server"

(3), 192.168.153.200

A. Install elasticsearch

Elasticsearch depends on jdk, so install jdk first

[root@elk-redis ~] # yum install java-1.8.0-openjdk

[root@elk-redis ~] # java-version

Openjdk version "1.8.0,91"

[root@elk-redis] # tar-xf elasticsearch-1.6.0.tar.gz-C / usr/local/

Start Elasticsearch in the background

[root@elk-redis] # / usr/local/elasticsearch-1.6.0/bin/elasticsearch-d

Access port 192.168.153.200, which indicates that es starts successfully.

[root@elk-redis ~] # curl http://192.168.153.200:9200

{

"status":

"name": "elasticsearch-node01"

"cluster_name": "elasticsearch"

"version": {

"number": "1.6.0"

"build_hash": "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0"

"build_timestamp": "2015-06-09T13:36:34Z"

"build_snapshot": false

"lucene_version": "4.10.4"

}

"tagline": "You Know, for Search"

}

B. Install redis 3.0.6

[root@elk-redis ~] # tar zxvf redis-3.0.6.tar.gz

[root@elk-redis ~] # cd redis-3.0.6

[root@elk-redis ~] # make PREFIX=/usr/local/redis install

/ / tangled here. If redis does not specify a prefix path, it will compile and generate bin files in your extracted folder by default.

[root@elk-redis ~] # ln-sv / usr/local/redis/bin/redis-server / usr/bin/redis-server

[root@elk-redis ~] # ln-sv / usr/local/redis/bin/redis-cli / usr/bin/redis-cli

[root@elk-redis ~] # cp tmp/redis-3.0.6/utils/redis_init_script / etc/rc.d/init.d/redis

Configure redis

[root@elk-redis ~] # vi / etc/rc.d/init.d/redis.conf

/ / then insert the chkconfig configuration on the second line, and then modify EXEC and CLI. The first few lines of my file look like this.

#! / bin/sh

# chkconfig: 2345 90 10

# Simple Redis init.d script conceived to work on Linux systems

# as it does use of the / proc filesystem.

REDISPORT=6379

EXEC=/usr/local/redis/bin/redis-server

CLIEXEC=/usr/local/redis/bin/redis-cli

PIDFILE=/var/run/redis_$ {REDISPORT} .pid

CONF= "/ etc/redis/$ {REDISPORT} .conf"

[root@elk-redis ~] # mkdir / etc/redis/

/ / this directory is used to put our configuration files.

[root@elk-redis ~] # mkdir / var/rdb/

/ / this directory stores the database files of redis

Redis.conf is included in the redis source package, but this is only a template. The configuration is based on your own environment.

[root@elk-redis ~] # vi / etc/redis/redis.conf

Start redis

[root@elk-redis ~] # / etc/init.d/redis start

Starting Redis server...

1447 May M 18 May 17 03 it was originally set to 50.342 * Increased maximum number of open files to 10032.

_. _

_.-``_'-. _

_.-``` `_. '' -. _ Redis 3.0.6 (00000000Universe 0) 64 bit

.-``.-```. ```\ / _, _'-. _

(',.-`|`,) Running in standalone mode

| | `-. _` -.-`_ _.-.`` -. _ |'` _. -'| Port: 6379 |

| | `-. _`. _ / _. -'| PID: 1447 |

`-. _` -. _ `. /. -'_. -'

| | `-. _ _. -'|

| | `-. _` -. _ _. -'_. -'| http://redis.io |

`-. _` -. _ `. -'_. -'

| | `-. _ _. -'|

| | `-. _` -. _. -'_. -'|

`-. _` -. _ `. -'_. -'

`-. _`. _ _. -'_. -'

`-. _. -'

`-. _. -'

1447:M 18 May 17:03:50.345 # WARNING: The TCP backlog setting of 511 cannot be enforced because / proc/sys/net/core/somaxconn is

Set to the lower value of 128.

1447:M 18 May 17:03:50.346 # Server started, Redis version 3.0.6

1447:M 18 May 17:03:50.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix

This issue add 'vm.overcommit_memory = 1' to / etc/sysctl.conf and then reboot or run the command 'sysctl

Vm.overcommit_memory=1' for this to take effect.

1447 THP M 18 May 17 THP 0315 THP 50.346 # support enabled in your kernel. This will create

Latency and memory usage issues with Redis. To fix this issue run the command 'echo never >

/ sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your / etc/rc.local in order to retain the setting after a

Reboot. Redis must be restarted after THP is disabled.

1447 seconds M 18 May 17 seconds 03 seconds 50.357 * seconds

1447 The server is now ready to accept connections on port M 18 May 1715 315 * The server is now ready to accept connections on port 6379

1447 May 18 seconds 17 seconds 21 seconds 03.197 * 1. Saving...

1447 M 18 May 17 21R 03.198 * Background saving started by pid 1466

1466 C 18 May 17 21R 03.202 * DB saved on disk

1466 MB of memory used by copy-on-write C 18 May 17 MB of memory used by copy-on-write 21 MB of memory used by copy-on-write 03.202 * MB of memory used by copy-on-write

1447 M 18 May 17 21R 03.299 * Background saving terminated with success

1447 changes in M 18 May 17 seconds 26 seconds 04.090 * 10 Saving...

1447 M 18 May 17 14 26 04.090 * Background saving started by pid 1468

1468 C 18 May 1714 26 04.104 * DB saved on disk

[root@elk-redis] # redis-cli

127.0.0.1 purl 6379 > MONITOR

OK

1463623574.234636 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623575.258853 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623575.453969 [0 192.168.153.172 timestamp 36662] "rpush"logstash:redis"{\" @ timestamp\ ":\" 2016-05-19T02:03:58.848Z\ ",\" @ version

\ ":\" 1\ ",\" ossec_group\ ":\" pam,syslog,authentication_success,\ ",\" reporting_source\ ":\" 192.168.153.172\ ",\" rule_number\ ":

\ "5501\",\ "severity\": 3,\ "signature\":\ "Login session opened.",\ "@ message\":\ "May 19 10:03:57 ossec-server sshd [22805]:

Pam_unix (sshd:session): session opened for user root by (uid=0)\ ",\" @ fields.hostname\ ":\" ossec-server\ ",\" @ fields.product\ ":

\ "ossec\",\ "raw_message\":\ "* * Alert 1463623437.4316:-pam,syslog,authentication_success,\\ n2016 May 19 10:03:57 ossec-

Server- > 192.168.153.172\\ nRule: 5501 (level 3)-> 'Login session opened.'\\ nMay 19 10:03:57 ossec-server sshd [22805]: pam_unix

(sshd:session): session opened for user root by (uid=0)\ ",\" ossec_server\ ":\" ossec-server\ "}"

1463623575.456066 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623576.477031 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623601.018922 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623601.534860 [0 192.168.153.172 timestamp 36662] "rpush"logstash:redis"{\" @ timestamp\ ":\" 2016-05-19T02:05:17.007Z\ ",\" @ version

\ ":\" 1\ ",\" ossec_group\ ":\" pam,syslog,\ ",\" reporting_source\ ":\" 192.168.153.172\ ",\" rule_number\ ":\" 5502\ ",\" severity\ ": 3

\ "signature\":\ "Login session closed.\",\ "@ message\":\ "May 19 10:05:16 ossec-server sshd [22805]: pam_unix (sshd:session):

Session closed for user root\ ",\" @ fields.hostname\ ":\" ossec-server\ ",\" @ fields.product\ ":\" ossec\ ",\" raw_message\ ":\" * * Alert "

1463623516.4585:-pam,syslog,\\ n2016 May 19 10:05:16 ossec-server- > 192.168.153.172\\ nRule: 5502 (level 3)-> 'Login session

Closed.'\\ nMay 19 10:05:16 ossec-server sshd [22805]: pam_unix (sshd:session): session closed for user root\ ",\" ossec_server\ ":

\ "ossec-server\"} "

1463623601.542622 [0 127.0.0.1 48009] "blpop"logstash:redis"0"1"

1463623601.562655 [0 192.168.153.172 timestamp 36662] "rpush"logstash:redis"{\" @ timestamp\ ":\" 2016-05-19T02:05:43.092Z\ ",\" @ version

\ ":\" 1\ ",\" ossec_group\ ":\" syslog,sshd,authentication_success,\ ",\" reporting_ip\ ":\" 192.168.153.187\ ",\" reporting_source\ ":

\ "/ var/log/secure\",\ "rule_number\":\ "5715\",\ "severity\": 3,\ "signature\":\ "SSHD authentication success.",\ "src_ip\":

\ "192.168.153.1\",\ "acct\":\ "root\",\ "@ message\":\ "May 19 10:06:18 localhost sshd [4834]: Accepted password for root from

192.168.153.1 port 31537 ssh3\ ",\" @ fields.hostname\ ":\" agent15\ ",\" @ fields.product\ ":\" ossec\ ",\" raw_message\ ":\" * * Alert "

1463623542.4820:-syslog,sshd,authentication_success,\\ n2016 May 19 10:05:42 (agent15) 192.168.153.187-> / var/log/secure\

\ nRule: 5715 (level 3)-> 'SSHD authentication success.'\\ nSrc IP: 192.168.153.1\\ nUser: root\\ nMay 19 10:06:18 localhost sshd

[4834]: Accepted password for root from 192.168.153.1 port 31537 ssh3\ ",\" ossec_server\ ":\" ossec-server\ "}

C, redis set password access

[root@elk-redis ~] # vi / etc/redis/redis.conf # this file is in the root directory by default.

# remove the comments from requirepass foobared, and change foobared to your own password. Here, I change to

Requirepass xxxxxxxx

Restart the service

[root@elk-redis ~] # / etc/init.d/redis restart

Test connection:. / redis-cli-h 192.168.153.200-p 6379

Entering a command prompts (error) NOAUTH Authentication required. This is a normal phenomenon.

Let's enter the password auth xxxxxxxx # you just set.

D. Install logstash

[root@elk-redis ~] # wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

[root@elk-redis] # tar-xf logstash-1.5.2.tar.gz-C / usr/local/

Logstash profile

[root@elk-redis ~] # cat / usr/local/logstash-1.5.2/logstash-ossec.conf

Input {

Redis

{

Host = > "127.0.0.1"

Data_type = > "list"

Port = > "6379"

Key = > "logstash:redis"

Type = > "ossec"

}

}

Output {

Stdout {codec = > rubydebug}

If [type] = = "ossec" {

Elasticsearch {

Host = > "127.0.0.1"

Port = > "9300"

# cluster = > "ossec"

Index = > "logstash-ossec-% {+ YYYY.MM.dd}"

Document_type = > "ossec"

Template_name = > "template-ossec"

Template = > "/ usr/local/share/logstash/elasticsearch_template.json"

Template_overwrite = > true

}

}

}

Running logstash in the background

[root@elk-redis] # / usr/local/logstash-1.5.2/bin/logstash-f / usr/local/logstash-1.5.2/logstash-ossec.conf &

{

"@ timestamp" = > "2016-05-19T02:05:43.103Z"

"@ version" = > "1"

"ossec_group" > "pam,syslog,authentication_success,"

"reporting_ip" = > "192.168.153.187"

"reporting_source" = > "/ var/log/secure"

"rule_number" = > "5501"

"severity" = > 3

"signature" = > "Login session opened."

"@ message" = > "May 19 10:06:18 localhost sshd [4834]: pam_unix (sshd:session): session opened for user root by

(uid=0) "

"@ fields.hostname" = > "agent15"

"@ fields.product" = > "ossec"

"raw_message" = > "* * Alert 1463623542.5137:-pam,syslog,authentication_success,\ n2016 May 19 10:05:42 (agent15)

192.168.153.187-> / var/log/secure\ nRule: 5501 (level 3)-> 'Login session opened.'\ nMay 19 10:06:18 localhost sshd [4834]:

Pam_unix (sshd:session): session opened for user root by (uid=0) "

"ossec_server" = > "ossec-server"

"type" = > "ossec"

E. Install kibana

[root@elk-redis] # tar-xf kibana-4.1.1-linux-x64.tar.gz-C / usr/local/

[root@elk-redis ~] # nohup / usr/local/kibana-4.1.1-linux-x64/bin/kibana &

(4) visit kibana

Http://192.168.153.200:5601

Elk installation reference article

Http://baidu.blog.51cto.com/71938/1676798

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.