Shulou(Shulou.com)06/02 Report--

First, background introduction

ELK is not a piece of software, but an acronym for Elasticsearch, Logstash and Kibana. All three are open source software, usually used together, and are successively owned by Elastic.co, so they are referred to as ELK Stack for short. According to Google Trend, ELK Stack has become the most popular centralized logging solution.

Elasticsearch: distributed search and analysis engine, with the characteristics of high scalability, high reliability and easy management. Based on Apache Lucene, it can store, search and analyze large-capacity data in near real time. It is usually used as the basic search engine for some applications, making it have complex search functions; Logstash: data collection engine. It supports dynamic data collection from a variety of data sources, and data filtering, analysis, enrichment, unified format and other operations, and then stored to the location specified by the user; Kibana: data analysis and visualization platform. It is usually used with Elasticsearch to search, analyze, and display the data in a statistical chart; a new member of the Filebeat:ELK stack, a lightweight open source log file data collector, based on Logstash-Forwarder source code development, is an alternative to it. After installing Filebeat on the server that needs to collect log data, and specifying the log directory or log file, Filebeat can read the data and quickly send it to Logstash for analysis, or directly to Elasticsearch for centralized storage and analysis. Architectural mode:

II. Description of Elasticsearch environment

CentOS7 4C 8G

IPportversion192.168.18.792007.5.0192.168.18.892007.5.0192.168.18.992007.5.0

III. Environmental installation and deployment

1. Download the installation package wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.0-linux-x86_64.tar.gz2, install jdk yum install-y java-1.8.0-openjdk3, and extract tar-xf elasticsearch-7.5.0-linux-x86_64.tar.gz-C / opt/.

4. Modify the configuration file

[root@es01 ~] # grep-n-v'# / etc/elasticsearch/elasticsearch.yml | grep-v'^ $'cluster.name: escluster # set the cluster name index.number_of_shards: 3 # set the number of es index shards, distributed storage, improve query efficiency, etc. Index.number_of_replicas: 1 # set the number of shard copies, provide high data availability node.name: es01 # set the name of this node The other two can set es index data storage location network.host: 192.168.18.7 # set this node binding IPdiscovery.zen.ping.unicast.hosts: ["192.168.18.7", "192.168.18.8", "192.168.18.9"] # set cluster node members

5. Detailed explanation of parameters:

1. Node.attr.rack specifies the tribal attribute of the node, which is a larger scope than the cluster.

2. Node.master specifies whether it is the primary node. This attribute can not be specified and can be elected independently between nodes.

3. Node.data specifies whether to store data (data node)

4. Cluster.initial_master_nodes parameter: you can boot the startup cluster by setting the hostname or IP address of a series of nodes that meet the conditions of the master node for the cluster.initial_master_nodes parameter. You can provide this information on the command line or in elasticsearch.yml. You also need to configure the discovery subsystem so that nodes know how to find each other.

6. Jvm.options configuration

7. Start es

Since elasticsearch cannot be started with a root user, you need to create a new user

[root@es01] # useradd elasticsearch-g root

Add elasticsearch to the root user group

[root@es01 elasticsearch-7.5.0] # chown-R elasticsearch / opt/elasticsearch-7.5.0

Switch to elasticsearch user

Su-elasticsearch

. / elasticsearch-d start

IV. Error reporting and resolution

1. Seccomp unavailable error solution: elasticsearch.yml configuration bootstrap.memory_lock: false#bootstrap.systemcallfilter: false2. Max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536] solution: modify / etc/security/limits.conf, configuration: elasticsearch soft nofile 65535 # soft limit where elasticsearch is the startup user elasticsearch hard nofile 65535 # hard limit 3. Max virtual memory areas vm.maxmapcount [65530] is too low solution: modify / etc/sysctl.conf Add: vm.max_map_count=262144 and then sysctl-p takes effect 4. The default discovery settings are unsuitable...., last least one of [....] Must be configured solution: elasticsearch.yml enable configuration: node.name: node-1cluster.initialmasternodes: ["es01"] *

The configuration of other nodes is the same.

V. Verification

You can see that they all belong to the same cluster by accessing the mapped ports of the three machines through the browser.

At this point, the elasticsearch cluster deployment is complete. The deployment and log collection methods of kibana and logstash will be introduced later.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.