Shulou(Shulou.com)06/03 Report--

This blog post is only for simple cluster configuration. For more in-depth information, please refer to the official documentation.

Elasticsearch official documentation. Kibana official documentation. First, prepare before installation

The environment is as follows:

System version IP roles centos 7.5192.168.20.2ES 01, logstash, ES-Head, logstash, kibanacentos 7.5192.168.20.7ES 02centos 7.5192.168.20.8ES 03

Note: all the software packages used in this article can be downloaded from my network disk link, or go to the es Chinese community to download es-related components.

1. Configure domain name resolution [root@node01 src] # cat > / etc/hosts 192.168.20.2 node01 > 192.168.20.7 node02 > 192.168.20.8 node03 > EOF > # send hosts files to other nodes [root@node01 src] # scp / etc/hosts root@node02:/etc/hosts [root@node01 src] # scp / etc/hosts root@node03:/etc/hosts2, configure java environment

Note: the following actions need to be configured on all nodes.

Official download address of JDK

# Uninstall the included java environment [root@node01 src] # rpm-qa | grep jdkjava-1.8.0-openjdk-headless-1.8.0.161-2.b14.el7.x86_64copy-jdk-configs-3.3-2.el7.noarchjava-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64java-1.7.0-openjdk-headless-1.7.0.171-2.6.13.2.el7.x86 _ 64java-1.7.0-openjdk-1.7.0.171-2.6.13.2.el7.x86_64 [root@node01 src] # rpm-e java-1.8.0-openjdk-headless-- nodeps [root@node01 src] # rpm-e java-1.7.0-openjdk-headless-- nodeps# configuration jdk environment [root@node01 src] # tail-4 / etc/profile # append content as follows 0_241export JRE_HOME=/usr/local/jdk1.8.0_241/jreexport CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH# refreshes the configuration and checks to see if the configuration is correct [root@node01 src] #. / etc/profile [root@node01 src] # java-versionjava version "1.8.0mm 241" Java (TM) SE Runtime Environment (build 1.8.0_241-b07) Java HotSpot (TM) 64-Bit Server VM (build 25.241-b07, mixed mode) II, configure ELK Cluster 1, download rpm installation package for es

Note: download and installation need to be performed on all nodes

# download and install ElasticSearch [root @ node01 src] # wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-x86_64.rpm[root@node01 src] # ElasticSearch setting [root@node01 src] # systemctl daemon-reload [root@node01 src] # systemctl enable elasticsearch.service2, Optimize es (skip) 1) modify its default memory usage # View elasticsearch's configuration file directory [root@node01 elasticsearch] # pwd/etc/elasticsearch [root@node01 elasticsearch] # lselasticsearch.keystore jvm.options role_mapping.yml userselasticsearch.yml log4j2.properties roles.yml users_roles#elasticsearch default memory usage is 1G You can change the following configuration to modify the default memory usage [root@node01 elasticsearch] # cat jvm.options-Xms1g-Xmx1g

Note: it is recommended that the values of Xms and Xmx be set to the same in the production environment, which is generally set to half of the physical memory, but the maximum value should not exceed 30g.

2) modify the size of the number of open files

If the number of server files and threads are low, the following exception will be generated:

1. Max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] the maximum number of files opened simultaneously per process is too small. Max number of threads [3818] for user [es] is too low, increase to at least [4096] the maximum number of threads is too low

The following changes can be made to change the size of the number of files that can be opened

[root@node01 elasticsearch] # vim / etc/security/limits.conf* soft nofile 65536 * hard nofile 65536 * soft nproc 4096 * hard nproc 409 Note: after modifying the above configuration, you need to log out of the current user and log in again to take effect # after re-login You can use the following command to check whether it is in effect # View the maximum number of threads [root@node01 elasticsearch] # ulimit-Hu4096 [root@node01 elasticsearch] # ulimit-Su4096# to view the maximum number of files opened at the same time per process [root@node01 elasticsearch] # ulimit-Sn65536 [root@node01 elasticsearch] # ulimit-Hn655363, Configure es Cluster 1) node01 configuration is as follows: [root@node01 ~] # vim / etc/elasticsearch/elasticsearch.yml # modify es main configuration file cluster.name: my-es # Cluster name node.name: node01 # es Node name bootstrap.memory_lock: false # do not lock memory at startup network.host: 0.0.0.0 # listening address http.port: 9200 # listening port discovery.seed_hosts: ["node01" "node02", "node03"] # here specify the host node cluster.initial_master_nodes participating in the cluster: ["node01", "node02", "node03"] # ditto # add the following at the end of the configuration file Later, when es-head connects to the es cluster, you need http.cors.enabled: true # to add the line, enable cross-domain access to support http.cors.allow-origin: "*" # add the line, and the domain address allowed for cross-domain access # start the service And send the modified configuration file to other nodes [root@node01 ~] # systemctl start elasticsearch [root@node01 ~] # scp / etc/elasticsearch/elasticsearch.yml root@node02:/etc/elasticsearch/ [root@node01 ~] # scp / etc/elasticsearch/elasticsearch.yml root@node03:/etc/elasticsearch/2) configure other nodes # node02 node configuration as follows (just modify the node name) [root@node02 ~] # sed-I 's#node.name: Node01#node.name: node02#g' / etc/elasticsearch/elasticsearch.yml [root@node02 ~] # systemctl start elasticsearch#node03 node configuration is as follows: [root@node03 ~] # sed-I 's#node.name: node01#node.name: node03#g' / etc/elasticsearch/elasticsearch.yml [root@node03 ~] # systemctl start elasticsearch3) to check whether the cluster is configured successfully

Access port 9200 of each node and you will see the following page:

4. Configure elasticsearch Head

Google provides the elasticsearch head plug-in, which can be installed by searching the App Store as follows:

Although you can use the chrome plug-in, each user needs to install the chrome and install the ElasticSearch head plug-in. Due to the current domestic restrictions on google, it is troublesome to install chrome. As an alternative, we can install ElasticSearch Head applications on the linux server.

The plug-in exists in github, you can refer to the installation.

Note: the following configuration can be done on one of the nodes.

[root@node01 ~] # git clone git://github.com/mobz/elasticsearch-head.git [root@node01 ~] # cd elasticsearch-head/# install npm source and update openssl [root@node01 elasticsearch-head] # yum-y install epel-release [root@node01 elasticsearch-head] # yum-y update openssl# install npm (Note: epel source must be installed to install npm) [root@node01 elasticsearch-head] # yum-y install npm [root@node01 elasticsearch-head] # npm install

Note: if you stay in the following interface for a long time when executing the npm install command, or report an error, you only need to rerun the command after Ctrl+c terminates.

Start elasticsearch head in the background, otherwise it will occupy the current terminal all the time:

[root@node01 elasticsearch-head] # npm run start & # launch [root@node01 ~] # ss-lnp in the background | grep 9100 # make sure that port 9100 is listening

The browser accesses port 9100 of the host where the elasticsearch head resides and connects to port 9200 of the es cluster to view the cluster status in the browser, as follows:

Note: Elasticsearch does not allow third-party access by default. You can modify the configuration file elasticsearch.yml of Elasticsearch and add the following configuration (when I first modified the configuration file, I have added the following configuration):

Http.cors.enabled: truehttp.cors.allow-origin: "*" 5. Install kibana1) download and install Kibana [root @ node01 ~] # wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-x86_64.rpm[root@node01 src] # rpm-ivh kibana-7.5.1-x86_64.rpm2) configure kibana [root@node01 ~] # vim / etc/kibana/kibana.yml # Edit the main configuration file server.host: "0.0.0.0" # refers to Specify the kibana snooping address elasticsearch.hosts: ["http://192.168.20.2:9200"," "http://192.168.20.7:9200","http://192.168.20.8:9200"]# specifies the es snooping address You can write the listening address of a node in the cluster kibana.index: ".kibana" # add a .kibana index to the elasticsearch # set up and start Kibana [root @ node01 ~] # systemctl enable kibana [root@node01] # systemctl start kibana [root@node01 ~] # ss-lnp | grep 5601 # make sure the kibana port is listening on tcp LISTEN 0128 *: 5601 *: * users: ("node" Pid=2593,fd=28) 2) Chinese kibana (optional)

Due to the previous version of kibana 7, there is no official support for Chinese, so you need to download the patch pack.

Address: https://github.com/anbai-inc/Kibana_Hanization

In the version of kibana 7, you have officially added an option in Chinese. You only need to modify the configuration file of kibana, as follows:

[root@node01 bin] # sed-I 's/#i18n.locale: "en" / i18n.locale: "zh-CN" / g' / etc/kibana/kibana.yml

Note: after modifying the configuration file, remember to restart to take effect!

After the startup is complete, access port 5601 of the host and you will see the following interface:

6. Install logstash to collect logs

In a real production environment, most small and medium-sized architectures might look like this: filebeat=== "kafka===" logstash=== "elasticsearch===" kibana. I just want to install and configure some of the components of the new version, so I omit the first two and use logstash to collect host logs directly.

Note: the logs of nginx are collected below. Please configure nginx by yourself.

# download logstash and install [root@node01 src] # wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.rpm[root@node01 src] # rpm-ivh logstash-7.5.1.rpm# Settings Boot [root@node01 src] # systemctl daemon-reload [root@node01 src] # systemctl enable logstash.service# to edit the configuration file Collect logs [root@node01 src] # cd / etc/logstash/conf.d/ [root@node01 conf.d] # vim nginx_log.confinput {file {path = > "/ var/log/messages" type = > "system" start_position = > "beginning"} file {path = > "/ var/log/nginx/access.log" type = > "access" start_position = > "beginning" } file {path = > "/ var/log/nginx/error.log" type = > "error" start_position = > "beginning"} output {if [type] = = "system" {elasticsearch {hosts = > ["192.168.20.2 Vera 9200"] index = > "system-% {+ YYYY.MM.dd}"} } if [type] = = "access" {elasticsearch {hosts = > ["192.168.20.2access 9200"] index = > "access-% {+ YYYY.MM.dd}"} if [type] = = "error" {elasticsearch {hosts = > ["192.168.20.2access 9200"] Index = > "error-% {+ YYYY.MM.dd}"}} [root@node01 conf.d] # chmod + r / var/log/messages # gives read permission to others in this directory [root@node01 conf.d] # ln-sf / usr/share/logstash/bin/logstash / usr/local/bin/ # soft link to commands [root@node01 conf.d] # systemctl start logstash # launch Mobile service [root@node01 conf.d] # logstash-f nginx_log.conf & # is put into the background to run 7. Log in to es to see if an index is created

The display is as follows, indicating normal:

Well, as for kibana how to add the index, do your own research, do not have the patience to write, you can refer to my previous blog post to add the index on kibana.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.