Shulou(Shulou.com)06/02 Report--

This article mainly explains the principle and application of ELK Stack. The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn the principle and application of ELK Stack.

Introduction to ELK Stack

ELK is not a piece of software, but an acronym for Elasticsearch, Logstash and Kibana. All three are open source software, usually used together, and are successively owned by Elastic.co, so they are referred to as ELK Stack for short. According to Google Trend, ELK Stack has become the most popular centralized logging solution.

Elasticsearch: distributed search and analysis engine, with the characteristics of high scalability, high reliability and easy management. Based on Apache Lucene, it can store, search and analyze large-capacity data in near real time. It is usually used as the basic search engine for some applications, making it have complex search functions.

Logstash: data collection engine. It supports dynamic collection of data from various data sources, filtering, analysis, enrichment, uniform format and other operations, and then stored in the location specified by the user.

Kibana: data analysis and visualization platform. It is usually used in conjunction with Elasticsearch to search, analyze and display the data in a statistical chart.

A new member of the Filebeat:ELK stack, a lightweight open source log file data collector based on Logstash-Forwarder source code development, is an alternative to it. After installing Filebeat on the server that needs to collect log data, and specifying the log directory or log file, Filebeat can read the data and quickly send it to Logstash for analysis, or directly to Elasticsearch for centralized storage and analysis.

Introduction to the common architecture and usage scenarios of ELK

In this chapter, we will introduce several common architectures and usage scenarios.

The simplest architecture

In this architecture, there is only one instance of Logstash, Elasticsearch, and Kibana. Logstash obtains data from a variety of data sources (such as log files, standard input Stdin, etc.) through the input plug-in, then processes the data through the filter plug-in, and then outputs the data to Elasticsearch through the Elasticsearch output plug-in, which is displayed by Kibana. See figure 1 for details.

This architecture is very simple, and the usage scenarios are limited. Beginners can build this architecture to understand how ELK works.

Logstash as a log collector

This architecture is an extension of the above architecture, extending a Logstash data collection node to multiple machines, sending the parsed data to Elasticsearch server for storage, and finally querying in Kibana, generating log reports, and so on. See figure 2 for details.

This structure needs to deploy Logstash on each server, and it consumes CPU and memory resources, so it is more suitable for servers with rich computing resources, otherwise it is easy to cause server performance degradation, and may even lead to malfunctioning.

Beats as a log collector

This architecture introduces Beats as a log collector. Currently, there are four types of Beats:

Packetbeat (collect network traffic data)

Topbeat (collects data such as CPU and memory usage at the system, process, and file system levels)

Filebeat (collect file data)

Winlogbeat (collects Windows event log data).

Beats sends the collected data to Logstash, which is parsed and filtered by Logstash, then sent to Elasticsearch storage and presented to users by Kibana. See figure 3 for details.

This architecture solves the problem that Logstash takes up high system resources on each server node. Compared with Logstash,Beats, the CPU and memory of the system are almost negligible. In addition, SSL/TLS encrypted transmission is supported between Beats and Logstash, and two-way authentication between client and server ensures the security of communication.

Therefore, this architecture is suitable for scenarios where data security is high and the performance of each server is sensitive.

The architecture of introducing message queuing mechanism

When the author collates this article, Beats does not support output to the message queue, so there can only be Logstash instances at both ends of the message queue. This architecture uses Logstash to collect data from various data sources and then outputs it to the message queue via the message queue output plug-in. Currently, Logstash supports Kafka, Redis, RabbitMQ and other common message queues. Then Logstash gets the data from the queue through the message queue input plug-in, analyzes and filters it, sends it to Elasticsearch through the output plug-in, and finally displays it through Kibana. See figure 4 for details.

This architecture is suitable for situations where the log size is relatively large. However, because the Logstash log resolution node and Elasticsearch are heavily loaded, they can be configured in cluster mode to share the load. Message queue is introduced to balance the network transmission, thus reducing the possibility of network blocking, especially the possibility of data loss, but there is still the problem that Logstash takes up too much system resources.

Beats is directly imported into es as a log collector

Thank you for your reading, the above is the content of "the principle and Application of ELK Stack". After the study of this article, I believe you have a deeper understanding of the principle and application of ELK Stack, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.