Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to use ElasticSearch, Logstash, Kibana to build a real-time log analysis platform". In daily operation, I believe many people have doubts about how to use ElasticSearch, Logstash and Kibana to build a real-time log analysis platform. I have consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use ElasticSearch, Logstash and Kibana to build a real-time log analysis platform". Next, please follow the editor to study!

Introduction to ELK platform

Logs mainly include system logs, application logs and security logs. Through the log, the system operation and developers can know the software and hardware information of the server, check the errors in the configuration process and the causes of the errors. Regular analysis of logs can understand the load, performance and security of the server, so as to take timely measures to correct errors.

Usually, logs are scattered and stored on different devices. If you manage dozens or hundreds of servers, you are still checking logs using the traditional method of logging in to each machine in turn. Does this feel tedious and inefficient? As a top priority, we use centralized log management, such as open source syslog, to collect and summarize logs on all servers.

After centralizing the management of logs, log statistics and retrieval has become a more troublesome thing. Generally, we can use Linux commands such as grep, awk and wc to achieve retrieval and statistics. However, it is hard to avoid using this method for higher query, sorting and statistics requirements and a large number of machines.

The open source real-time log analysis ELK platform can perfectly solve the above problems. ELK is composed of three open source tools: ElasticSearch, Logstash and Kiabana. Official website: https://www.elastic.co/products

Elasticsearch is an open source distributed search engine, its characteristics are: distributed, zero configuration, automatic discovery, index automatic slicing, index copy mechanism, restful style interface, multiple data sources, automatic search load and so on.

Logstash is a completely open source tool that collects, filters, and stores your logs for later use (e.g., search).

Kibana is also an open source and free tool, and Kibana provides a friendly Web interface for log analysis for Logstash and ElasticSearch to help you aggregate, analyze, and search important data logs.

As shown in the figure: Logstash collects the Log generated by AppServer and stores it in the ElasticSearch cluster, while Kibana queries the data from the ES cluster to generate charts and returns them to Browser.

Build Elasticsearch installation on ELK platform

Click to download

Decompress zip or tar package, etc.

In bin/elasticsearch (the windows version is bin\ elasticsearch.bat)

Request http://localhost:9200/ to see the return value with version.

Logstash installation

Click download

Decompress zip or tar package, etc.

Create a logstash.conf file in the bin directory as follows

Input {tcp {port = > 4567 / / Port type = > "logs"}} filter {} output {stdout {codec = > rubydebug} elasticsearch {hosts = > ["localhost:9200"] / / ip port of Elasticsearch}}

Bin/logstash-f logstash.conf is fine.

Kibana installation

Click download

Decompress zip or tar package, etc.

Vi config/kibana.yml file modifies the ip port of elasticsearch.url = Elasticsearch, such as the ip port of Elasticsearch

Execute bin/kibana (windows execute bin\ kibana.bat)

Browser opens http://localhost:5601

Verification

Logback.xml configuration

Test class

Logstash log

Kibana display

At this point, the study on "how to use ElasticSearch, Logstash, Kibana to build a real-time log analysis platform" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.