Shulou(Shulou.com)06/03 Report--

As an operation and maintenance engineer, we need to troubleshoot the server every day, so the first thing that can help us locate the problem is to check the server log, which can quickly locate the problem. At present, we say logs mainly include system logs, application logs and security logs. System operators and developers can learn about server software and hardware information through logs, check errors in the configuration process and the causes of errors. Log analysis is often needed to understand server load, performance, security, and to take timely action to correct errors. And the logs are distributed across different devices.

If you manage hundreds of servers, we log into each machine the traditional way to look up logs. Does this feel cumbersome and inefficient? It is imperative that we use centralized log management, such as open source syslog, to aggregate log collection across all servers.

After centralized log management, log statistics and retrieval has become a more troublesome thing, generally we use find, grep, awk and wc Linux commands to achieve retrieval and statistics, but for higher requirements such as query, sorting and statistics and the huge number of machines still use this method inevitably a bit inadequate. The open source real-time log analysis ELK platform shared today can perfectly solve our above problems. ELK consists of three open source tools: ElasticSearch, Logstash and Kiabana.

1)Elasticsearch is an open source distributed search engine, its characteristics are: distributed, zero configuration, automatic discovery, index automatic fragmentation, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc., ELK official website: www.elastic.co/

2)Logstash is a completely open source tool that collects, filters, and stores your logs for later use (e.g., search).

3)Kibana is also an open source and free tool that Kibana provides for log analysis for Logstash and ElasticSearch.

4)Logstash and Elasticsearch are written in Java, while Kibana uses the node.js framework, and the ELK environment is configured to ensure that the system has JAVA JDK development libraries.

ELK schematic topology diagram

1)ELK Workflow

Collect the log data of client APP through logstash, filter out all logs, store them in Elasticsearch search engine, and then display them to users through Kibana GUI at WEB front end. Users need to be able to view specified log contents.

You can also join redis communication queues:

Figure 1.

Figure 2.

2) Workflow after joining Redis queue

Logstash contains Index and Agent (shipper), Agent is responsible for client monitoring and filtering logs, Index is responsible for collecting logs and handing them to ElasticSearch, ElasticSearch stores logs locally, establishes indexes and provides search, kibana can obtain desired log information from ES cluster.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.