Shulou(Shulou.com)06/03 Report--

What this article shares with you is the method of improving the ELK log system into ELFK. I believe that most people have not yet learned this skill. In order to let you learn, I have summarized the following content for you. Without saying much, let's read on.

One: the initial stage of ELK log system

When we first came to the company, our company's log collection system ELK often failed to query the latest logs. Later, we found that the nodes of ES were often yellow or red. Sometimes I receive complaints from developers. This ELK system was also built by another colleague.

The architecture diagram is as follows:

ElasticSearch is a cluster of three servers.

The version of ElasticSearch is 6.2.x, and Logstash runs on each server. All kinds of logs are collected by Logstash, processed by plug-ins such as Grok,Geoip, and then sent to the ElasticSearch cluster.

Kibana does a graphical presentation.

Our previous elk architecture is relatively simple, but there are some problems:

1. Logstash relies on Java virtual machines to occupy a large amount of memory and CPU of the system.

2. When there is a large amount of data, Logstash can easily cause other business applications to crash and affect the normal use of the business.

3. With the accumulation of time, es space can not meet the status quo.

4. Kibana has no security control mechanism, no permission audit, and poor security.

5. ElasticSearch master node is also a data node, which leads to slow query sometimes.

Second, Filebeat is introduced into the improvement of ELK log system.

For the version of ElasticSearch, we chose the original version 6.2.x, and then rebuilt a set of ELK log system.

The version of ElasticSearch 6.x must rely on X-Pack if it is to be used for authentication, but X-pack is a paid product, so we look for cracking patches on the Internet, and then crack ElasticSearch 6.x.

The architecture diagram is as follows:

The specific improvements to the entire architecture are as follows:

The main contents are as follows: 1. The client chooses the lighter Filebeat,Filebeat to be written in Golang language, which has the advantages of small temporary system resources and high collection efficiency.

2. After Filebeat data collection, it is sent to multiple Logstatsh for unified filtering, and then the filtered data is written to the ElasticSearch cluster.

3. Increase the original 3 es nodes to 6 nodes, of which 3 ES nodes are master nodes and the rest are data nodes. If the disk is not enough, the data node can be scaled out.

4. X-pack is introduced to realize authority control at Index level to ensure data security.

5. The hard disk of ElasticSearch cluster adopts the hard disk of SSD.

At this point, our log system is temporarily normal and can meet the needs of log checking, the phenomenon of stutter is rare, and the resource utilization of the server has been directly reduced by half.

However, it will be slow to check the data from a few months ago, so we have made the following optimizations on the basis of the above:

6. ElasticSearch separates hot and cold data.

7. The index data of 60 days ago is turned off and opened manually when needed.

8. The version of ElasticSearch adopts the version of ElasticSearch 7.x, and user authentication is implemented by its free basic authentication (because the new version of 7.x is optimized in performance, query and write speed will be faster)

3: ELFK with improved ELK log system

Because the development language of our main business is PHP,PHP, there are not many logs, but PHP is an interpretive language after all, and the running efficiency is not high, but our company's business concurrency is very high. The concurrency is at least 100000 or more. Some businesses are Java, such as location reporting, and microservices are also developed by the company itself. The framework may not be perfect and it is not as mature as Spring Boot. There are so many logs that a Java microservice generates only a few terabytes of data every day. Some microservices still log at the info level.

With the accumulation of time, the number of logs is hundreds of T and there are PB-level logs.

At the same time, big data's department is also looking up the interface of the ElasticSearch cluster, resulting in a particularly great pressure on ElasticSearch. As a result, it is sometimes slow to query the history log.

The current Filbeat + Logstash+ ElasticSearch+ Kibana architecture can no longer meet the requirements. So we thought of using MQ for buffering, message queue for buffering, which product should be chosen, and several software considered by message middleware, such as Redis,Rabitmq,ActiveMq,Kafka, etc., for these considerations we did not hesitate to choose Kafka, because the throughput of Kafak is higher than others, and the performance of Kafka is much better than ActiveMQ, RabbitMQ and so on.

The architecture diagram is as follows:

The specific improvements to the entire architecture are as follows:

1. Filebeat data is collected and stored in kafka, and then consumed one by one with Logstash, and written to es to ensure the integrity of the data.

2. Logstash runs multiple nodes, multiple processes and multiple threads for consumption.

3. Kafka multi-Topic multi-partition storage to ensure throughput.

4. Big data's department directly checked the interface of ElasticSearch cluster at the beginning, and changed it to consume Kafka data directly, so that the pressure on ElasticSearch was greatly reduced.

At this point, the current architecture has met the enterprise's PB-level log requirements, check the history log is not stuck, but also can meet the daily needs.

When we use Kafka-Manager to monitor and manage the status information of Kafka, we find that at the peak of business, the topic of Kafka has a very small accumulation.

But it does not affect the development and operation and maintenance to check the log. So I, who love to toss about, decided to write my own program by hand instead of Logstash consumption, so I had the following content.

Fourth: Filbeat+Go+ElasticSearch+Kibana log collection system architecture

If you write your own program instead of Logstash consumption, the languages you are familiar with are Python and Golang, so you decide to write in one of these two languages, considering that Python is an interpretive language and has global locking restrictions. Golang, on the other hand, is a compiled language and naturally supports collaborative programs. Concurrency is supported. So Golang is used for kafka consumption.

The architecture diagram is as follows:

The specific operation methods of the entire architecture are as follows:

1. Establish different topic for different log types

2. Filebat uses different tag to collect data to different topic.

3. Golang enables cooperation to consume different topic and send it to ElasticSearch cluster.

At this point, after we use Kafak-Manager to check the status information of Kafka, there will be no small accumulation of messages even during peak periods.

This is the end of the way to improve the ELK log system into ELFK. I hope the above content can be helpful to you and learn more knowledge. If you like this article, you might as well share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.