Shulou(Shulou.com)06/01 Report--

This article is reproduced from the official account "Sina Security Center" of Wechat, original author: candy LUA

Summary

Threat analysis is now part of day-to-day work. Big data tool based on ELK has become a popular choice for log analysis. Open source is easy to deploy and provides a good user experience for log retrieval and aggregation. The candy lab has previously introduced the overall log processing scheme based on Graylog, a kind of ELK tool. After the summary of the experience after the production practice, the advantages and disadvantages of this log processing scheme are found. As the follow-up system continues to access new security log data, the log analysis requirements of more schools, the system will become more and more complex. For complex policy queries, sometimes we provide log data based on ES and REST API. When dealing with complex query development, the development efficiency slows down with the increase of scale. We need a higher level of abstraction, a business language directly related to business data, and operation instructions similar to SQL or DSL to reduce the landing cost of security policy implementation.

Review of ELK Model

In order to better illustrate the problem from the upper deployment, we use a cartoon way to describe the system structure, which does not involve more detailed nodes such as load balancing and line guarantee. Let's first recall a log analysis scheme based on a similar Graylog. The log data is encapsulated and abstracted into the concept of Stream flow, and the log is logically abstracted to a higher level by referring to the Pipeline pipeline, so that we do not directly face the concepts of files and indexes, but with the schema design of Stream, Input, Output, Pipeline, we can better classify the original log data and get close to the business.

From log collection, to data formatting, to ES storage, to REST API data query, to automatic query, to data visualization is our general use of routines, we have been working on this idea for a period of time, more articles can refer to the confectionery laboratory before the article, here is a high summary of the structure of this system.

The weapon of the fighting nation ClickHouse

Let's introduce the data acquisition and analysis scheme based on ClickHouse again. In fact, from the data collection, processing, query, display, most of the user experience is also somewhat similar, the difference is that ClickHouse provides SQL query. Graylog itself also contains components such as MongoDB and Kafka, while ClickHouse is not an integrated solution. A relatively simplified introduction. From the point of view of the system deployment composition, it is very similar.

Differences between schemes

ClickHouse and ES are two different data retrieval engines. ClickHouse provides SQL-based query function, ClickHouse support for SQL and performance, we will give the relevant data at a later stage. A holistic solution like Graylog provides its own data query DSL, but this DSL is a system independent of Graylog itself, and SQL is more versatile. ES also supports ES SQL, but it depends on who uses it. The core difference between the two schemes lies in the different data retrieval schemes of ES and ClickHouse, and the security business will produce different security audit requirements for different data. Data collection and data presentation are similar, of course, ES also has ES SQL, but this is not the difference between SQL, but the difference between the two ecology and design.

We can use the ClickHouse table in the same way as using MySQL. The monitored server pushes its own data to the Kafka through a specific tool, ClickHouse gets the pushed data, and then stores the data into the table of two-dimensional data structure, and then we can use SQL language names to achieve automatic log security audit. We have already used Graylog as a kind of ELK service in production, and the design based on REST API is very convenient for front-end and mobile audit applications to expand. Based on threat data analysis, we experiment with a new solution based on ClickHouse, which focuses on complex data retrieval and business data collision. With the high abstract implementation of SQL, the pure code is less dependent on DSL operations, the code is written less, and the security policies are translated into SQL statements, but the underlying engine is different at the same time.

Commonality between schemes

For users, the general idea of these two schemes also connects log with "flow" and "pipeline", logical log data flow direction, no matter what kind of tools and storage, log data aggregation mode is similar, but in protocol, whether to use syslog protocol or JSON protocol, or both support, based on the point of view of data aggregation, the two schemes can achieve the goal. But for the implementation of security policy, that scheme is faster and more convenient, and we will have new experimental memory and data implementation in the future. The biggest commonality is that the pattern in which data is collected to external data is similar.

The above diagram greatly simplifies the physical deployment of services in actual production, using a single point instead of a cluster. In the end, the flow direction of the log data can be seen relatively clearly. From the data generated by the visitor when requesting the service, to the data is pushed to the Kafka queue, and then the Kafka consumer consumption data is stored in ClickHouse, then the Openresty-based API gateway is provided, and then it is provided to API users.

The API gateway based on Graylog and ELK is based on ES data retrieval, and the gateway will convert the security policy into query, while the API gateway based on ClickHouse uses the security policy based on SQL query based on ClickHouse to execute on the ground. When we design the system, we make the security policy and the system independent, or the general security policy does not consider whether the solution is ELK or ClickHouse, as long as it is a security analysis policy, it can be parsed and executed in a script or a language similar to DSL.

Summary

ClickHouse is a product of the fighting nation. CloudFlare has been used in production analysis and will continue to explore new trends and practices of these products. Combine traffic analysis and log analysis statistics to analyze threats and find threats. Some system forms are means, the system can achieve the strategy of security personnel and effectively solve security problems, which is the goal to be achieved in practice. We can develop a more high-level abstract DSL based on ClickHouse to describe the security policies of security personnel and interact with other systems to complete threat analysis and protection. Some related designs and tools and code will be introduced later.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.