Shulou(Shulou.com)11/24 Report--

Security log records the entire execution process of enterprise servers, cloud infrastructure, applications, etc. Retrospective analysis of log data can accurately and clearly understand the status of enterprise IT facilities, troubleshoot security risks, retrieve the source of failure, and so on.

With the growth of enterprise scale, the deepening of digitization and the increase of security devices, the number of security logs increases exponentially. When dealing with today's PB-level security logs, either the enterprise's current log processing platform needs to pay very high authorization fees and professional labor costs, or performance bottlenecks can not support PB-level mass log processing.

How to adopt a new architecture that can support enterprises to achieve second-level query in PB-level data while the cost is controllable?

On September 20, Tencent safely released a new generation of cloud native security data lake, focusing on massive log data analysis, helping enterprises to build an integrated cloud native data lake platform and move towards proactive security. It is reported that under the same data scale, the hardware cost of this product is only 1 / 10 of that of similar open source software. in addition, the query performance, especially the aggregate query performance, has been doubled, and the second-level query of PB-level logs can be realized.

"at present, enterprises' log processing platforms are usually quickly built using components such as open source big data ELK, mainly to collect alarm data, and analyze incomplete data. Moreover, it takes minutes or even hours to perform long-cycle data queries, while network security is confronted in real time, so any data analysis products based on 'hours' are definitely not applicable." Yang Junyu, a senior researcher at Tencent Security big data Lab, said.

Two years ago, in the process of serving customers, Tencent Security found that customers generally encountered the problems of rising log storage costs and low query efficiency, so Tencent Security big data Lab was based on big data's analytical and processing capabilities for many years. It took two years to independently develop a secure data lake product for cloud native.

Tencent Cloud Native Security data Lake is a self-developed data analysis platform based on cloud native. It takes advantage of log data without modification, repetition of a large number of fields, timestamp and other features to make several major innovations:

Leading ● architecture: MPP architecture, developed in Rust language, specially optimized for log and security scenarios

● extreme cost reduction: using column storage to achieve the ultimate compression ratio, indexing-free architecture to avoid indexing overhead

● integration engine: integration of data processing, query, storage and analysis through atomic capabilities

● plug-in expansion: supports different analysis scenarios and "plug-in" expansion through SQL / SPL statements

● easy operation and maintenance: for cloud native architecture, the separation of storage and calculation, the separation of reading and writing, the elastic expansion of one-click capacity and the switching of failure seconds are realized.

Relying on the above technological innovation, Tencent Cloud's native secure data lake has achieved the ultimate compression ratio and data processing efficiency, which can reduce the enterprise's secure operation and storage costs by 90%. In the underlying architecture, it is oriented to cloud native design and supports multi-instances and multi-users. flexible expansion can be achieved according to the actual needs of the enterprise.

In addition, Tencent Cloud's native secure data lake supports pan-secure data access, processing, storage, analysis, alarm, visualization and other services, as well as plug-in application development capabilities. Enterprise users can customize upper applications according to their needs, and build a complete log application ecosystem through platform + App + partners to fully enable all kinds of security scenarios.

At present, the data lake has been integrated into Tencent Security SOC + products to provide a base for enterprise security operation and management. In the future, Tencent Security will also provide independent products to help enterprises build a cloud native security data lake platform.

Facing the intelligent era, safety operation and management is the security immune central system of enterprises, and the intelligent analysis ability of security big data will become the basis for enterprises to move towards intelligent security. Tencent Security's security operation product matrix always revolves around one thing, that is, how to create value for customers. In the future, Tencent Security will continue to open its technological atomic capabilities, integrate Tencent's leading technologies into the enterprise's existing security capabilities, and inject digital security immunity into the security practice of Qianhang Baiye.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.