Shulou(Shulou.com)05/31 Report--

What this article shares with you is about the network traffic analysis engine QNSM and its application. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article. Let's take a look at it.

Guide reading

The network boundaries of the infrastructure of Internet companies with a certain business scale usually show a certain degree of complexity and multi-partition, so how to carry out effective security protection and control will become the focus and difficulty of security system construction.

Complexity of Internet enterprise boundary

The above figure shows the network architecture of a typical medium and large Internet company, which is usually divided into:

The office network may also have many small branches in the lower right corner, and even have office computer rooms.

Core data centers, which may be distributed in many parts of the country and interconnected through dedicated lines, on this basis to build their own private cloud.

CDN network, these CDN nodes and the core computer room are also interconnected to some extent.

If you use public cloud infrastructure services to present a hybrid cloud model, these basic services are interconnected with the core data center.

A large number of these partitions may be connected to the Internet in different ways.

Finally, the emergence of BYOD and various wireless hotspots as well as mobile wireless hotspots has led to the emergence of a large number of fragmented so-called new boundaries.

For enterprise security defense, it presents a variety of new challenges:

Security defenses have become fragmented and multi-layered, and there may be more small boundaries within the larger boundaries.

The traffic of a single boundary may be very large, especially at the boundary of Internet interconnection, and more than 100Gps has become the norm. If traffic detection is carried out at these boundaries, it is required to have high performance and good horizontal scalability to cope with the expansion of boundary traffic at any time.

The internal and external threats faced by enterprises are still very serious, such as flow attacks, leaky attacks, boundary penetration, internal personnel leakage, destruction and control.

The system of security defense is also gradually evolving, from simple border defense, defense in depth combined with multi-level internal defense, and security models based on zero trust or hypothetical collapse of roles and permissions are all driving our security defense to constantly upgrade and iterate to deal with increasingly serious threats. The evolution of the security defense system does not mean that the boundary has disappeared. Border defense is still the foundation and an important link. Effective traffic analysis and control has become an important data source and joint arrangable control points of the new security defense system.

Therefore, we developed the bypass traffic analysis engine QNSM (iQiYi Network SecurityMonitoring). Through the collaborative analysis of business traffic and bypass traffic, and integrating a variety of schedulable protection capabilities, we coordinated to deal with various types and levels of attacks, and the security operation system opened, gradually forming a closed loop from access, preplan, response, linkage, defense, and traceability.

Introduction to QNSM

Full traffic analysis is very important and can be used for asset discovery, network monitoring and visualization. For security, through the analysis of network traffic, a baseline model of traffic is built, anomalies, risks and detection attacks can be found from traffic, data content can also be extracted from traffic, potentially sensitive data flow or disclosure can be found, and ACL policy proofreading can be carried out. And through machine learning and expert analysis of the data characteristics of network traffic output, we can mine more information and trace the source of forensics and events.

As shown in the figure above, QNSM runs as service software on an ordinary multi-core X86 server. Each server can have multiple 10 Gigabit network cards, and the network traffic to be analyzed can be obtained by splitting or switching port mirroring. if the traffic is relatively large, it can be further shunted through a shunt to spread the traffic to different servers for analysis. Furthermore, QNSM uses DPDK to implement multi-core processors and multi-queue network cards to achieve high-speed packet processing. Its high performance comes from:

Zero copy (Zero-copy)

Prefetch and batch receive packets to reduce cache miss and improve throughput

The design pattern of Share Nothing realizes lock-free and CPU-free switching.

Make full use of the RSS feature of the network card, packet receiving queue and CPU core binding

Basic library

QNSM encapsulates and builds various basic libraries on the basis of DPDK, which provides basic capabilities for building upper-level pipelining, including:

PORT: the logical encapsulation of the ring queue between the network card queue and the core is the basis for parallel processing and linear expansion.

MSG: encapsulates communication messages between CPU cores that support callbacks. These messages can be policy messages or data messages. This kind of non-blocking inter-core messages support one-to-one and one-to-many communication, so that the data is separated from the control plane, and the output of the data set is separated from the packet processing.

ACL: is a policy description of a quintuple that supports callbacks, such as aggregating, processing, dump, and so on, for packages that satisfy what kind of policies you want to specify.

TBL: is the encapsulation of DPDKrte_ hash table, is the storage of data set, provides CURD operation interface, and realizes the allocation of table item resources based on mempool.

SCHED: encapsulation of worker threads, supporting custom package processing, policy enforcement, custom computing logic, message distribution and scheduled callbacks, etc.

Pipeline

QNSM builds different pipeline components to meet the security applications of different scenarios. In order to support more security application scenarios, we can build more pipeline components on the basis of the basic library to achieve a variety of network traffic processing capabilities.

SESSM: responsible for packet parsing, flow data aggregation and replication and forwarding, processing policy messages, etc.

SIP_AGG: aggregates and outputs features to the source IP and opens them in response to a policy message during an attack.

VIP_AGG: self-learning of the target VIP (business IP to be protected), feature extraction and output to EDGE based on the target VIP.

DUMP: save the packet as a PCAP file for subsequent events to be traced back, and open it through a response policy message during an attack.

EDGE: responsible for outputting multidimensional data sent by upstream components to external Kafka for further analysis.

DETECT: integrated Suricata library to support IDPS detection.

Based on the existing pipelined components of QNSM, we have applied it to many scenarios such as DDoS attack detection, IDPS detection and protection, traffic monitoring and network DLP, and supported the development of various upper security products. You can design and insert self-developed components according to the needs of your different security applications.

Control layer

Master is the master of the whole engine. It receives the policy message from the management platform through Kafka and sends it to the pipeline component to realize the configuration and processing control of the pipeline.

Security application

The figure above shows how iqiyi uses QNSM to meet various security requirements. Iqiyi's QNSM service nodes are distributed on the boundaries of each network partition and are managed and maintained through the border control center. The Border Control Center (Aegis) is the core service of iqiyi's network security protection. It has the following functions:

Manage and configure all QNSM clusters, control and interact with Kafka and QNSM

Manage and configure QNSM integrated IDPS (Suricata) through IDPS gateway

As the unified service background of iqiyi WAF, there is no key introduction here.

Iqiyi's internal security big data analysis engine combines threat intelligence and other external data to analyze and process the data output from the QNSM cluster EDGE components to Kafka. The resulting network attacks will be sent to the border control center, which will further interface with the situation awareness system according to the policy, and then dock with the security operation system to achieve closed-loop operation.

The border control center can convert QNSM dump network traffic into PCAP files according to event-driven traffic, and ETL into Moloch (https://github.com/aol/molocha largescale, open source, indexed packet capture and search system) to establish packet index, which is convenient for packet-level analysis and source tracing of events.

We will introduce the architecture and design of the border control center in a follow-up sharing article, which I will not repeat here. The following is a brief description of how we use QNSM to meet our DDoS attack detection and extension support for IDPS capabilities.

DDoS attack detection

When the service is connected to the border control center, the VIP (Virtual IP) to be protected will be provided. in addition, the QNSM will actively discover the target VIP in the traffic and compare it with the CMDB to find the VIP to be protected. The main purpose of the DDoS attack detection method is to aggregate the characteristic data based on the traffic targeted by the protected VIP for the security big data analysis engine to judge to achieve DDoS detection. The overall detection idea is to build the target VIP and the traffic baseline of the computer room, calculate the traffic characteristics, and carry out multi-dimensional anomaly detection and identification attacks. The common traffic baselines include the component baselines of VIP and the computer room, as well as the traffic baseline and the upper boundary of the computer room. The current traffic and the baseline are calculated in real time to construct multi-dimensional index characteristics that represent the deviation between the current traffic and the baseline. The strong explanatory model is used to detect and judge (such as the scorecard model, etc.), and the positive and negative feedback of the border control center operation on the event will be used to further modify the baseline and training model.

After the traffic attack is identified by the previous multi-dimensional anomaly detection, the border control center will take the following actions immediately after receiving the attack and alarm:

The border control center sends a variety of policy messages management and guidance to Master components through Kafka, including dump packet forensics, attack source IP discovery, attack source port extraction, reflection attack protocol DFI and other policy messages.

The Master component wakes up the SIP_AGG component and the DUMP component according to the policy, wherein the SIP_AGG component aggregates characteristic data based on the source IP (which can be used to help find the source of subsequent attacks IP), the DUMP component carries out dump packets, and the PCAP files from the dump are delivered to Moloch for further indexing and expert analysis.

The VIP_AGG component aggregates the characteristic data based on VIP+SPORT, and the SESSM component also does protocol DFI identification for the attacked VIP to help identify whether there is a reflection attack of a certain kind of protocol.

Through the EDGE component, QNSM aggregates data into Kafka, which can be used as a data source for security big data analysis and interact with other different security services.

When the DDoS attack is over, the border control center sends a policy to the Master component through Kafka to turn off message management and guide the work of heavy components in the pipeline.

When an attack is detected on the ingress traffic of VIP, it is usually necessary to further judge whether the attack is a reflection attack. We will use the VIP+SPORT aggregate characteristic data produced by QNSM and the DFI protocol of SESSM components to identify the characteristic data, and then calculate the traffic proportion distribution and packet proportion distribution of different source ports in the security big data engine, and then calculate the entropy value. The smaller the entropy value, the higher the risk (the higher the proportion). The higher the risk, if the traffic and packet share are all concentrated in one port, the entropy will be 0). We will build a scorecard model based on multi-dimensional characteristics, and finally determine whether it is a reflection attack of a certain kind of protocol.

After the attack is determined, traffic traction will be carried out according to the emergency plan. Iqiyi built a private traffic cleaning center, and combined cloud cleaning and near-source cleaning of operators to form a trinity cleaning capability.

IDPS capability integration

Suricata is an IDPS engine based on network traffic. It has an extensive set of rules to monitor network traffic and triggers alerts when intrusions occur. QNSM integrates Suricata by using library files, and updates Suricata-related detection rules through the IDPS gateway receiving border control center. Packets copied and forwarded from SESSM components will be processed by calling Suricata through Detect components to achieve real-time detection and trigger events and alarms. And output the events and alarms to the security big data analysis engine for further analysis and processing through Kafka. Through the integration of Suricata, QNSM can be compatible with a large number of open source and custom IDPS rule sets, and the rules are managed in exactly the same way.

With Suricata's DFI capability, QNSM also rapidly extends the ability to identify various types of database, cache and other cloud service access traffic from traffic, and to extract file information (including file name, file size, file type, MD5, etc.) from traffic and output it to security big data analysis engine through Kafka, finally outputting data leakage and illegal access events to the DLP platform (Green Shield).

At present, it supports the identification of many protocols and related tools, such as HTTP, MySQL,Redis,CouchBase, Memcached,MongoDB,Elasticsearch,Kafka,VNC,RSYNC, which are common in iqiyi, and supports the monitoring of HTTP,SMTP,FTP file transfer channels.

Open source

At present, QNSM has been applied to iqiyi, including DDOS attack detection, IDPS, network DLP and other security detection scenarios. 22 + clusters have been deployed totaling 130 + analysis nodes, and the total analysis bandwidth capacity has reached 1TBps.

We need you to work with us to improve QNSM to make it more powerful, welcome more cooperation and contributions, and cover more or even not limited to security application scenarios.

This is what the network traffic analysis engine QNSM and its application are like. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.