Shulou(Shulou.com)06/01 Report--

Network shunt-Rong Teng network-DPI detection is a popular front-end mode of network monitoring, and the importance of network shunt for network security can be said to be important in the end! Today, let's talk about DPI testing.

Deep packet Inspection (DPI)

Deep packet inspection (Deep packet inspection, abbreviated as DPI) is a special network technology. Generally, network devices will only view the Ethernet header and IP header, but will not analyze the contents of the TCP/UDP. This is called shallow packet inspection; the corresponding DPI will check the contents of the TCP/UDP, so it is called deep packet inspection.

DPI is generally a piece of hardware or software, which is generally connected to the network by means of "side-by-side". It examines every packet in the network, identifies the application layer protocol, and takes certain measures according to the identified protocol (such as recording HTTP access behavior). For the TCP protocol, it can identify the complete TCP interaction process (for example, a HTTP request will have multiple TCP packets sent from the request to the response).

Rong Teng network mobile Internet collector supports 160 10G and 20 100G

NDPI

NDPI is a DPI library written in C language, which is used to implement software DPI system. It extends from OpenDPI, and the architecture and implementation of the two are basically similar.

After compilation and installation, it generates / usr/local/lib/libndpi. (aMaginso) library files (a static library file, so dynamic link library); / usr/local/include/ installs the relevant header files.

Personally, I like to use static library files, which merge all the binaries into one executable file and run without having to install a lot of libraries. In addition, I don't like to put things under / usr/local/lib, so the code I provide is compiled with a "all in one" through cmake. Github is version 1.6. If you want to update the code, just replace the src folder.

How to use nDPI

NDPI's code is poorly written, and there is no architecture that is just a mess (even a little writing is better than this). But at least it works and is the only "open" DPI library, so whatever the reason you choose it, you have to put up with its "ugliness".

NDPI has almost no documentation, except for an example of "ndpiReader", which is written like a stream of water (don't complain). This is the reason why I am writing this article. I hope I can give some help to the students who use nDPI.

One of the most important data structures of nDPI is ndpi_detection_module_struct_t, which is constructed through ndpi_init_detection_module.

Network shunt

The first parameter is used to calculate the various timeouts of the nDPI analysis protocol, which is generally accurate to 1000 milliseconds (the nDPI protocol analysis part and the "global part" are very tightly coupled, and this data is actually only needed by the "protocol analysis module").

The second and third parameters are the encapsulated "memory allocation" function; the memory management of nDPI is very messy, and in some places we apply for memory ourselves and nDPI releases it for us. Therefore, nDPI must not directly use applications such as malloc or free to release memory, but leave it to the programmer to provide the functions.

The third parameter is the debug function. If NDPI_ENABLE_DEBUG_MESSAGES is defined, then nDPI will call this function to output some debugging information.

All nDPI API is this kind of "ghost animal" style, almost all kinds of entanglement. Fortunately, we only need to use very little API to complete the task.

Configure the protocol analysis module

NDPI supports a variety of protocols, all in the protocols folder. When compiled, all protocols are placed in the nDPI library. When using it, we can set which analysis modules we need to open.

NDPI_PROTOCOL_BITMASK defines the "bitmap" of the open protocol, and the supported protocol can be added through the NDPI_BITMASK_ADD function. Finally, ndpi_set_protocol_detection_bitmask2 is called to configure the bitmap.

The first argument to the ndpi_set_protocol_detection_bitmask2 function is ndpi_detection_module_struct_t (the data structure we initialized above); the second argument is the bitmap flag.

Special note: the more protocols open, the slower the recognition speed; when nDPI identifies the protocol, it is a serial structure, no matter whether it is successful or not, it will traverse the configured protocol seriously.

Sub-agreement

A subprotocol is a breakdown of a protocol, for example, if we want to analyze all "Google" HTTP requests, the first step is to analyze the "HTTP" request, and the second step is to determine that the HOST contains google.com. The second step here is the "subagreement".

NDPI's only "QuickStartGuide" explains this further, and subprotocol identification is provided to nDPI in the form of a configuration file. such as

It also supports port mode (81 and 8181 of TCP are directly marked as HTTP no longer does content detection)

Network shunt

NDPI the implementation here uses a very famous algorithm-Aho-Corasick. Take the first picture as an example, there are two rules "Google" and "Veneer". We have a string (HOST). How can we tell that this string conforms to that rule? The easiest way is to loop through all the rules, which can be very slow if there are many rule entries. Aho-Corasick is such an algorithm that can complete all matching tasks in O (n).

Load the subprotocol through the ndpi_load_protocols_file function.

Begin to identify

The API that identifies the protocol is very simple-- the ndpi_detection_process_packet function. It is this tricky function, and the degree of metamorphosis can almost be described as outrageous.

Global structure of ndpi_struct

Flow is special. We'll talk about it later.

Packet's pointer to the IP header

Packetlen packet size

Current_tick_l current time (accurate to milliseconds) is used to determine "expired TCP requests"

Src,dst is of no use, and the documentation says it has nothing to do with state machines. The only purpose is to update the configuration of the Analysis Protocol. It is generally set to NULL.

TCP protocol is a flow protocol, which is a "request-> response" structure starting from a three-way handshake. DPI can trace one or more of these packets, or all of them (I'll cross-use TCP session, session, and flow later, and the three nouns are actually the same).

Instead of recording a complete TCP packet internally, nDPI uses a very vaguely defined ndpi_flow_struct type to represent a TCP session (this data structure also contains part of the "protocol analysis" data, so the definition is very vague). To facilitate the analysis of complete TCP requests, we define our own data structure, dpi_flow_t,ndpi_flow_struct, as a member of it. Use pseudo code to represent the analysis process:

If it falls on the code, it is the get_ndpi_flow function; in implementation, we will sort the target and source port and then do hash;. This is because the packet is "communicating with each other", so the sender and receiver are relatively speaking, otherwise the data identified may be "one side".

Generally, we use a binary tree to store all the TCP sessions being analyzed. NDPI transplants a set of functions in FreeBSD, such as ndpi_tfind, ndpi_tsearch, ndpi_twalk, ndpi_tdelete, etc., to implement common data structure operations.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.