Shulou(Shulou.com)05/31 Report--

Tide cyberspace exploration platform construction ideas, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Tide introduction

Tide (Tide) is a cyberspace asset search engine independently developed and launched by the Fashion Information Tide security team. Threat monitoring and security situation analysis of Internet infrastructure through uninterrupted service detection and fingerprint identification. Combined with network security technology and big data analysis and processing technology, Tide is more like a nautical chart in Internet space.

Tide (Tide) currently implements functions such as cyberspace asset detection, fingerprint collection, fingerprint retrieval, vulnerability detection, vulnerability lifecycle management, poc directional detection, dark chain detection, horse detection, sensitive word detection, DNS monitoring, website usability monitoring, vulnerability database management, security early warning, report export and so on.

Here, I would like to share with you some methods and skills of Tide in spatial surveying and mapping. Although it is not comparable with zoomeye and fofa, the overall cost of the platform is relatively low, which is more suitable for small team development and operation.

Statement

Construction background

With more and more work with customers, sometimes it is often necessary to model and arrange the hosts or application systems within a certain range of customers.

For example, we are located in Shandong, while some customers in Shandong Province have more Internet assets and are not densely distributed, so it takes a lot of time and energy to scan some ports and services every time. If you want to quickly obtain the asset information of a specified region, you can only obtain it from zoomeye, shodan, fofa and other similar platforms, but these platforms generally have certain restrictions, such as the number of queries, such as the number of entries, for example, the industrial control port cannot view ip, and the accuracy may be slightly lower in a particular region, so I came up with the idea of building an asset detection platform by imitating the eyes of the clock.

At first, I just wanted to imitate a zoomeye, which could detect a certain range of space assets, but then I added a lot of functional modules on and off, and now I have the Tide. At present, it can basically meet the collection and detection of assets in a certain area, and some data can be quickly produced in the cooperation of some network security, and some other functional modules are also gradually improving, and open source will be considered after general improvement.

Main functions and implementation methods

Before drawing a frame diagram, most of it has been realized, and some are still working hard.

Asset discovery

At present, it is mainly to discover and detect the regional assets required by customers.

1. Due to the limited manpower, material and financial resources, only about a dozen detection nodes (still 512m memory) are deployed for continuous detection. Detection can be divided into two modes: fuzzy detection and accurate detection. Fuzzy detection is about 500W per day, accurate detection is about 20W host, 30,000 web applications.

Fuzzy detection mainly uses the combination of masscan and zmap to find open hosts and ports. Precise detection uses some scripts written by itself to find the availability of test ports one by one, and then uses nmap to obtain fingerprint information. We are also writing some detection scripts for some industrial control protocols and IoT protocols and adding them to fingerprint identification. Later, I will slowly write some tips on port detection and fingerprint identification.

In the early stage of the platform, we only used nmap to scan and get fingerprints, but later found that the efficiency was too low and unbearable, and nmap often had memory errors when getting fingerprints, which led to the situation that it could not be scanned, so later we used a combination of masscan and zmap, and now we are trying to use some new methods such as zgrab to improve efficiency and accuracy.

Ten scanning nodes are managed by xshell:

2. Half of the more than ten detection nodes are used to detect new tasks, and new tasks can be submitted by the foreground, and random ip will be detected when there are no tasks in the foreground. The other half of the nodes are used to detect old assets by regression, which basically ensures that the data in the system can be relatively new. It is a pity that at present, only the latest data is saved, and the historical data is not saved.

The task scheduling of the scanning node takes some effort, at present, it is scored according to the tasks received by each scanning node and the time spent, so that some high priority tasks can be scheduled to perform tasks with high scoring scanning nodes. However, at present, they are all in the same local area network environment, so the score is not very different. After testing, there is indeed some difference in the score between 512 memory and 1G memory.

3. The detection accuracy and speed of a wide range of ip addresses need to be improved. According to the current detection efficiency, the global Internet accurate detection is estimated to be too far behind zoomeye or fofa in two years. Left tears without skill.

Fingerprint identification

1. Identify the operating system version, the open port, the service provided and the service version.

This part is mainly realized by nmap. At present, we are also studying some principles of fingerprint identification, and we intend to achieve a detection engine on our own in the future. The biggest problem with nmap is its low efficiency. Sometimes configuring some parameters (such as-- min-hostgroup and-- min-parallelism) can improve the speed, but the false alarm rate will increase. Therefore, when scanning, the node will automatically select some parameters according to the network situation.

More commonly used parameters:

Nmap-sS-sV-T4-O-script=banner-open-min-hostgroup 1024-min-parallelism 1024-host-timeout 30m-p 80

The scanning results are extracted and analyzed and imported into the database.

I didn't use the pynmap library. I used some time to find that I often reported errors, which seemed to have something to do with the memory optimization of nmap. So then directly call nmap, generate .xml and then parse.

2. Identify the server language, Web development framework, Web application CMS, front-end library, third-party components and so on.

The fingerprint of web application is a little more complex. I have implemented several small tools to detect http head, waf, cms and so on, drawing lessons from some ideas of whatweb and wafw00f.

3. Obtain all the sub-domain names under the domain name by means of interface query and dictionary enumeration, and map out the overall structure of the target network as needed.

At present, sub-domain name enumeration is carried out for each web, so that more web applications can be found. Sub-domain name enumeration draws lessons from the idea of sub-domain name enumeration of lijiejie and Pig Man, and uses Google, Baidu sub-domain name retrieval and other methods, and predicts pan-resolution.

Scan management

1. Vulnerability scanning mainly integrates wvs, w3af and some small platforms developed by ourselves, collates vulnerabilities and exports detailed reports.

Because the results of wvs are in English, and the degree of friendliness is not high if you need to provide reports to customers, so the scanning results, reinforcement suggestions and vulnerability descriptions of wvs are translated into Chinese.

The Sinicization of wvs is mainly crawling the official vulnerability library of wvs, manually translating the more common vulnerabilities, and then using Google Translation to translate and manually check other vulnerabilities, and finally contains about 670 vulnerabilities.

The Chinese database can be obtained from this project, https://github.com/TideSec/WDScanner, in the vul_ CN table of the database file.

2. The application system can be scanned automatically according to the configured address, and the relevant functions such as timing scanning, scanning cycle, scanning range and so on can be set to meet the diversified needs of users.

3. The concept of loopholes and the whole life cycle of vulnerabilities can be recorded, including the generation, discovery, disclosure, management and demise of vulnerabilities. It also introduces in detail the causes, forms, discovery methods and specific countermeasures of vulnerabilities in each stage of the life cycle.

POC detection

1. POC detection uses the mature POC framework POCSUITE, modifies some interfaces, can be perfectly docked with Tide, and rewrites many plug-ins to make detection more automatic.

POC detection should be the most useful function, you can retrieve a port in a certain area, and then import the results into the poc detection task for fast and batch detection.

POC management

2. Detect common vulnerabilities such as plug-in vulnerabilities and middleware vulnerabilities, such as weblogic deserialization, Struts2 and other vulnerabilities. When a new vulnerability breaks out, users can use this feature for targeted detection.

POC test results

Safety monitoring

The main contents are as follows: 1. Continuously monitor the key customers in the sensitive information, detect the sensitive information of the website by means of full-text retrieval, application catalogue enumeration and picture file inspection, and export the sensitive information found, so as to assist users to supervise the website effectively.

Website compliance monitoring

2. It can monitor the website 24 hours a day, simulate visitors' request to visit, obtain the response status, response time and response results in real time, and find the abnormal situation of the website in time.

Website usability monitoring

3. When adding, deleting, modifying, hanging horses and other security events occur on the website, take real-time snapshots and keep them, users can quickly find the security event records for tracing.

Other introductions at the front desk

Front desk main interface

Query the main interface, the display of various protocols and ports, the brief introduction of the platform and so on.

Data statistics

Carry on the summary display to the national scope or a certain area data, carry on the chart display to each port and protocol.

Query syntax

Provide query syntax, query interface and so on.

Asset display

Draw lessons from the display format of zoomeye and shodan.

Detailed data

Asset details: url, ip, title, port, port service, service version number, cms information, operating system information, web middleware information, waf information, xpb information, ip address information, etc.

back-stage management

Administrative console

It is still improving, adding more chart data and striving for taller and bigger ones.

Threat intelligence

Main functions:

It is said that threat intelligence is actually a pile of leak database and security monitoring data, mainly collecting and crawling each public leak database and sorting it out, monitoring dark chains and sensitive words for key customers normally, and adding the monitoring of fangong hackers, which is a very fancy function.

Leak library

Security perception

Overseas intelligence

Fingerprint retrieval

Host fingerprint management

Host fingerprint details

Web fingerprint management

Web fingerprint details

Comprehensive retrieval

The comprehensive retrieval can query the fingerprints in the system in more detail, which is more powerful than the foreground retrieval, and the retrieval results can be derived in batches.

ToDo

Make your own front end!

Improve the strategy to improve the efficiency and accuracy of space exploration

Add more fingerprint libraries and identify more assets

Improve the whole life cycle management of vulnerabilities

Reduce the false alarm rate of sensitive words, dark chain and webshell.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.