Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to analyze the source code of InsectsAwke automation tools. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Brief introduction

InsectsAwke is a well-known open source vulnerability scanning system based on Python. Enlightened by Bugscan, born in Pocsuite. FLASK+MongoDB is used to provide WEB service. It has the robustness of Por.

Main function

Vulnerability scanning

Scan by calling Chuangyu's Pocsuite, and scan plug-in through Seebug

The scanning target can only be a single IP or URL, and does not support network segment scanning (the company is a small and medium-sized company, so forget to write this requirement). By default, there are more than 80 plug-ins, most of which are Seebug's free PoC.

Task cycles can be temporary, daily, weekly, or monthly

assets management

Vulnerability scanning tasks can be created through each asset library, and the asset library can only be a single IP or URL.

When the port discovery feature is enabled, the backend will call nmap regularly to scan the port of the asset. The port to be scanned can be configured in the settings.

Domain name discovery function

That is, the subdomain name burst function, but the current function is not perfect. You can only use the configuration dictionary to guess violently. The domain name dictionary can be configured at the setting. A subdomain name dictionary is provided in the project tests folder (the wydomain project of the dictionary source ring04h).

Installation process

Install virtual machine 16.04

Update to domestic source

Get the project source code

Git clone https://github.com/jeffzh4ng/InsectsAwake.git

Install Python and pip

Sudo apt updatesudo apt install python python-pip

Install MongoDB

The enterprise version of MongoDB is installed, and ordinary MongoDB cannot support the operation of the system.

Many people install freebuf unsuccessfully because they load the installation source of version 3.4, but when we execute sudo apt-get install-y mongodb-enterp, if we don't specify the version number, apt will install the latest version 3.6 by default, so there will be problems with mongodb-enterprise installation. Here we are loading the 3.6 installation source, so we can install it successfully.

Sudo apt-key adv-- keyserver hkp://keyserver.ubuntu.com:80-- recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5echo "deb [arch=amd64,arm64,ppc64el,s390x] http://repo.mongodb.com/apt/ubuntu xenial/mongodb-enterprise/3.6 multiverse" | sudo tee / etc/apt/sources.list.d/mongodb-enterprise.listsudo apt-get updatesudo apt-get install-y mongodb-enterprise

For other system installations, refer to the official manual:

Https://docs.mongodb.com/manual/installation/

Install the Python dependency package

Cd InsectsAwakesudo pip install pip-Usudo pip install-r requirements.txt

Install nmap

Sudo apt install nmap

Configuration database

Sudo mkdir-p / data/dbsudo service mongod startmongo-- host 127.0.0.1 purl 27017

Use InsectsAwakedb.createUser ({user:'you username',pwd:'you password',roles: [{role:'dbOwner',db:'InsectsAwake'}]}) exit

Add user

Start the database

Modify scanner configuration

Class Config (): WEB_USER = 'admin' / / scanner login user WEB_PASSWORD =' whoami' / / scanner login password WEB_HOST = '127.0.0.1' / / local access WEB_PORT = 5000 / / Web service port POCSUITE_PATH = basedir +'/.. / InsectsAwake/views/modules/scanner/pocsuite_plugin / 'class ProductionConfig (Config): DB_HOST =' 127.0.0.1'/ / Database address DB_PORT = 27017 / / Database Port DB_USERNAME = 'testuser' / / Database user DB_PASSWORD =' testpwd' / / Database password DB_NAME = 'test' / / Database name / / Database collection name PLUGIN_DB = 'test_plugin_info' TASKS_DB =' test_tasks' VULNERABILITY_DB = 'test_vuldb' ASSET_DB =' test_asset' CONFIG_DB = 'test_config' SERVER_DB =' test_server' SUBDOMAIN_DB = 'test_subdomain' DOMAIN_DB =' test_domain' WEEKPASSWD_DB = 'test_weekpasswd'

Relatively lazy, clone down directly chmod-R 777 and then you can happily modify the file

Scanner profile path: InsectsAwake-Project/instance/config.py

Initialize the database

Cd / InsectsAwake/migrationpython start.py

Operating system

Sudo. / run.sh restart

Run.sh will report errors related to nohup. Please refer to https://blog.csdn.net/educast/article/details/28273301 for appropriate modification.

Success

The project runs at 127.0.0.1 WEB_HOST 5000 by default (default WEB_HOST and WEB_PORT can be modified) and cannot be accessed via public network. It is recommended to configure Web service proxy access such as Nginx or Caddy.

Postscript

Update the system after installation. Tsinghua Source is recommended.

When installing mongodb-enterprise, it is recommended to go slowly and succeed, otherwise the installation will be very troublesome. Apt various errors are basically equivalent to scrapping and need to start all over again.

When you start run.sh, you can wait to see the web page.

Work flow

The following figure shows the simple SDL flow chart of the system operation.

Project structure ├── InsectsAwake │ ├── app.py flask blueprint Registration │ ├── _ _ init__.py │ ├── static Page static folder │ ├── templates Page html templates folder │ └── views │ ├── asset_management.py flask Asset Management processing Page │ ├── authenticate.py flask Page Certification Page Log in and log out of │ ├── dashboard.py dashboard page │ ├── index.py home page │ ├── _ _ init__.py │ ├── lib │ │ ├── _ _ init__.py │ │ ├── mongo_db.py create and manage mongo database │ ├── modules │ │ ├── discovery Asset Discovery │ ├── _ _ init__.py │ ├── port_scanner.py scan Port and Target Service Information │ │ ├── _ _ init__.py │ │ ├── scanner vulnerability scan │ ├── _ _ init__ .py │ ├── pocsuite_plugin folder where vulnerability scanning poc plug-ins are stored │ ├── pocsuite_scanner.py calls the pocsuites framework to scan targets for vulnerabilities │ ├── vulnerability_plugin.py management vulnerabilities poc plug-ins Add plug-in information to the database │ │ ├── subdomain subdomain blasting │ ├── _ _ init__.py │ ├── subdomain.py subdomain blasting │ │ └── week_passwd │ │ ├── _ _ init__.py │ │ The ── week_http_passwd_test.py author has not yet implemented the │ ├── plugin_management.py flask poc plug-in management page │ ├── settings.py flask platform parameter settings page Such as the number of threads Dictionary │ ├── sql_injection.py author has not implemented │ ├── subdomain_brute.py flask subdomain name burst page │ ├── task_management.py flask task management page │ ├── vulnerability_management.py flask vulnerability scan results management page │ ├── weak_passwd_test.py falsk weak password management page ├── InsectsAwake.py main Execute file ├── instance │ ├── config.py flask configuration and database configuration │ ├── _ _ init__.py ├── LICENSE ├── logs │ ├── db.log │ └── log.log ├── migration │ ├── DataModels │ └── .py create a database ├── requirements.txt ├── run.sh ├── tests domain.dict subdomain blasting dictionary project mind map

Master file analysis def scanner (): call vulnerability scanning module: return: config_db db_name_conf () [] scanner_time int (connectiondb (config_db). Find_one () []) print () scanner_loop_execute (scanner_time) def manage (): call flask: return: app.runflask_app.config.get () Flask_app.config.get () def discovery (): call the asset discovery module: return: print () scheduler BlockingScheduler () try: scheduler.add_job (MultiProcess () .start_port_scan, ) scheduler.start () except Exception as e: print (e) def subdomain (): call subdomain name blasting module: return: scanner_time print () subdomain_loop_execute (scanner_time) _ _ name__: start four threads to execute these four modules T1 threading.Threadscanner, ()) T2 threading.Threadmanage, ()) T3 threading.Threadsubdomain () T4 threading.Threaddiscovery, () t1.start () t2.start () t3.start () t4.start () t1.join () t2.join () t3.join () t4.join () subdomain name blasting module

This module is separate from the server Flask. Flask is responsible for storing and updating the information about subdomain name cracking accepted by the front end to the database, while the core subdomain name brute force cracking program uses the scheduler to periodically check the configuration files in the database. When there is data in the database to meet the conditions for the brute force cracking program, it starts to execute the program, violently crack the subdomain name of the target, and store the results in the database.

Brief introduction of InsectsAwake/views/subdomain_brute.py core files, which are used to find InsectsAwake/views/modules/subdomain/subdomain.py flask backend files for subdomain names of targets. Used to accept the front-end data and store it in the database to execute the function def subdomain (): run the subdomain name demolition program scanner_time print () subdomain_loop_execute (scanner_time) every 30s to analyze the detail function class DomainsBrute: def _ init__ (self, target_domain, subdomain_dict, domain_id) Domain_name): initialization class and member variables: param target_domain: target domain name host: param subdomain_dict: subdomain name burst dictionary: param domain_id: IP: param domain_name: domain name corresponding to the domain name in the database For example, Baidu def resolver_check (self): randomly generate a domain name and deal with it If this random domain name exists, the resolution result is returned: return: [] or False def handle_domain (self): form a new secondary or tertiary domain name: return:: [u, u, u] def handle_result (self): get the processing result. If the secondary or tertiary domain name exists, it is stored in the database. : return: def save_db (self Result): store the domain name burst result in the database test_subdomain: param result:: return: def run_multi (self): multi-process resolution domain name: return: self.handle_domain () scanner_pool multiprocessing.Pool) self.result scanner_pool.map (ha_resolver_domain) Self.domain_list) scanner_pool.close () scanner_pool.join () self.handle_result () def ha_resolver_domain (domain): resolve domain name Return the resolution result: param domain:: return: {: [,]} or {} def start_brute (inc_time):: param inc_time:: return: schedule.enter (inc_time, start_brute, (inc_time) ) subdomain_list connectiondb (config_db) .find_one () [] domain_text connectiondb (domain_db) .find (): domain_text []: domain_list domain_text [] domain_id domain_text [] domain_name domain_text [] print () start_date datetime .now () connectiondb (domain_db) .update_one ({: ObjectId (domain_id)} {: {:}}) target domain_list: DomainsBrute (target, subdomain_list, domain_id Domain_name) .run _ multi () domain_text []: result connectiondb (subdomain_db). Find ({: ObjectId (domain_id)}): next_subdomain eval (result []). Keys () [0] DomainsBrute (next_subdomain, subdomain_list, domain_id Domain_name) .run _ multi () connectiondb (domain_db). Update_one ({: ObjectId (domain_id)}, {: {:}}) scan_time datetime.now () start_date print (, scan_time.total_seconds () def subdomain_loop_execute (inc,): schedule.enter (inc, start_brute, (inc) )) schedule.run ()

With a little modification, https://github.com/cmustard06/subdomain can now be used separately

Port scan module

The port scan module is still separate from the falsk framework, and it is an independent module, and the data exchange is mainly carried out through the database. Run the program periodically. By looking at the configuration information of the port scan module in the database, if the configuration meets certain conditions, the program starts to execute. The module uses a third-party module APScheduler, which is a Python timing task framework, which is very convenient to use. Provides tasks based on dates, fixed intervals, and crontab types, and can persist tasks and run applications as daemon. The function implements the start_port_scan function that is executed at 14:47 every day.

Cheduler.add_job (MultiProcess (). Start_port_scan, 'cron', day='1-31, hour=14, minute=47) def nmap_scanner (target_host): scan the specified disconnection of the target host and return the scan result: param target_host:: return: [{:,:},] bug exists in the original function Modify port_scanner.scan (target_host .join (% port port eval (target_ports)) class MultiProcess: def _ _ init__ (self): self.target_list [] self.server_db db_name_conf () [] self.asset_db db_name_conf () [] self.processes_count int (connectiondb (config_db). Find_one () []) self.asset_id Self.asset_name def scan_pool (self): multi-process port scanning with different targets Update the result to the database: return: def start_port_scan (self): extract the data from the database and call the start_pool function: return: vulnerability scanning module

The vulnerability scanning module uses pocsuite, an open source scanning framework that knows Chuangyu. Because the framework has modified many functions in the process of later upgrade and maintenance, exceptions may occur when running the program using the latest version of the pocsuite framework. Version 2.0.4 of the framework is used here. For example, when testing, due to an error when importing the following module, you can find out by looking at the original function.

From pocsuite.lib.utils.password import genPassword

There is no genPassword function, so the password.py file is modified manually, and the contents of the modified file are as follows

Copyright (c) pocsuite developers (https://seebug.org)See the file copying permissionimport stringfrom pocsuite.lib.core.common import getFileItemsfrom pocsuite.lib.core.data import pathsfrom random import choicedef getWeakPassword (): return getFileItems (paths.WEAK_PASS) def getLargeWeakPassword (): return getFileItems (paths.LARGE_WEAK_PASS) def genPassword, string.letters string.digits): return .join ([choice (chars) _ range (length)])

After the modification is completed, continue to run the test program, running successfully. The results obtained the data of the vulnerability scan results, as follows

(, (0,)) code analysis

Function analysis

Def verify_poc (self, target): poc verification function: param target:: return: def start_scan (self): use multithreading for vulnerability scanning: return: def periodic_tasks (self): get task information from the TV series library Scan periodically according to configuration information. Target: return: def scanner_loop_execute (inc): main program loop execution module: param inc:: return: database module database structure

# PLUGIN_DB plugin collection

-

Plugin_appversion affects version

Plugin_vultype vulnerability name

Plugin_vuldate vulnerability date

Plugin_filename file path

Plugin_name plug-in name

Plugin_appname Application name

Plugin_author plug-in author

_ id ObjectId

-

# TASKS_DB task collection

-

Task_status task status

End_date end time

Scan_target_list scan objects (list)

Task_name Task name

Plugin_id plug-in id

_ id ObjectId

Start_date task start time

Task_plan scan schedule

-

# VULNERABILITY_DB vulnerability Collection

-

Scan_result scan result

Target scan object

Task_id Task ID

Appname Application name

Scan_date scan date

Poc_name plug-in name

Vulversion vulnerability affects version

Poc_vultype vulnerability Typ

Task_name Task name

Plugin_id plug-in ID

_ id ObjectId

-

# ASSET_DB Asset Library Collection

Asset_date creation date

Scan_option asset discovery

Asset_text assets

Asset_name Asset Library name

Name of dept_name department

Admin_name administrator

_ id ObjectId

-

# CONFIG_DB configuration set

-

Port_thread Port scan Thread

Scanner_thread vulnerability detection thread

Port_list Port scan list

Config_name profile name

Subdomain subdomain name dictionary

-

# SERVER_DB Service Collection

-

Host host

Asset_id Asset Bank ID

Port port

Port_server service

Banner fingerprint (cpe)

Scan_date scan date

Asset bank to which asset_name belongs

-

# DOMAIN_DB Service Collection

-

Domain_text primary domain name

Scan_option three-level domain name scanning

Dept_name domain name belongs to the department

Domain_date creation date

Domain_name domain name

_ id ObjectId

-

# SUBDOMAIN_DB Service Collection

-

Date scan date

Domain primary domain name

_ id ObjectId

Result subdomain name

Domain_id primary domain name ID

Domain_name domain name

-

# WEEKPASSWD_DB Service Collection

-

Date scan date

Target detection object

Task_name Task name

Post_data login packet

Status detection status

Number of week_passwd_count weak passwords

Error_data failure flag

Success_data success Mark

Username account number

Password password

The result of weak password in week_passwd_result

_ id ObjectId

-

`

Database details

Test_asset: this table collection is used to store asset information. Through the information in this table, the vulnerability scanning module and nmap module can be called later to scan the asset.

Test_config: this table is used to store data such as domain name dictionary, ports to be scanned, number of processes opened, etc.

Test_domain: this table is used to store domain name information and perform subdomain name explosions on the domain names in the table through the data in the table.

Test_plugin_info: this table is used to store scan plug-in information, including plug-in name, function, and the path where the plug-in is stored.

Test_server: this table stores asset information, including open port information and services

Test_subdomain: this table is used to store the burst results of subdomains.

Test_tasks: this table is used to store vulnerability scanning task information

All the modules used in the system communicate with each other through the database, and each module runs relatively independently. There is no problem with testing separately, and it is also very friendly when analyzing the code. One notch is that there are some problems with the pocsuite open source framework used by the vulnerability scanning module. New functions are added and old functions are removed in each iteration. As a result, the official pocsuite poc verification plug-in is prone to problems, and the old version of the poc plug-in cannot run under the new framework!

The above is the editor for you to share how to carry out InsectsAwke automation tool source code analysis, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.