Shulou(Shulou.com)06/03 Report--

This article is to share with you about the design points of distributed crawlers. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Distributed crawler solution.

In order to capture the data of a large station in batches, it is best to maintain 4 queues.

1. Url task column-stores url data to be captured.

2. The original url queue-stored in the captured web page, but not yet processed.

The main purpose of the processing is to check whether capture is required and whether repeated capture is required.

3. Raw data queue-the stored data is not processed.

4. Second-hand data column-stores the data to be stored in the data processing process.

Each of the above queues monitors the process of task execution, namely:

Crawler crawling process-monitors the url task queue, captures web page data, and loses the captured raw data to the original data queue.

Url processing: monitor the original url queue, filter exception url and repeatedly captured url.

Data extraction process: monitor the original data queue, extract the key data of the original data queue, including new URL and target data.

Data stored procedure: the second-hand data is sorted out and stored in mongodb.

The main points of the design of reptiles.

If you want to crawl a website in bulk, you need to build your own crawler framework. Before building, you should consider several issues: avoid being blocked IP, image CAPTCHA recognition, data processing and so on.

Relatively simple picture verification code can be written through the pytesseract library recognition program, which can only identify simple picture data. Slide the mouse, slider, dynamic image verification code and other more complex places can only consider the purchase of coding platform for recognition.

For data processing, if you find that the data you get is disrupted, the solution is to identify its disruption rules, or to obtain data extraction through the pythonexecjs library or other js libraries through the source js code.

Thank you for reading! This is the end of this article on "what are the design points of distributed crawlers". I hope the above content can be of some help to you, so that you can learn more knowledge. If you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.