Shulou(Shulou.com)06/01 Report--

Distributed crawler architecture did not emerge from the beginning. It's a process of gradual evolution.

At the beginning, we usually complete the introduction and development of the crawler on the personal computer, but in the real production environment, we can not use the personal computer to run the crawler, but deploy the crawler on the server. Taking advantage of the fact that the server does not shut down, the crawler can run 24 hours a day. The structure of the stand-alone crawler is shown below.

However, because the crawler crawls the data, the crawling frequency can not be too fast, even if the crawler runs continuously on the server, the efficiency may not meet the actual demand. At this time, it is necessary to deploy crawlers on multiple computers and use distributed crawler architecture for data crawling. The architecture of a distributed crawler is generally as follows.

The adoption of a distributed crawler architecture brings the following benefits.

1. The efficiency of reptiles is improved. This is obvious. It used to be running on a single machine, but now it is running on multiple machines distributed, and the efficiency has been significantly improved. 2. The reliability and availability of the crawler is improved. Previously deployed on a server, the crawler is not available when the server fails or the crawler fails. After adopting the distributed crawler architecture, the crawler task producers, task queues and crawler task consumers all adopt the distributed architecture deployment, in which some machines fail, which does not affect the overall availability, and the reliability of the system is greatly enhanced.

Summary

The above is the whole content of this article. I hope the content of this article has a certain reference and learning value for everyone's study or work. Thank you for your support. If you want to know more about it, please see the relevant links below.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.