Shulou(Shulou.com)06/03 Report--

This article mainly introduces how the Python crawler architecture is composed, which is very detailed and has a certain reference value. Friends who are interested must finish reading it!

Python crawler architecture is mainly composed of five parts: scheduler, URL manager, web page downloader, web page parser and application.

1. Dispatcher.

Equivalent to the CPU of a computer, it is mainly responsible for scheduling the coordination between the URL manager, the downloader and the parser.

2. URL manager, including the URL address to be crawled and the URL address to be crawled.

Prevent repeated fetching of URL and circular fetching of URL. There are three main ways to implement URL manager, namely, memory, database and cache database.

3. Webpage downloader.

Download the web page by transmitting the URL address and convert the web page into a string. The webpage downloader includes urlib2 (Python official basic module), including login, agent, cookie, requests (third party package).

4. Web page parser.

Parse a web page string.

Useful information can be extracted according to our requirements, or according to the parsing method of the DOM tree. Web page parsers have regular expressions (intuitive, converting web pages into strings, extracting valuable information through fuzzy matching, which can be very difficult to extract data when documents are complex), html.parser (included with Python), beautifulsoup (third-party plug-ins, which can be parsed with html.parser that comes with Python, or lxml, which is better than other types), lxml (third-party plug-ins) Can parse xml and HTML), html.parser, beautifulsoup, and lxml are all parsed by the DOM tree.

5. Application program.

Is an application made up of useful data extracted from a web page.

The above is all the content of the article "how the Python crawler architecture is made up". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.