Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Python for crawler technology", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use Python crawler technology" bar!

1. Grab

You don't have to use py's urllib, but you have to learn, if you haven't used it yet.

Better alternatives are third-party libraries that are more user-friendly and sophisticated, such as requests, and if pyer doesn't understand various libraries, it's for nothing.

The most fundamental thing to crawl is to pull the web page back.

If you go further, you will find that you have to face different web page requirements, such as authentication, different file formats, coding processing, a variety of strange url compliance processing, repeated crawling problems, cookies following problems, multi-threading and multi-process crawling, multi-node crawling, crawling scheduling, resource compression and a series of problems.

So the first step is to pull the web page back, and you will gradually find all kinds of problems to be optimized.

two。 Storage

When it comes back, it will generally be saved with a certain strategy, rather than direct analysis. I think a better framework should be to separate the analysis from the capture, and to be more loose. If something goes wrong in each link, it can isolate the problems that may arise in the other link. Easy to troubleshoot or update and release.

So how to save the file system, SQLorNOSQL database and in-memory database is the key point of this link.

You can choose to save the file system to start, and then name it with certain rules.

3. Analysis

Analyze the text of the web page, extract the link or extract the text, always look back to the demand, but what must be done is to analyze the link.

You can use what you think is the fastest and best method, such as regular expressions.

Then apply the results of the analysis to other links:)

4. Show

If you do a lot of things, there is no output at all, how to show value.

So to find a good display component, to show out of the muscle is also the key.

If you want to write about crawlers in order to be a station, or to analyze the data of something, don't forget this link and better show the results to others.

PySpider is an open source implementation of a crawler architecture done by binux. The main functional requirements are:

Crawl and update specific pages for scheduling multiple sites

It is necessary to extract structured information from the page.

Sensitive and scalable, stable and monitorable

This is also the requirement of most python crawlers-directional crawling and structured parsing. However, in the face of a variety of websites with different structures, a single crawling mode is not necessarily satisfactory, and sensitive crawling manipulation is necessary. In order to achieve this goal, simple configuration files are often not sensitive enough, so manipulating the crawl through a script is the last choice.

Functions such as de-rescheduling, queuing, fetching, exception handling, and monitoring are provided to the crawling script as structures and ensure sensitivity. Finally, the modified debugging environment of web and web mission monitoring are added to become this set of structure.

The design foundation of pyspider is: crawl ring model crawler driven by python script.

Through the python script to extract structured information, follow link scheduling crawling manipulation, to achieve the maximum sensitivity

After web-based scripting, debugging the environment. Web displays scheduling status

The capture ring model is mature and stable, and the modules are independent of each other. Through the connection of the audio queue, the capture ring model is expanded from single-process to multi-machine distributed sensitivity.

Pyspider-arch

The architecture of pyspider is mainly divided into scheduler (scheduler), fetcher (crawler) and processor (script execution):

Audio queues are used to connect various components, except that scheduler is a single point, fetcher and processor can be distributed and deployed with multiple instances. Scheduler is responsible for the overall scheduling and control.

The mission is initiated by scheduler, fetcher grabs web content, and processor performs pre-written python scripts to output the results or generate a new chain mission (sent to scheduler), forming a closed loop.

Each script can sensitively use a variety of python libraries to parse the page, use the structure API to control the next grab action, and control the parsing action after setting callback.

Thank you for your reading, the above is the content of "how to use Python for crawler technology". After the study of this article, I believe you have a deeper understanding of how to use Python for crawler technology, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.