Shulou(Shulou.com)06/02 Report--

Python web crawler commonly used several skills, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Crawlers (web crawlers) can be understood as a kind of spiders crawling on the web. The Internet is like a big web, and crawlers are spiders crawling around on this web. If the crawler encounters a resource, it will grab it. As for what resources to grab, this is controlled by the user.

For example, a crawler crawls a web page and finds a path in the web, that is, a hyperlink to the web page, and it can climb to another web page to obtain data. In this way, the whole web is within reach of the spider, and it is not a problem to climb down in minutes.

There are many ways for Python to crawl, and now I will introduce some of the more commonly used ones.

1. Basic method

The most basic web crawl in Python can be achieved by using request in the urllib module. These two functions have been implemented in network programming, and the code is as follows:

As a result, you can output a lot of html-style text, most of which is useless information. Although this method is very simple, the captured information has not been processed, so it is not of much use.

2. Use a proxy server

Why use a proxy server? at present, many websites have an anti-crawler mechanism. Once it is found that an IP requests too many times or too frequently within a certain period of time, the IP may be marked as a malicious IP, thus restricting the access of the IP, or adding the IP to the blacklist so that it can no longer visit the site. At this point, we need to use a proxy server to continue grabbing the information we need by using different proxy servers.

The sample code is as follows:

As with the basic method, such crawling information is not processed and the results are of little use. It needs to be reprocessed to further reflect the value.

3.cookie processing

For sites with a slightly higher level of security, data cannot be crawled using the first two methods. These websites need to send URL request again to provide cookie information, otherwise the request cannot be successful.

The sample code is as follows:

Of course, this is also a simple way, and can be extended to a more complex pattern.

4. Disguise as a browser

At present, many websites have an anti-crawler mechanism, which will reject all crawler requests.

How can a program tell whether a request is a normal request or a crawler request? The program judges whether a request is a normal request by judging whether there is browser information in the request. When visiting a website with an anti-crawler mechanism, we set the browser information in the request (disguised as a browser) by modifying the header in the http package.

The sample code snippet is as follows:

After reading the above, do you have several ways to master the common skills of Python web crawlers? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.