Shulou(Shulou.com)06/01 Report--

Most people do not understand the knowledge points of this article "how to optimize python crawler", so the editor summarizes the following content, detailed content, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "how to optimize python crawler" article.

Use header

This is a very important knowledge, when making a page request, there will be some information in the request header. If you use a crawler, there is no such information by default (so when the server sees that there is no relevant information in the request header, it suddenly knows that you are a crawler). Some servers do not return responses to such requests (that is, relatively simple anti-crawlers). In fact, when using request for page fetching, the get () method can pass in a header. As long as the corresponding browser request header information is stored in header, the crawler can pretend to be a browser, so that the server will return the response normally.

Request library

Some crawlers use the urllib library that comes with python for crawler development, which is very powerful. However, the request library, which is more powerful than the urllib library, can simulate browser operations with less code.

Beautiful Soup library

This is an efficient xml parsing library that can extract data from HTML or XML. Using this library, you can easily and quickly locate and extract html data. If you can use a css selector, you can even use it in conjunction with a css selector. With him, we can basically bid farewell to regular matching.

Selenium library

This is a library that is often involved in automated testing. This library can control the browser by simulating user operations. Crawlers can also use this library to control browsers and obtain data. However, because you have to open a browser to run Selenium, crawlers are clumsier and slower than crawlers that do not use Selenium. However, because he operates the browser directly, he does not need browser camouflage, and some data requires users to have certain operations when crawling, which only Selenium can do.

Use multithreading

A single-threaded crawler is like a person working, after all, it can't do too many threads. The use of multi-threading can greatly improve the crawling speed of your reptiles.

Use the ip proxy

Header was introduced earlier. In order to prevent crawlers (especially those disguised by browsers), some servers will process requests with the same ip address (when the same ip makes multiple requests to the server in a short period of time). At this time, learn to use the ip proxy pool to disguise your ip address to bypass this detection mechanism.

Use Cookie

When you need to log in, you can use Cookie to log in.

Note: for login, you can also use Selenium to automate login, or use a form to request the server.

Data storage

This is divided into three cases, salted fish without dreams can directly save the data as a text file using the built-in file function.

If you want to save the data in csv format, you can learn about this library-the csv library. This library can read and write csv files, save the file in this format can use Excel to open this kind of data, tabulated data is more intuitive.

If you want to save the data to the database, you can use the pymysql library. This library can operate the database, it is easier to manage the files in the database, and it is also convenient for other applications to call.

Crawler frame-scrapy

As with other languages, some technologies can be integrated into a framework. Crawlers also have such a framework, and that is scrapy. Using this framework, you can develop crawlers more quickly.

The above is about the content of this article on "how to optimize python crawlers". I believe we all have a certain understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.