Shulou(Shulou.com)06/02 Report--

How to set the proxy IP after the scrapy request exception? for this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

We all know that scrapy can set proxy IP, but we don't know if you have encountered such a scenario:

The regular proxy IP setting will take effect for every request, that is, every request will use the proxy

But some requests don't need a proxy at all. I want to set the proxy IP when my request goes wrong or times out.

This not only saves resources, but also shortens the request time. After all, the quality of some agents is really indescribable.

So how do you get it to use the proxy IP to rerequest when the request times out?

It's easy to think of downloading middleware, DowmloaderMiddleware

A hook between request and response, used to modify request and process response

First of all, we need to catch exception requests such as timeouts, and add the following code to the download middleware:

From twisted.internet.error import TCPTimedOutError, TimeoutError

Def process_exception (self, request, exception, spider):

If isinstance (exception, TimeoutError):

Self.process_request_back (request, spider) # connection timeout to enable proxy ip mechanism

Return request

Elif isinstance (exception, TCPTimedOutError):

Self.process_request_back (request, spider)

Return request

The purpose of the above code is to catch the exception request, call the function that sets the proxy IP, and return the request

The following code sets up the IP

Def process_request_back (self, request, spider):

Request.meta ["proxy"] = xun.proxy

Request.headers ["Proxy-Authorization"] = xun.headers

Note that the process_request () method cannot be used. If the agent is added under this method, it is the regular proxy IP method.

At the same time, when an exception such as TimeoutError,TCPTimedOutError occurs in scrapy, you can also solve it through the following solutions:

Change mainly through the settings file

1. Reduce the number of simultaneous requests CONCURRENT_REQUESTS = 5

2. Increase the timeout time DOWNLOAD_TIMEOUT = 300,

3. Increase the number of abnormal retries RETRY_TIMES = 5

This is the answer to the question about how to set the proxy IP after the scrapy request is abnormal. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.