Shulou(Shulou.com)06/03 Report--

This article mainly explains "what ways do crawlers use http proxies?" Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "What are the ways for crawlers to use http proxy"!

1. Each process randomly takes out the IP list from the interface and reuses it. After failure, call API to get.

The general logic is as follows:

(1) Each process randomly retrieves a portion of ip from the interface and repeatedly tries the ip directory to obtain data;

(2) If the visit is successful, move on to the next one.

(3) If it fails, take a batch of IP from the interface and continue to try.

Disadvantages: Each IP has an expiration date. If you extract 100 and use the 20th, most of the rest may not be available. If the HTTP request is set to take more than 3 seconds to connect and more than 5 seconds to read, it may take 3-8 seconds and may fetch hundreds of times in 3-8 seconds.

2. Each process randomly selects an IP from the interface to use. If it fails, call API to get IP.

The general logic is as follows:

(1) Each process randomly retrieves an ip from the interface, uses it to browse resources,

(2) If the visit is successful, move on to the next one.

(3) If it fails, take an IP randomly from the interface and continue to try.

Disadvantages: API is called very frequently to obtain IP, which will cause great pressure on the proxy server, affect the stability of API interface, and may limit extraction. This scheme is not suitable for long-term stable operation.

3, first extract a large number of IP import local database, extract IP from the database.

The general logic is as follows:

(1) Create a table in the database, write an import script, how many APIs are needed per minute (consult the proxy IP service provider for advice), and import the IP list into the database.

(2) Record import time, IP, Port, expiration time, IP availability status and other fields;

(3) Write a grab script that reads available IPs from the database, and each process fetches an IP from the database for use.

Execute fetch, judge result, process cookie, etc., As long as there is a Captcha or failure, give up ip and replace ip again.

At this point, I believe that we have a deeper understanding of "what ways do crawlers use http proxies?", we may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.