Shulou(Shulou.com)06/03 Report--

This article mainly shows you "what are the http agents ip that can be used by reptiles", the content is simple and clear, and I hope it can help you solve your doubts. Let me lead you to study and learn what http agents ip can be used by reptiles.

First, each process interface randomly takes the IP list and uses it repeatedly, and if it is invalid, call API to get it.

The general logic is as follows:

1. Each process, randomly recycle a part of ip from the interface, and repeatedly try the ip directory to capture data.

2. If the visit is successful, continue to catch the next one.

3. After failure, take IP from the interface and continue to try.

Disadvantages of the scheme: all IP have deadlines, 100 are extracted, and when the 20th is used, the rest may not be available. When setting up a HTTP request, the connection time is more than 3 seconds, and the read time is more than 5 seconds, which may take 3 to 8 seconds, and may be caught hundreds of times in those 3-8 seconds.

Second, first extract a large number of IP, import it into the local database, and then extract IP from the database.

The general logic is as follows:

1. Create a table in the database, write an import script for API per minute (please consult the agent IP service provider for advice), and import the IP list into the database.

2. Record the import time, IP, port, expiration time, IP availability and other fields in the database

3. Write a crawl script that reads the available IP from the database, and each process fetches an IP usage from the database.

4. When crawling, judging the result, dealing with cookie, etc., whenever there is a CAPTCHA or error, abandon the IP and replace the IP.

The above is all the contents of this article entitled "what are the http proxy ip that can be used by crawlers?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.