Shulou(Shulou.com)06/03 Report--

Editor to share with you how distributed crawlers use proxy IP, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's learn about it!

1. In method 1, each process randomly takes an IP from the interface API to use, and if it fails, call API to get an IP. The logic is as follows:

(1) each process randomly retrieves an IP from the interface and uses this IP to access resources

(2) if the visit is successful, continue to catch the next one.

(3) if it fails, take a random IP from the interface and continue to try.

Note: the behavior of calling API to obtain IP is very frequent, which will put great pressure on the proxy server, affect the stability of the API interface, and may be limited to extraction. This kind of plan is not suitable and can not run stably and endurably.

2. In method 2, each process randomly takes an IP list from the interface API to recycle, and if it fails, call API to get it. The logic is as follows:

(1) each process randomly retrieves a batch of IP from the interface and iteratively tries the IP list to grab the data.

(2) if the visit is successful, continue to grab the next one.

(3) if it fails, take another batch of IP from the interface and continue to try.

Note: each IP has a validity period. If 100 is extracted, when the 10th is used, most of the rest may be invalid. If you set the HTTP request time-out time-out to 3 seconds and the read time-out to 5 seconds, you will probably spend 3-8 seconds, which may have been able to grab dozens of times.

These are all the contents of the article "how distributed crawlers use proxy IP". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.