Shulou(Shulou.com)06/03 Report--

This article is to share with you about how crawlers add proxy ip. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

The crawler uses the proxy ip to break the frequency limit, which allows the crawler to grab information efficiently. Basically, a large number of crawlers are driven by tasks, and in order to speed up the completion of these tasks, it is necessary to use a proxy ip. Next, how to add a proxy ip to a crawler?

First, the acquisition of proxy IP

It can be obtained from the following ways:

1. Free ip

Get it from a free website, the quality is very low, and there is very little IP available

2. Paid ip

For fee-based agency services, the quality is much higher

3. Set up a proxy server

Build their own proxy server, stable, but need a lot of server resources.

The proxy IP pool in this paper is through the crawler to obtain the proxy IP from several free websites in advance, and then check whether the IP is available, store it in the MongoDB if available, and finally display it on the front-end page.

2. Obtain available Proxy

The core code for getting the agent is ProxyManager, which is implemented in RxJava2 and mainly does the following:

1. Create a ParallelFlowable

Crawl in parallel for each page that provides a free proxy IP.

2. Crawl each page

Return to List

3. Check

Check the list of proxy IP obtained from each page to determine whether it is available or not

4. Save to proxyList in turn

When doing crawlers, it is necessary to maintain an available proxy IP pool. Of course, if you want to pursue a more stable proxy IP, it is better to consider high quality.

If we do not use proxy ip, we can only extend the interval time and frequency of requests in the process of crawling, in order to better avoid being banned by the server, so we have a large number of proxy ip resources, so it is more convenient to crawl.

Thank you for reading! This is the end of this article on "how to add agent ip for reptiles". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.