Shulou(Shulou.com)06/01 Report--

Many friends will encounter a problem when they just do a crawler. Obviously, I have used the crawler agent ip. I happily went to grab a website. I was about to make a cup of coffee. I found that the crawler program was terminated. Once checked, the real ip was blocked. At this time, I would be very puzzled. I obviously used the proxy ip, ah, why is my real IP still blocked? Is it the quality and problem of IP? I was cheated by the proxy IP provider?

Of course, this possibility cannot be ruled out, for example, if you use a transparent proxy, the target server clearly knows that you have used a proxy and can also know your real IP, and of course, it will be blocked; another example is Puni proxy, where the target server knows that you are a wolf in sheep's clothing, and of course, it will block your real IP;. If you use high concealment agent ip, high concealment ip agent is the same as a normal family surfing the Internet, no difference. If the transparent proxy and the Punidon agent are identified by the target server and blocked IP, it should be that the quality of the proxy IP is not good. However, after using the Gaoni agent IP, it is still blocked by IP. What on earth is going on?

First of all, we need to know clearly, why do crawlers need a proxy IP? What is the purpose of proxy IP? What is the principle of proxy IP? The reason for the need for proxy IP is that a single IP will frequently crawl a target server, which will cause a huge load on the server. The server naturally has to impose certain restrictions (limit access frequency or limit access times, which is basically to reduce server pressure). If the frequency or number of times exceeds the limit, the anti-crawler will work, such as blocking the IP or blocking the entire IP. The role of proxy IP is to replace the blocked IP to continue crawling. When the threshold is reached, the proxy IP will also be blocked, and then a proxy IP will be blocked, and there are thousands of proxy IP, one after another, just to complete the crawling task. It can be seen that it is normal for the agent IP to be blocked. Many friends think that I used the agent IP, so why seal me? did I buy a junk IP agent? Then he came to the customer service to complain that he had been cheated and asked for a refund.

In fact, there are many reasons for sealing IP, the above kind of blocking is inevitable, it is not that the use of a proxy IP will not be blocked. But there are also abnormal blocked, such as the use of a "garbage agent" (transparent proxy). There is also the use of Gaoni agent ip, also did not crawl a few times to be blocked, this is why? That is because there is no good simulation of manual access to the server, that is to say, improper crawling strategy, such as not clearing cookie, or it is possible that the anti-crawling strategy of the target server has been upgraded. There are many reasons why the proxy IP is blocked. To find out why each proxy IP can play the greatest role is the job responsibility of the crawler, that is, to upgrade the crawler strategy.

As we mentioned above, when the frequency or number of times exceeds the limit, the anti-crawler will work, such as blocking the IP or blocking the entire IP. This is why the IP availability rate of some agents is low, because when an IP segment is blocked, the next IP you dial is also blocked, so it is not available, resulting in a low overall IP availability rate and wasting a lot of time.

Because many ADSL dialing agents IP dial out IP with serial numbers, such IP appears to be very fake and is very different from normal user access, so many anti-crawling programs will set up to block the entire IP address, resulting in batch IP unavailable, so the best IP resources are IP distributed disconnected, which can effectively improve the efficiency of various businesses.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.