Shulou(Shulou.com)06/01 Report--

How to deal with the ip in the anti-crawling mechanism? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

As crawlers, we all know that many websites have anti-crawling measures to prevent others from crawling data, and blocking ip is one of the most common anti-crawling strategies. Usually their strategy is to find that if you visit too much in a short period of time, your ip will be blocked. To solve this strategy, it is very simple. We can OK by limiting the frequency of visiting the website or adding a pool of IP agents. In the crawler, we only need to add agents to do it, such as the example.

#!-*-encoding:utf-8-*-import requests import random # Target page to visit targetUrl = "http://www.weibo.com" # Target HTTPS page to visit # targetUrl =" https://www.weibo.com" # proxy server (product website www.16yun.cn) proxyHost = "t.16yun.cn" ProxyPort = "31111" # proxy tunnel authentication information proxyUser = "16VIZRXL" proxyPass = "125478" proxyMeta = "http://%(user)s:%(pass)s@%(host)s:%(port)s"% {" host ": proxyHost "port": proxyPort, "user": proxyUser, "pass": proxyPass,} # set http and https access using HTTP proxy proxies = {"http": proxyMeta, "https": proxyMeta } # set IP switchhead tunnel = random.randint (1p10 000) headers= {"Proxy-Tunnel": str (tunnel)} resp = requests.get (targetUrl, proxies=proxies, headers=headers) print resp.status_code print resp.text the answer to the question on how to seal ip in the anti-crawling mechanism is shared here. I hope the above content can help you to a certain extent, if you still have a lot of doubts to be solved, you can follow the industry information channel to learn more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.