Shulou(Shulou.com)06/02 Report--

This article mainly introduces "what are the knowledge points of Python reptiles breaking through anti-reptile mechanism". In daily operation, I believe that many people have doubts about Python reptiles breaking through anti-reptile mechanism knowledge points. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "Python reptiles break through anti-reptile mechanism knowledge points". Next, please follow the editor to study!

1. Build a reasonable HTTP request header.

The request header for HTTP is a set of properties and configuration information when you send a request to a network server. Because the browser and the Python crawler send different request headers, the anti-crawler is likely to be detected.

2. Establish a learning cookie.

Cookie is a double-edged sword. You can't have it, let alone without it. The site will track your visits through cookie, and will immediately interrupt your access if you are found to have crawler behavior, such as filling out forms too quickly or browsing a large number of web pages in a short period of time. And the correct handling of cookies, can also avoid many collection problems, it is recommended that in the process of collecting websites, check the cookie generated by these sites, and then think about which crawlers need to deal with.

3. Normal time difference path.

Python crawler should not break the principle of collection speed, as far as possible in each page access time to add a short interval, can effectively help you to avoid anti-crawling.

4. Use proxy IP. For distributed crawlers that have encountered anti-crawlers, using proxy IP will be your first choice.

When it comes to the development history of Python reptiles, it is simply a history of blood and tears in love with anti-reptiles. On the Internet, where there are web crawlers, there is absolutely no place without anti-crawlers. The premise of anti-crawler interception of the website is to correctly distinguish between people and web robots, and to prevent you from continuing to visit by restricting IP addresses and other measures when suspicious targets are found.

Knowledge point expansion:

Python3 Crawler-Anti-crawler coping Mechanism

Foreword:

Anti-crawler is more of an offensive and defensive war. Web crawlers generally have web crawlers and interface crawlers. To adopt the corresponding response mechanism for anti-crawler processing of websites, you generally need to consider the following aspects:

① access terminal restrictions: this can be achieved by forging dynamic UA

Restrictions on the number of ② visits: websites are generally located through cookie/IP, which can be countered by disabling cookie or using cookie pool / IP pool

③ access time limit: delayed request response

④ hotlink problem: generally speaking, the request of a web page is traceable, such as Zhihu's question answer details page. Normal user behavior must first enter the question page. When entering the answer details page, there is a strict request order. If you skip the previous request page, you may be judged to have arrived. This problem can be solved by forging the request header.

Specific anti-crawler strategies:

① verification code

Response: simple CAPTCHA codes can be identified by machine learning, and the accuracy can be as high as 50-60%. Complex CAPTCHA codes can be typed manually through a special coding platform (according to the complexity of CAPTCHA codes, the average charge for each code is 1-2 cents)

② seal ip (easy to kill by mistake)

Response: ip can be obtained through ip proxy pool / vps dialing, and hundreds of thousands of IP can be obtained at low cost.

③ sliding CAPTCHA: compared with conventional CAPTCHA, sliding CAPTCHA is easy to be recognized by machine learning, and sliding verification has certain advantages.

Response: simulated sliding to verify

④ context / hotlink protection: use the recording ability of token/cookie to correlate the context of the request, and determine whether the request is a crawler by judging whether the request has gone through a complete process; heavy and anti-crawler (Zhihu, headlines all have this mechanism)

Response: analyze the protocol and conduct full simulation

⑤ javascript participates in the operation: taking advantage of the fact that the simple crawler cannot perform json operations, the intermediate results are parsed / operated by js.

Response: automatic parsing can be done by having its own js engine module or directly using unprovoked browsers such as phantomjs.

⑥ session blocking: session requests exceed the threshold, thus blocking (easily lead to manslaughter)

⑦ UA blocking: ua requests exceed the threshold, thus blocking (easily lead to manslaughter)

⑧ web-fongt anti-crawler mechanism: the source code does not show the content, but provides the character set, which is defined by font-face on the page, and is mapped and displayed through unicode.

⑨ other ways: such as code confusion, dynamic encryption scheme, fake data, etc.

At this point, the study of "what are the knowledge points of Python reptiles breaking through the anti-crawler mechanism" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.