Shulou(Shulou.com)06/02 Report--

This article is about how Python cracked the anti-crawler, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Through the anti-crawler mechanism of using JS to generate random strings locally, it is a common problem when using Python to write crawlers.

Crack Youdao Translation's anti-crawler mechanism

Youdao Translation on the web side could climb directly before. In other words, as long as you get his interface, you can unscrupulously use his interface for translation without paying any fee. Well, since Youdao Translation launched his API service, he has made an anti-crawler mechanism for this interface. This anti-crawler mechanism is a very classic technical means in the field of reptiles. So what is the principle of his anti-reptile mechanism? How to crack it? Next, let's take you to find out.

1. Normal crawler process:

In the figure above, we can see that a lot of network requests have been sent. Here, we click on * * network requests to view:

Explain some of the more important data:

Other data types are not very important for the time being, they are all written in a fixed way, and we can just use shoes when we write the code later. Up to now, we can write a simple crawler to call the interface of youdao translation. The network request library we use here is urllib that comes with Python3, and the related code is as follows:

Second, crack the anti-reptile mechanism:

Then copy the formatted code, open it with sublime or pycharm, and search salt to find the relevant code:

Once we know how salt and sign are generated, we can write Python code to interface with his interface. Here is the code:

This is how Python cracked the anti-crawler. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.