Shulou(Shulou.com)06/02 Report--

Through the previous 35 articles, completed the python and docker together to grab the mobile end, let's sum it up.

Summary

Crawlers are divided into: web web page data crawling, app application data crawling, software application data crawling, this "docker practical article" is mainly aimed at crawling app application data, what are the pain points of crawler engineers when crawling applications?

I don't know how to grab the bag. I don't know how to grab the bag for mobile data.

In order to prevent cracking, some app add shells and strengthen certificate verification, resulting in no way to start.

Only know that appium mobile automation tool can support automation, but do not know that it can also be a crawler

Do not know how the crawler system is deployed to the server

Solution

In order to solve the problem that there is no Android second-hand phone, we use the simulator to simulate the Android phone to crawl. It really hurts to use your own mobile phone, no matter you can use the simulator.

The use of fiddler and mitmpoxy, interception filtering and the interaction of Python scripts have all been tested.

When a strong certificate is verified, it can be dealt with by cracking.

The way to crack the js of Douyin in web interface is also explained.

Appium is a sharp weapon in the hands of automated test engineers, and crawlers can also be used.

After the python crawler script is ready, you can map the local file to the virtual machine and then to the docker container by mounting the shared file on the virtual machine.

PS: finally, in the practice of docker, I also fulfilled my promise about the advanced docker of reptiles. In fact, most of the time, there is a lack of a train of thought. All roads lead to Rome, learn more and ask more. Through the practice of reptiles, let the old iron go further and further to docker, and use it more and more professionally. Thank you for your support.

The original article is welcome to reprint. Reprint please indicate: reproduced from IT Story Association, thank you!

> > original link address: "docker practice" python's docker- to create a multi-tasking app application data capture system (termination) (36)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.