Shulou(Shulou.com)06/02 Report--

Original articles, welcome to reprint. Reprint please indicate: reproduced from IT Story Association, thank you!

Original link address: "docker practical article" python's docker crawler technology-guidance (1)

This album begins with a full set of combat processes from environment building to app data capture.

What will we learn?

Old man, what to do if there is only a computer without a mobile phone? it's all right. I thought of using a simulator. to be honest, the simulator is not as efficient as the mobile phone, but the simulator knows how to use the mobile phone or not?

Construction of Android Simulator package grab tool automatic control tool use

Analog click, analog input and other functions

Python writes crawler app data what can a multi-app data crawling system based on docker do to get the data?

Do an Internet project without basic data, all right?

Database analysis

Compared with the status of other companies, it can guide the operation of the company. Analyze the behavior and consumption needs of users.

User portrait

Accurately locate users and classify users.

Statistical system

Get the report data as a guide

Business competition

Through the data launched by rival app applications, recently launched such a new product in a city, a certain marketing data and trend, you can understand the opponent's operation in a city, so as to respond relatively.

For one's own good

The improvement of skills

There are many ideas for crawling on the web side, and there are a lot of github. But app has few skills to grab application data.

The interview must ask how to grab the data of the App application? Which applications have you crawled? How to use the bag grabbing tool? Is the data of App easy to grasp?

It's simple and difficult.

simple

The data in app is easier to grab than the web side, and the anti-crawling ability is not so strong. Most of them are http or https protocols, and most of the returned data types are json. Just parse the json directly.

Difficulties may require proper decompilation, parsing the encryption algorithm and grabbing information may be reinforced, need to be shelled, and then decompiled, analyze the encryption algorithm and capture information through a variety of certificates, signatures, device binding and other methods Find the skill points needed by the hidden encryption algorithm python crawler engineer python crawler development experience java development experience android development experience app reverse app shelling to crack the ultimate goal of encryption algorithm

Python automation applications, package grabbing tools, parsing app application methods to grab data. There is no need to crack the encryption algorithm.

PS: I have also studied python for a while, climbed some data and app, the company is still stuck in manual click testing, my colleagues also urged me to help write tutorials, preferably to complete the automated testing of python around docker, using the app testing tool appium. And we're on the move. There are a lot of grey technology, fashion technology, multi-tool comparison, . I hope the old man can arrive at get!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.