Shulou(Shulou.com)06/03 Report--

How to use Python to solve the problem that can not be found in a good Douyin video? for this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

Today, we'll learn how to use Python to crawl Douyin APP data.

The tools you need to use to crawl Douyin APP data using Python are:

Programming tool: pycharm

App package grab tool: mitmproxy

App Automation tool: appium

Operating environment: windows10

Train of thought:

Suppose we have configured the tools we need.

1. Use mitmproxy to grab the app package of the phone and get the content we want.

2. Use appium automated testing tool to drive app to simulate human movements (slide, click, etc.)

3. Combine 1 and 2 to achieve the effect of automatic crawler.

Mitmproxy/mitmdump grabs the bag

Make sure that mitmproxy is installed, that the phone and PC are on the same LAN, and that the CA certificate of mitmproxy is also configured. There are many related configuration tutorials on the Internet, which I will skip here.

Because mitmproxy does not support the windows system, so here is one of its components mitmdump, it is the command line interface of mitmproxy, you can use it to dock our Python script, using Python to achieve the processing after listening.

After mitmproxy is configured, entering mitmdump on the console and opening Douyin app,mitmdump on the phone will present all requests on the phone, as shown below:

You can slide down all the way down on Douyin app. If you look at the request shown in mitmdump, you will find that the prefix is:

The url of these three types of prefixes is our target Douyin video url.

The next step is to write a python script to download the video, using mitmdump-s scripts.py (in this case, the python file name) to execute the script.

The code is rough, but the basic logic is clear, so we can download the Douyin video, but there is a drawback in this method, which is that to get the video, we need people to constantly slide Douyin's next video, which we can use a powerful automated testing tool for appium.

Appium simulates the phone to ensure that the environments Android and SDK that appium depends on are configured, and there are many tutorials on the Internet, so I won't talk about it here.

The use of appium is very simple. First, let's open appium. The startup interface is as follows:

Click the Start Server button to start the appium service.

Connect the Android phone to the PC through the data line, and at the same time turn on the USE debugging function. You can enter the adb command (specifically you can find it on the Internet) to test the connection. If the following results occur, the connection is successful.

Model is the name of the device, which is needed for later configuration. Then click the button indicated by the arrow below and a configuration page will appear:

Configure the Desired Capabilities parameters to launch app in JSON Representation in the lower right corner, which are paltformName, deviceName, appPackage, and appActivity, respectively.

PlatformName: the name of the platform, usually Android or iOS.

DeviceName: device name, specific type of mobile phone

AppPackage:App package name

AppActivity: entry Activity name, usually with. The beginning

PlatformName and deviceName are relatively easy to obtain, while appPackage and appActivity can be obtained in the following ways.

Enter the command adb logcat > D:\ log.log on the console, and open Douyin app on your phone, then open the log.log file on disk D and look for the Displayed keyword.

From the figure above, we can see that the com.ss.android.ugc.aweme after Displayed corresponds to appPackage,.main.MainActivity and corresponds to appActivity. Finally, our configuration result is as follows:

Then click Start Session to start Douyin app on the Android phone and go to the launch page, at the same time, a debugging window will pop up on PC, from which you can preview the current mobile page and simulate various operations on the phone, which is not the focus of this article, so skip it.

Below we will use the python script to drive app, which can be run directly in pycharm.

The effect is as follows:

On how to use Python to solve good Douyin video accidentally crossed over can not find the answer to the question to share here, I hope the above content can be of some help to you, if you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.