Shulou(Shulou.com)06/02 Report--

This article introduces how to grab Douyin app hot data in Python, the content is very detailed, interested friends can refer to, I hope it can be helpful to you.

First of all, we need to install charles grab package APP data, which is a fee-based package modification tool that is easy to use, easy to control data requests, simple to modify, and easy to pause the start of grabbing data. There is also a Chinese version on the Internet, the download address is http://www.zdfans.com/html/42074.html, and ok is installed by default all the way.

After the installation is complete, to set the agent, click Agent-Agent Settings.

Then set the proxy on the mobile, as shown in the following figure:

In the case of ensuring that the phone and the computer are on the same local area network, the host name of the proxy server is set to the ip address of the computer and the port is set to 8888.

Finally, the certificate is installed on the computer side and the mobile end respectively.

Computer-side installation method: click help-- ssl Agent-- install the charles root certificate, and click the following figure to install.

Mobile installation method: help-ssl agent-install charles root certificate on mobile device or remote browser.

Then type chls.pro/ssl in the simulator browser, and the mobile certificate will be downloaded automatically.

Finally, on the mobile phone, click Settings-Security-install from the SD card.

Name the certificate, click OK and the installation is successful.

Open charles, and then open the hotspot list interface of Douyin app. The data interface can be easily found in charles. 50 pieces of data are returned at a time, as shown in the following figure.

Its url information is shown in the following figure.

This API can only return hot spot data at this moment. If you want to return new data, you need to change the parameter information. However, the parameters of the data interface on the App side are relatively complex, so we will not analyze them in depth here.

In order to solve this problem, we can use appium to simulate the operation of the mobile phone, and then use mitmproxy to intercept the data. (there are many tutorials on the introduction and installation of appium and mitmproxy, which will not be repeated here.)

The Appium script is shown in the following figure:

This automated test script is relatively simple, mainly to repeatedly get the latest information about hot spots.

The Mitmproxy script is shown below:

There are four points to pay attention to:

1. Before grabbing the package with mitmproxy, set the mobile agent ip port to 8080. The setting method is the same as above.

two。 If you want to run the external function in this script, you must add the first two lines, or you will make an error

3. If url in flow.request.url in the script is the judgment condition of the data flow. If url is in the url request data of the data stream, the data is judged to be Douyin app hotspot data.

4. Finally, run the following program in the path where the script is located:

About how to crawl Douyin app hot spot data in Python to share here, hope the above content can have certain help to everybody, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.