Shulou(Shulou.com)05/31 Report--

This article mainly explains "python how to grab Douyin app". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how python crawls Douyin app".

To grab app, you first need a mobile phone or install an emulator on your computer. You need to know the default port of each simulator.

Step two, grab the bag.

There are many tools to grab bags. Just find what you think is convenient. What I recommend here is mitmproxy.

Mitmproxy is a package grab program that supports HTTP and HTTPS. It has functions similar to Fiddler and Charles, but it is a console operation.

Mitmproxy also has two associated components. One is mitmdump, which is the command line interface of mitmproxy. With it, we can dock Python scripts and use Python to realize the processing after listening. The other is mitmweb, which is a Web program through which we can clearly observe the requests captured by mitmproxy.

The mobile phone and PC are in the same local area network, and the proxy is set to the proxy address of mitmproxy, so that when the phone accesses the Internet, the traffic packets will flow through the mitmproxy,mitmproxy and forward these packets to the real server. When the server returns the data packets, the mitmproxy forwards them back to the phone, so that mitmproxy acts as a middleman to grab all Request and Response. In addition, this process can also interface with mitmdump. The specific contents of the captured Request and Response can be processed directly with Python. For example, after we get the Response, we can directly parse it and store it in the database, thus completing the data parsing and storage process.

Install mitmproxy (it is recommended to switch to a domestic source for installation, the method is self-Baidu) pip3 install mitmproxy

For mitmproxy, if you want to intercept HTTPS requests, you need to set up a certificate. Mitmproxy will provide a set of CA certificates after installation. As long as the client trusts the certificate provided by mitmproxy, it can obtain the specific content of the HTTPS request through mitmproxy, otherwise mitmproxy cannot parse the HTTPS request.

First, run the following command to generate the CA certificate and start mitmdump: go to the directory cmd and run it directly

Mitmdump

Then find the CA certificate in the user directory

There are five certificates, which are briefly described in Table 1-1.

Table 1-1 5 certificates and their descriptions

Mitmproxy-ca.pem

Certificate private key in PEM format

Mitmproxy-ca-cert.pem

PEM format certificate for most non-Windows platforms

Mitmproxy-ca-cert.p12

Certificate in PKCS12 format for Windows platform

Mitmproxy-ca-cert.cer

Same as mitmproxy-ca-cert.pem, except that the suffix has been changed to apply to some Android platforms

Mitmproxy-dhparam.pem

Secret key file in PEM format to enhance SSL security

Windows

Double-click mitmproxy-ca.p12, and the boot page for importing the certificate appears, as shown in figure 1-61.

Just click the "next" button, and the password setting prompt will appear.

There is no need to set the password here, just click the "next" button.

Next, you need to select the storage area of the certificate. Click the second option, "put all certificates in the following storage".

Then click the Browse button and select the certificate store location as trusted Root Certificate Authority

Then click the OK button, and then click the next button.

Finally, if a security warning pops up, just click the "Yes" button.

This completes the configuration of the CA certificate under Windows.

I didn't even post the pictures. I don't feel necessary. They are basically the next step, stupid style.

Android

Just now, send the circle in my picture to the mobile phone to install, mitmproxy-ca-cert.cer. If this fails to install on the phone, send mitmproxy-ca-cert.pem to the phone to install.

There may be a safety warning. Just ignore it. Done.

Configure the proxy IP. Connect to the same local area network, add proxy IP and port to the settings of WiFi. Proxy IP is the IP of the computer, and the port is the port for packet capture and monitoring.

Cmd runs the code mitmdump-s decode_douyin_fans.py-p 8889 description: mitmdump-s py file-p simulator proxy port

You can also run mitmdump directly. The default port is 8080.

Please refer to this:

Import json

# the function name must be written in this place: response

Def response (flow):

# obtain the requested API by grabbing the software package

If 'aweme/v1/user/follower/list' in flow.request.url:

# data parsing

For user in json.loads (flow.response.text) ['followers']:

Douyin_info = {}

Douyin_info ['share_id'] = user [' uid']

Douyin_info ['douyin_id'] = user [' short_id']

Douyin_info ['nickname'] = user [' nickname']

Print ('fan list:', douyin_info,'-') Thank you for your reading. This is the content of "how python crawls Douyin app". After the study of this article, I believe you have a deeper understanding of how python crawls Douyin app, and the specific usage needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.