Shulou(Shulou.com)06/01 Report--

This article mainly explains "how Python crawls the comment data of Weibo Big V", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "how Python crawls the comment data of Weibo Big V".

Theoretical chapter

Just imagine a question, if we want to grab the comment data of a Weibo Big V Weibo, how should we achieve it? The easiest way is to find the Weibo comment data interface, and then change the parameters to get the latest data and save it. First, look for the interface to grab comments from Weibo api, as shown in the following figure.

But unfortunately, the frequency of the interface is limited, it is banned in a few times, and it gets cool before it starts to take off.

Then click the parameters tab, and you can see that the parameters are as shown in the following figure:

Actual combat chapter

With the basics above, let's start to iterate the code and implement it using Python.

2. You need to bring cookie data with you when making a request. Weibo cookie is valid for a long time, which is enough to catch a comment from Weibo. Cookie data can be found in browser analysis tools.

4. In order to save the comment content, we need to remove the facial expressions from the comments and use regular expressions to process them, as shown in the following figure.

6. The key point is that a maximum of 16 pages of data can be returned through this API (20 entries per page). It is also said on the Internet that 50 pages can be returned, but the number of returned data items varies with different interfaces, so I added a for loop in one step. Traversal is still very powerful, as shown in the following figure.

8. Do de-reprocessing to the obtained data, as shown in the following figure. If the comment is already in it, just pass it. If not, continue to add it.

At this point, I believe you have a deeper understanding of "how Python crawled the comment data of Weibo Big V". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.