Shulou(Shulou.com)06/01 Report--

This article shows you how to use Fiddler to crawl APP comments, the content is concise and easy to understand, absolutely can make your eyes shine, through the detailed introduction of this article I hope you can gain something.

If you are not a football fan, you can also take a look and learn how to grab APP data. All right, let's cut the crap and start grabbing!

1. packet capture analysis request

Mobile phone grab package we can use Fiddler software to grab, if you don't know how to grab, you can see this article grab package software Fiddler to understand?

After the configuration is done, start to grab the package.

First find the article you want to crawl

Get the ball app screenshot

Article link: m.dongqiudi.com/article_share/896482.html

After configuring the capture package, click on the comments below to see

Comment screenshot

Screenshot of package capture

It's easy to find a comment request for an article, which is the following one.

You can see the requested link: api.dongqiudi.com/v2/article/896482/comment? sort=down&version=177 ，

The request method is GET, and the next step is easy to do. Let's look at the request to slide down to see more comments.

As you can see, there are two more parameters when turning down the page, but it is easy to know that the next parameter is a timestamp, and the pn parameter is the number of pages, starting from 0.

But how do you tell if all the comments have been crawled? We can look at the details of the data, below the json data format, in the figure below you can see that there is data in the next page, then this is easy, haha

After analysis, the code part is next.

2. code portions

This is the main part, crawling comments from the first comment link and finding the comment address on the next page to continue crawling. This is where the database is stored in mongodb.

The main crawling logic can be seen to be relatively simple, because there is no encryption parameter involved, but there is a problem, every time a request is made, sometimes it will return with the same comment, so we also need to simply de-duplicate in the database.

The following is the code section for warehousing and deduplication

The rest is gone.

3. View the resulting data

Because the data analysis is not familiar, so temporarily only the word cloud map.

You need to write data to text first.

The word cloud is:

It can be seen that last night the national football team lost a game, but also by a lot of people sprayed, but there are still a lot of people have been supporting, always cheer for the national football, inside also said, China and South Korea is a certain gap, and still a little big, lost is normal, there is no need to spray, besides, I think last night's game has been much better than the first game (the second time did not see), or there is progress, I am full of expectations for the future knockout of the national football team, I believe I can go further!

The above content is how to crawl APP comments with Fiddler. Have you learned knowledge or skills? If you want to learn more skills or enrich your knowledge reserves, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.