Shulou(Shulou.com)06/03 Report--

How to use Python web crawler to download the starting point novel, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

Today, I would like to share with you a novel crawling case-the novel download of the starting point novel.

Before we do this case, we need to analyze it.

1. Interface analysis, as shown in the figure:

Through the analysis, we can easily find our get request parameters, and then get the novel title and link of the corresponding page:

After getting the data, we randomly select a novel to download, and we choose the first one.

Then open its article directory, and you can see something like this, as shown in the figure:

Basically this novel is very long, you can see that volume one and volume two are free, and the fees are later, so today we will only climb the free chapters.

So let's now analyze the structure of the web page, as shown in the figure:

Then, we can first print out the name of volume one and the number of chapters, as well as the name of each chapter under the chapter.

First of all, we can analyze the web address, as shown in the figure:

Https://book.qidian.com/info/1014243481#Catalog

Found that the front has not changed, basically the latter has changed, adding an info/1014243481#Catalog, the following analysis:

Info: the meaning of information

1014243481: the corresponding ID of the novel

# Catalog: data completion does not make much sense

Since you have just crawled out the content of the article link, you only need to concatenate a # Catalog:

Next, we can make a request to it and then analyze its page. First, we can initiate a get request. According to the previous web page analysis structure, we should write:

As you can see, because there is asynchronous loading here, our requests will not be displayed all at once and require constant requests, of course, it is best to add a delay.

In this way, we get all the novels on this page, and we can also do this, because we are not looking for an interface, so forced parsing can only parse part of the content, but it is also very comprehensive. As shown in the figure:

The search is quite detailed, but it is not as standard and good-looking as the data obtained when looking for the interface.

This is the answer to the question about how to use Python web crawler to download the starting point novel. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.