Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how python uses Xpath to grab JD.com net commodity information". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

HTML file is actually organized by a set of tags composed of angle brackets, each pair of angle brackets form a tag, there is an upper and lower relationship between the tags, forming a tag tree; XPath uses path expressions to select nodes in the XML document. Nodes are selected by following the path or step.

Jingdong net dog food commodity

First of all, enter the JD.com network, enter the goods you want to query, and send a web page request to the server. Here the editor still uses the keyword "dog food" as the search object, and then gets the latter string of URLs: https://search.jd.com/Search?keyword=%E7%8B%97%E7%B2%AE&enc=utf-8, in which the parameter means the keyword we entered, in this case the parameter represents "dog food". For details, you can refer to Python God to teach you to deal with JD.com commodity information with regular expressions. So, as long as you enter the parameter keyword and encode it, you can get the target URL. Then request the web page, get the response, and then use the bs4 selector for the next step of data collection.

The source code of some of the web pages for commodity information on JD.com 's official website is as follows:

The source code of dog food information on JD.com 's official website

If we take a closer look at the source code, we can find that the target information we need is under the tag, so then we are like peeling onions, layer by layer to get the information we want.

Usually the way of URL encoding is to convert the characters that need to be encoded into the form of% xx. Generally speaking, URL coding is based on UTF-8, and of course some are related to the browser platform. The quote method is provided in the urllib library of Python, which can encode the string of URL so that it can enter the corresponding web page.

Copy Xpath expressions online

Many friends find it difficult to write Xpath expressions, but it is not difficult to master the basic usage. Copy the Xpath expression online as shown in the figure above, and you can easily copy the Xpath expression. However, the Xpath expressions obtained by this method generally cannot be used in the program, and they are too long to be seen. So Xpath expressions usually have to be done by yourself.

Go to the code directly and use Xpath to extract the target information, such as product name, link, picture and price. The specific code is shown in the following figure:

Crawler code

Here, the editor tells you a Xpath expression matching technique. I have read several articles before, and the bosses have recommended that Xpath expressions use nested matching. In this example, the items is defined first, as follows:

Items = selector.xpath ('/ / li [@ class= "gl-item"]')

Then match the target information one by one from the web page through the range function, rather than directly copying the Xpath expression, as shown below:

Title = selector.xpath ('/ / div [@ class= "p-img"] / a') [I] .get ('title')

I hope all my friends can enter this pit less in the future.

The final effect is as follows:

Final effect picture

This is the end of the content of "how python uses Xpath to grab JD.com net commodity information". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.