Shulou(Shulou.com)06/01 Report--

This article mainly introduces python how to crawl Meituan website store information, the article introduces in great detail, has a certain reference value, interested friends must read it!

We have got the information about the streets and districts and counties, and then we have the information about the shops under each street and analyze the page.

At present, it is to obtain this information, and then continue to save it in the database.

Press F12PowerF5 to refresh and analyze the response data.

The following figure is the response data in response. Analyze and find out the data you need.

Get the url of the street from the database, access, analyze the response data, and extract the data from the script. The response data in response can be written as HTML first, and then formatted in the tool, so that it is clear and easy to analyze (the figure is the details, not the brief information of the store).

Save the data, put it into item, and the scrapy framework will automatically save the data back to a generator through the pipeline py file.

To judge whether there is the next page, the data in Meituan, the data on one page is 32.

If there is a next page, the url is re-spliced, and the current page can be retrieved in the script. The url starting from the second page needs to save the original url in the request.meta parameter each time.

Continue to send requests and download data after splicing

The above is all the contents of the article "how python crawls the shop information on Meituan's website". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.