Shulou(Shulou.com)06/01 Report--

In this article, the editor introduces "how to use python Scrapy shell" in detail, the content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to use python Scrapy shell" can help you solve your doubts.

Foreword:

We want to use xpath, beautifulsoup, regular expressions, css selectors, etc., to extract the desired data in the crawler, but because scrapy is a heavy framework, we have to wait a period of time for each run, so it is troublesome to verify that our extraction rules are correct, so scrapy provides a shell. The test rules used for convenience are certainly not limited to this function.

Open Scrapy shell:

Enter the command line terminal, go to the directory where the scrapy project is located, then enter the virtual environment where the scrapy framework is located, type the command scrapy shell [Link], and you will enter the shell environment of scrapy. In this environment, you can use it in the same way as in the crawler's parse method.

Go to the directory where the scrapy project is located:

Cd / Volumes/development/Python_learn/PycharmProjects/scrapy_demo/bmw_img_demo/

Enter the virtual environment where the scrapy framework resides:

Source / Volumes/development/Python_learn/PycharmProjects/venv/crawler_evn/bin/activate

Enter the command scrapy shell [link]:

Rapy shell https://car.autohome.com.cn/pic/series/66.html

Enter the statement we need to test:

Srcs = response.xpath ('/ / div [contains (@ class, "uibox-con")] / ul/li/a/img/@src') .getall ()

After reading this, the article "how to use python Scrapy shell" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it yourself. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.