Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to install the Scrapy framework in python", which is easy to understand and clear. I hope it can help you solve your doubts. Let the editor lead you to study and learn how to install the Scrapy framework in python.

Scrapy installation

The official website https://scrapy.org/

Installation mode

Under any operating system, you can install Scrapy using pip, for example:

$pip install scrapy

To confirm that Scrapy has been successfully installed, first test whether the Scrapy module can be imported in Python:

> import scrapy > scrapy.version_info (1,8,0)

Free online viewing of Python crawler, data analysis, website development and other case tutorials

Https://space.bilibili.com/523606542

Then, test whether the command Scrapy can be executed in shell:

(base) λ scrapy Scrapy 1.8.0-no active project Usage: scrapy [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [more] More commands available when run from project directory Use "scrapy-h" to see more info about a command

Passed the above two tests, indicating that the Scrapy installation is successful. As shown above, we are installing the latest version 1.8.0

Note:

Errors such as missing VC++ may be encountered in the process of installing Scrapy. Offline packages with missing modules can be installed.

After a successful installation, running scrapy under CMD is not really successful. Check whether the scrapy bench test is really successful. If there is no error, it means the installation is successful.

Specific Scrapy installation process reference: http://doc.scrapy.org/en/latest/intro/install.html##intro-install-platform-notes contains the installation methods of each platform

The global command $scrapy Scrapy 1.7.3-no active project Usage: scrapy [options] [args] Available commands: bench Run quick benchmark test # # tests the performance of the computer. Fetch Fetch a URL using the Scrapy downloader # # download the source code and display it genspider Generate new spider using pre-defined templates # # create a new spider file runspider Run a self-contained spider (without creating a project) # # this is different from starting the crawler through crawl Scrapy runspider crawler file name settings Get settings values # # get current configuration information shell Interactive scraping console # # enter scrapy interactive mode startproject Create new project # # to create a crawler project. Version Print Scrapy version view Open URL in browser, as seen by Scrapy # # download the document content of the web page and display the [more] More commands available when run from project directory Use "scrapy-h" to see more info about a command project command in the browser

Scrapy startproject projectname

Create a project

Scrapy genspider spidername domain

Create a crawler. After you have created the crawler project, you also need to create a crawler.

Scrapy crawl spidername

Run the crawler. Notice the directory in which the command is run.

These are all the contents of the article "how to install the Scrapy Framework in python". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.