How to install Scrapy Framework in python
This article mainly shows you "how to install the Scrapy framework in python", which is easy to understand and clear. I hope it can help you solve your doubts. Let the editor lead you to study and learn how to install the Scrapy framework in python.
Scrapy installation
The official website https://scrapy.org/
Installation mode
Under any operating system, you can install Scrapy using pip, for example:
$pip install scrapy
To confirm that Scrapy has been successfully installed, first test whether the Scrapy module can be imported in Python:
> import scrapy > scrapy.version_info (1,8,0)
Free online viewing of Python crawler, data analysis, website development and other case tutorials
Https://space.bilibili.com/523606542
Then, test whether the command Scrapy can be executed in shell:
(base) λ scrapy Scrapy 1.8.0-no active project Usage: scrapy [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [more] More commands available when run from project directory Use "scrapy-h" to see more info about a command
Passed the above two tests, indicating that the Scrapy installation is successful. As shown above, we are installing the latest version 1.8.0
Note:
Errors such as missing VC++ may be encountered in the process of installing Scrapy. Offline packages with missing modules can be installed.
After a successful installation, running scrapy under CMD is not really successful. Check whether the scrapy bench test is really successful. If there is no error, it means the installation is successful.
Specific Scrapy installation process reference: http://doc.scrapy.org/en/latest/intro/install.html##intro-install-platform-notes contains the installation methods of each platform
The global command $scrapy Scrapy 1.7.3-no active project Usage: scrapy [options] [args] Available commands: bench Run quick benchmark test # # tests the performance of the computer. Fetch Fetch a URL using the Scrapy downloader # # download the source code and display it genspider Generate new spider using pre-defined templates # # create a new spider file runspider Run a self-contained spider (without creating a project) # # this is different from starting the crawler through crawl Scrapy runspider crawler file name settings Get settings values # # get current configuration information shell Interactive scraping console # # enter scrapy interactive mode startproject Create new project # # to create a crawler project. Version Print Scrapy version view Open URL in browser, as seen by Scrapy # # download the document content of the web page and display the [more] More commands available when run from project directory Use "scrapy-h" to see more info about a command project command in the browser
Scrapy startproject projectname
Create a project
Scrapy genspider spidername domain
Create a crawler. After you have created the crawler project, you also need to create a crawler.
Scrapy crawl spidername
Run the crawler. Notice the directory in which the command is run.
These are all the contents of the article "how to install the Scrapy Framework in python". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!
