Shulou(Shulou.com)06/02 Report--

How to implement a scrapy crawler in Python? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

JAP also wrote some small reptiles before, in fact, those are crawler files. Before learning scrapy, we have to figure out the difference between the crawler file and the crawler project, in fact, it is very easy to understand, the crawler file is a single file to write the crawler, the crawler project is a large crawler, after all, it is called a project, we all know that the project can not only have a single file, it is composed of many files and there is a great connection between each file.

Scrapy is such a framework for writing crawler projects, how to play with it? Today, Mr. JAP is taking you step by step.

2 to fight a war, you must first have a weapon-the installation of the scrapy framework

In fact, installing scrapy is very simple, just a word. First, press the windows key + R at the same time, then enter cmd, and finally enter pip install scrapy. Cmd.

Because JAP uses windows10 development as simple as a few steps, but JAP also learned from the network that there are many holes in other system installations, but win10 installation may also have holes, of course, if there are holes, you can join our discussion group to solve them.

However, many friends still do not know whether the installation is successful or not. At this time, you still enter the scrapy command in the cmd window. If there is a response in the following picture, congratulations on your successful installation and have a good time.

3. If you have a weapon, you should know how to use it-scrapy's common instructions

After we have successfully installed scrapy, how to use it? I don't feel anything! Then we officially started to contact scrapy!

In fact, our scrapy operations are carried out through the command line, what commands are there?

1.scrapy-h (view all commands)

From here we can see most of the commands of scrapy, such as version above, and we can enter them as follows:

2.scrapy startproject spider_name (create a project project)

Let me make it clear here that before we use this command, we need to switch to a folder we created, and then enter this string of commands, and the project will be created in the corresponding folder.

You can take a look at my command input. I first switch to the folder of the scrapydemo I created, and then type scrapy startproject ceshi1, which is defined by everyone.

3.scrapy genspider name domain (build crawler)

I guess people are a little confused about domain. Domain is the domain name you want to crawl. Let's take a look at the next example.

You will notice that I use this command in our project directory, because we are going to build this crawler in this project. First of all, ceshi is our own defined name, baidu.com is the domain name we want to crawl, and you will probably wonder why it is not http://www.baidu.com? In fact, http is a protocol, and www belongs to the World wide Web and is not a part of the domain name, so here we can enter baidu.com directly.

As for the specific embodiment of this string of code, I will talk about it below.

4.scrapy crawl name (running crawler with log logs) and

Scrapy crawl name-nolog (running crawlers without log logs)

This command we will explain later, because this requires us to write a simple crawler, we just remember to run the crawler to use it two.

4 what is it like after the project is created?

With the command scrapy startproject ceshi1 above, we successfully built the first scrapy crawler project. Let's see what it is.

The ceshi1 folder is the same as the name we entered when we created it.

Scrapy.cfg: configuration file for the crawler project

We're digging into the ceshi1 folder.

First folder: needless to say, a cache folder

The second folder is where our crawler's code is located.

_ init__.py: the initialization file of the crawler project, which is used to initialize the project.

Items.py: the data container file of the crawler project, which is used to define the data to be obtained.

Pipelines.py: the pipeline file of the crawler project, which is used for further processing of the data in items.

Settings.py: the settings file for the crawler project, which contains the settings information for the crawler project.

Middlewares.py: the middleware file of the crawler project

Then we go deep into the spiders folder again

Here we find the ceshi.py file we generated through scrapy genspider ceshi baidu.com.

In fact, we can open this project with pycharm and write the project with pycharm. We will see it when we take you to practice a project next time.

This is the answer to the question about how to achieve a scrapy crawler in Python. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.