Shulou(Shulou.com)06/01 Report--

Centos7 how to install Scrapy, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Install the development package group and upgrade the operating system

# yum groupinstall "Development Tools"-y#yum update-y

Note:

1. If the python on your system is not above python2.7, please upgrade to the above python2.7 version (because Scrapy requires a version above python2.7)

# download python2.7

# wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2

# decompression

# tar-jxvf Python-2.7.3.tar.bz2 # cd Python-2.7.3

# installation

#. / configure#make all#make install#make clean#make distclean

# View python version

# / usr/local/bin/python2.7-V

# establish a soft connection to make the default python of the system point to python2.7

# mv / usr/bin/python / usr/bin/python2.6.6#ln-s / usr/local/bin/python2.7 / usr/bin/python

# after the system Python soft link points to the Python2.7 version, because yum is not compatible with Python2.7, yum does not work properly. We need to specify the Python version of yum.

Vim / usr/bin/yum

The header of the file

#! / usr/bin/python

Change to

#! / usr/bin/python2.6.6

2. It is strongly recommended to upgrade python2.7 and then install pip and setuptools. If you do not do so, there will be a lot of confused problems, which will make you sour until dawn!

3. If you are upgrading to python2.7, it is more likely that it will all be compiled and installed through python setup.py, including but not limited to these packages

Lxml,zope.interface,Twisted,characteristic,pyasn1-modules,service-identity,Scrapy

PS: I compiled and installed it from the beginning, and the most common problems are:

Error:command 'gcc' failed with exit status 1

Later, I found that if there is such a prompt is either a lack of devel package or a lack of a lib library file; what makes me laugh or cry is that the installation of Scrapy prompt is successful, but can not create a project, test samples can not run, and finally I decisively change the centos7!

The following are all operations on Centos 7. If you upgrade to python2.7, please make a detour.

2. Vim / etc/yum.repo/rpmforge.repo specifies rpmforge to install liffi-devel [if you do not specify a source, yum install liffi-devel will prompt that it cannot be found]. The method in the original blog article cannot be used, so you search for the solution and overwrite the file opened above with the following code

# Name: RPMforge RPM Repository for Red Hat Enterprise 5-dag#URL: http://rpmforge.net/[rpmforge]name = Red Hat Enterprise $releasever-RPMforge.net-dag#baseurl = http://apt.sw.be/redhat/el5/en/$basearch/dagmirrorlist = http://apt.sw.be/redhat/el5/en/mirrors-rpmforge#mirrorlist = file:///etc/yum.repos.d/mirrors-rpmforgeenabled = 1protect = 0gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-daggpgcheck = 1

Run the following command

Sudo rpm-- import http://apt.sw.be/RPM-GPG-KEY.dag.txtsudo yum install libffi-devel

Original solution: http://www.lxway.com/164125081.htm

Note: there is no need to install rpmforge first

Rpmforge is a combination of Dag, Dries, and other software packages. They provide more than 10000 software packages for CentOS. Rpmforge is not part of the redhat Linux product or CentOS, but it is designed for these Linux suites.

Note: because this installation source is not part of CentOS itself, to use rpmforge, you must first install the rpmforce Repository.

Acquisition method:

# 32-bit:

Wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.i386.rpm rpm-ivh rpmforge-release-0.5.1-1.el5.rf.i386.rpm

# 64 bit:

Wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

Installation:

Rpm-ivh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

After installation, generate under the / etc/yum.repos.d directory:

Mirrors-rpmforge-contains a series of mirror sites

Rpmforge.repo-configuration file for the yum source

Rpmforge-testing.repo-for testing.

Third, if audit is installed in the system, please remove it first, it will affect the installation of Scrapy.

# yum remove audit

Fourth, install the development packages required by Scarpy

# yum install-y python-devel openssl-devel libxslt-devel libxml2-devel

Install pip and setuptools

# yum install python-pip-y

Shows that there are no sources available:

This is because for spawned distributions like centos, their feeds sometimes lag behind in content updates, or sometimes extended feeds don't exist at all. So when you use yum to search python-pip, you will say that the package was not found. So in order to be able to install these packages, you need to install the extension source EPEL first. EPEL (http://fedoraproject.org/wiki/EPEL) is a project created by the Fedora community to provide high-quality software packages for RHEL and derivative distributions such as CentOS and Scientific Linux.

First install the epel extension source:

Sudo yum-y install epel-release

Then install python-pip

Sudo yum-y install python-pip

Don't forget to clear cache after installation.

Sudo yum clean all#pip install setuptools#pip install setuptoos-upgrade

6. Install Scrapy

# pip install Scrapy

Collecting Scrapy

Using cached Scrapy-1.0.3-py2-none-any.whl

Requirement already satisfied (use-- upgrade to upgrade): cssselect > = 0.9 in / usr/lib/python2.7/site-packages (from Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): queuelib in / usr/lib/python2.7/site-packages (from Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): pyOpenSSL in / usr/lib/python2.7/site-packages (from Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): w3lib > = 1.8.0 in / usr/lib/python2.7/site-packages (from Scrapy)

Collecting lxml (from Scrapy)

Using cached lxml-3.4.4.tar.gz

Collecting Twisted > = 10.0.0 (from Scrapy)

Using cached Twisted-15.4.0.tar.bz2

Requirement already satisfied (use-- upgrade to upgrade): six > = 1.5.2 in / usr/lib/python2.7/site-packages (from Scrapy)

Collecting service-identity (from Scrapy)

Using cached service_identity-14.0.0-py2.py3-none-any.whl

Requirement already satisfied (use-- upgrade to upgrade): cryptography > = 0.7 in / usr/lib64/python2.7/site-packages (from pyOpenSSL- > Scrapy)

Collecting zope.interface > = 3.6.0 (from Twisted > = 10.0.0-> Scrapy)

Using cached zope.interface-4.1.3.tar.gz

Collecting characteristic > = 14.0.0 (from service-identity- > Scrapy)

Using cached characteristic-14.3.0-py2.py3-none-any.whl

Collecting pyasn1-modules (from service-identity- > Scrapy)

Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl

Requirement already satisfied (use-- upgrade to upgrade): pyasn1 in / usr/lib/python2.7/site-packages (from service-identity- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): idna > = 2.0 in / usr/lib/python2.7/site-packages (from cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): setuptools in / usr/lib/python2.7/site-packages (from cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): enum34 in / usr/lib/python2.7/site-packages (from cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): ipaddress in / usr/lib/python2.7/site-packages (from cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): cffi > = 1.1.0 in / usr/lib64/python2.7/site-packages (from cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Requirement already satisfied (use-- upgrade to upgrade): pycparser in / usr/lib/python2.7/site-packages (from cffi > = 1.1.0-> cryptography > = 0.7-> pyOpenSSL- > Scrapy)

Installing collected packages: lxml, zope.interface, Twisted, characteristic, pyasn1-modules, service-identity, Scrapy

Running setup.py install for lxml

Running setup.py install for zope.interface

Running setup.py install for Twisted

Successfully installed Scrapy-1.0.3 Twisted-15.4.0 characteristic-14.3.0 lxml-3.4.4 pyasn1-modules-0.0.8 service-identity-14.0.0 zope.interface-4.1.3

7. Create a project

[root@localhost workspace]

# scrapy startproject tutorial

2015-10-15 21:54:24 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)

2015-10-15 21:54:24 [scrapy] INFO: Optional features available: ssl, http11

2015-10-15 21:54:24 [scrapy] INFO: Overridden settings: {}

New Scrapy project 'tutorial' created in:

/ workspace/tutorial

You can start your first spider with:

Cd tutorial

Scrapy genspider example example.com

VIII. Directory structure

[root@localhost workspace]

# tree

.

└── tutorial

├── scrapy.cfg

└── tutorial

├── _ _ init__.py

├── items.py

├── pipelines.py

├── settings.py

└── spiders

└── _ _ init__.py

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.