Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of Python3.8 to build Scrapy development environment, the content is detailed and easy to understand, the operation is simple and fast, and has a certain reference value. I believe you will gain something after reading this article on how to build Scrapy development environment in Python3.8. Let's take a look at it.

Environment

Virtualbox_v6.0.14 centos_v7.1908 python_v3.8.0 mongodb_v3.2.22 pip_v19.3.1 windows pycharm-2019.2.3 Enterprise Edition (Community Edition does not support remote connection linux development), need to activate and use the system with its own python2.7.5 without processing, tried pyenv installation 3.8.But when installing scrapy, it kept reporting errors, and finally uninstalled pyenv and re-compiled and installed manually.

Install centos

In order to ensure that the virtual machine and the host can access each other and communicate normally, the configuration mode of 2 virtual network cards is adopted.

In the virtualbox menu, set-> network, enable two virtual network cards (virtual machine enabled, figure 2 below). The connection mode of network card 1 is set to "network address translation nat" to ensure that the virtual machine accesses the external network and host, and the connection mode of network card 2 is set to "bridge network card" to ensure that the host can access the virtual machine.

When you install centos7 on a virtualbox virtual machine, it is best to configure and enable 2 network cards (centos system enabled) during the installation process.

The installation process is slight.

After starting the virtual machine, use # ip addr to view the network card usage and ip allocation of the centos system. Enp0s3 is the nat network card information, and ip is 10.0.2.15 by default. Enp0s8 is the network card information in bridging mode, and the default assigned ip is in the same IP address range as the host machine.

If you cannot see the ip information of a network card, it may be that the system did not enable the network card during installation. You can change the onboot parameter to yes in / etc/sysconfig/network-scripts/ifcfg-enp0s3 and / etc/sysconfig/network-scripts/ifcfg-enp0s8, and then restart the network service # systemctl start network.service or # service network restart.

The ssh service is available by default after the installation of the dvd image is completed. If the host uses ssh to connect to the virtual machine, you need to set the firewall to open port 22.

# firewall-cmd-zone=public-add-port=22/tcp-permanent # Firewall Open Port 22 # firewall-cmd-reload # reload Firewall configuration # firewall-cmd-zone=public-list-port # View Open Port

At this point, you can use the windows client tool ssh to remotely connect to the virtual machine to access the operation, mainly to facilitate sharing the clipboard with the host.

Configure yum domestic mirror source

For the use of Ali Yum source, please refer to the following paragraph

Download the wget tool # yum install-y wget and enter the folder where the yum source configuration file is located # cd / etc/yum.repos.d/ backup local yum source # mv centos-base.repo centos-base.repo_bak get Ali Yum source # wget-o centos-base.repo http://mirrors.aliyun.com/repo/centos-7.repo clean yum cache # yum clean all rebuild cache # yum makecache upgrade linux system # yum-y update# netstat-nltp # Check whether the port is occupied # ifconfig # View common commands of ip these commands are not installed by default You can execute the following command line installation # yum install net-tools install third-party dependencies

A big pile needs to be installed here, which is collected online and added by your own practice. If you don't want to go through all the trouble, you can follow it.

# yum-y groupinstall "development tools" # yum-y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libxml2-devel libxslt-devel libffi-devel install mongodb

Add mongodb domestic mirror source # vim / etc/yum.repos.d/mongodb.repo paste the following

[mongodb-org] name=mongodb repositorybaseurl= https://mirrors.tuna.tsinghua.edu.cn/mongodb/yum/el$releasever/gpgcheck=0enabled=1

Refresh the yum cache

# yum makecache

Install mongodb

# yum-y install mongodb-org modifies the mongod.conf bindip parameter # vim / etc/mongod.conf

Restart the mongod service

# / etc/init.d/mongod restart or # systemctl restart mongod.service

View mongod version

# mongod-version

Firewall opens port 27017

# firewall-cmd-zone=public-add-port=27017/tcp-permanent# firewall-cmd-reload# firewall-cmd-zone=public-list-port

Install python3.8

Download python3.8 source code compilation and installation

# wget https://www.python.org/ftp/python/3.8.0/python-3.8.0.tgz# tar-zxvf python-3.8.0.tgz#cd python-3.8.0.tgz

Check dependencies, generating makefile

#. / configure-prefix='/usr/local/python3'

Compile

# make

Installation

# make install

Make soft chain

# ln-s / usr/local/python3/bin/python3 / usr/bin/python3# ln-s / usr/local/python3/bin/pip3 / usr/bin/pip3

After python3.7,-with-ssl is not available. You can install successfully without adding-with-ssl when executing configure. If it is not successful, you can remove the following three line comments from the python-3.8.0/modules/setup file.

# _ ssl _ ssl.c\ #-duse_ssl-i$ (ssl) / include-i$ (ssl) / include/openssl\ #-l$ (ssl) / lib-lssl-lcrypto install scrapy

You can modify the pip source path first. In the pipfile file, you can change the url attribute under [source] to: url = "https://pypi.tuna.tsinghua.edu.cn/simple", or you can directly use the following command to perform the installation.

# pip3 install-- upgrade pip-I https://pypi.tuna.tsinghua.edu.cn/simple # specify the pip source update pip# pip3 install scrapy-I https://pypi.tuna.tsinghua.edu.cn/simple # of Tsinghua University install scrapy

A lot of dependent packages for scrapy will be installed next, and it will be very slow to install twisted, so be patient. The installation is successful when you see the picture below!

This is the end of this article on "Python3.8 's method of building a Scrapy development environment". Thank you for reading! I believe you all have a certain understanding of "Python3.8 's method of building a Scrapy development environment". If you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.