Shulou(Shulou.com)06/03 Report--

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

1. Purpose of document writing

Anaconda is a Python distribution for scientific computing, which supports Linux, Mac and Windows systems. It provides the functions of package management and environment management, and can easily solve the problems of multi-version python coexistence, switching and various third-party package installation.

In the enterprise cluster without external network environment, by building the enterprise private Python source, you can use pip tools to easily install to solve the problem of dependent packages that do not exist in the Anaconda environment.

This document describes how to install Anaconda offline in a CDH cluster, build a Python private source, and use pip tools.

Content Overview

1. How to install Anaconda on a CDH cluster

two。 How to build python Private Source

The use of 3.pip tools

Test environment

1. Operating system: CentOS6.5

Version 5.12.1 for 2.CM and CDH

3. Adopt root user operation

Precondition

The 1.CDH cluster is running normally

The 2.Apache service is running normally

2.CDH cluster installation Anaconda

Install Anaconda on the CDH cluster using the Parcel package at the download address:

Https://repo.continuum.io/pkgs/misc/parcels/

Select the Parcel package according to your operating system version. This document operates under the CentOS6.5 version.

2.1 deploy Parcel package http access for Anaconda

1. Download the Parcel package for Anaconda at the following address

Https://repo.continuum.io/pkgs/misc/parcels/Anaconda-4.2.0-el6.parcelhttps://repo.continuum.io/pkgs/misc/parcels/Anaconda-4.2.0-el6.parcel.shahttps://repo.continuum.io/pkgs/misc/parcels/manifest.json

two。 Download the above three files to the / var/ × × w/html/anaconda4.2.0 directory of the server where HTTP is located.

[root@ip-172-31-6-148html] # mkdir-p anaconda4.2.0/ [root @ ip-172-31-6148 html] # cd anaconda4.2.0/ [root @ ip-172-31-6148 anaconda4.2.0] # lltotal 541416 root root Jun-1 root root 554396033 Jun 19 17:47 Anaconda-4.2.0-el6.parcel-rw-r--r-- 1 root root 41 Jul 7 19:43 Anaconda-4 .2.0-el6.parcel.sha-rw-r--r-- 1 root root 3890 Jul 7 19:43 json [root @ ip-172-31-6-148json] #

3. Verify that http is accessed properly

2.2 install Anaconda

1. Configure parcels packet access address for Anaconda

two。 Click to download, assign and activate

2.3 configure Anaconda environment variables

Anaconda is installed in the / opt/cloudera/parcels/Anaconda directory by default, and the environment variable of Anaconda is configured on all nodes in the cluster, as follows:

Edit the / etc/profile file with root or a user with sudo privileges, and add the following configuration export ANACONDA_HOME=/opt/cloudera/parcels/Anacondaexport PATH=$ANACONDA_HOME/bin:$PATH at the end of the file

two。 Execute the source command to make the environment variable take effect immediately

[root@ip-172-31-6-148anaconda4.2.0] # source / etc/profile

3. Verify that the environment variable is configured successfully

[root@ip-172-31-6-148anaconda4.2.0] # pythonPython 2.7.12 | Anaconda 4.2.0 (64-bit) | (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2Type "help", "copyright", "credits" or "license" for more information.Anaconda is brought to you by Continuum Analytics.Please check out: http://continuum.io/thanks and https://anaconda.org>>>

3. Build a private source for Pyton

Pre-preparation:

1. Download the pip2pi installation package at:

Https://codeload.github.com/wolever/pip2pi/zip/master

two。 Download the Python source code dependency package (tar.gz/zip/whl and other format installation packages)

3.1 install the pip2pi tool

Upload the downloaded pip2pi source package to the server and decompress [root@ip-172-31-6-148148] # unzip master

two。 Use the Python command to install pip2pi

[root@ip-172-31-6-148python] # cd pip2pi-master/ [root@ip-172-31-6-148MUR] # lltotal 44 libpip2pi-rw-r--r-- 1 root root 19 Mar 19 15:51 MANIFEST.in-rw- root root 2797 Mar 19 15:51 CHANGELOG.txtdrwxr-xr-x 2 root root 4096 Mar 19 15:51 libpip2pi-rw-r--r-- 1 root root 1649 Mar 19 15:51 README.rst-rwxr-xr-x 1 root root 84 Mar 19 15:51 README.rst-rwxr-xr-x 1 root root 84 Mar 19 15:51 run-tests-rw-r--r-- 1 root root 20 Mar 19 15:51 setup.cfg-rw-r--r-- 1 root root 1422 Mar 19 15:51 setup.pydrwxr-xr-x 5 root root 4096 Mar 19 15:51 tests-rw-r--r-- 1 root root 194 Mar 19 15:51 tox.ini [root@ip- 172-31-6-148 pip2pi-master] # python setup.py install

3. Test whether the pip2pi and dir2pi commands are installed successfully

3.2 deploy dependency packages

Under the / var/ × × × w/html directory of the http server, create a new python-packages directory [root@ip-172-31-6-148html] # mkdir python-packages [root@ip-172-31-6-148html] # pwd/var/ × × w / html [root @ ip-172-31-6-148html] # lltotal 32drwxr-xr-x 2 root root 4096 Sep 11 15:52 python-packages [root@ip-172-31-6-148html] #

two。 Upload the Python dependency package downloaded locally to the server's / var/ × × w/html/python-packages directory

[root@ip-172-31-6-148RW python-packages] # lltotal 1780 Sep RW pymongo-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl-rw-1 root root 2421 Sep 24 2016 bit_array-0.1.0.tar.gz-rw-r--r-- 1 root root 1254 Sep 12 07:18 enum-compat-0.0.2.tar.gz-rw-r--r-- 1 root root 339017 Sep 12 07:27 pymongo-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl-rw 1 root root 368294 Sep 12 07:25 pymongo-3.5.1-cp27-cp27mu-manylinux1_x86_64.whl-rw-r--r-- 1 root root 799601 Sep 12 07:04 Routes-1.12.3.tar.gz-rw-r--r-- 1 root root 30409 Apr 23 2016 sasl-0.2.1.tar.gz-rw-r--r-- 1 root root 29630 Apr 21 2016 six-1.10.0.tar.gz-rw -root 87832 Jan 14 2017 thrift-0.10.0.zip-rw-r--r-- 1 root root 3885 Jun 9 2016 thrift_sasl-0.2.1.tar.gz-rw-r--r-- 1 root root 140861 Dec 3 2016 ym-impyla-0.14.0.tar.gz

3. Use the dir2pi command to index the package

[root@ip-172-31-6-148python-packages] # dir2pi / var/ × × w/html/python-packages/ [root@ip-172-31-6-148python-packages] # lltotal 1784 RWKui bit_array-0.1.0.tar.gz-rw-r--r---1 root root 2421 Sep 24 2016 bit_array-0.1.0.tar.gz-rw-r--r-- 1 root root 1254 Sep 12 07:18 enum-compat-0.0.2.tar.gz-rw-r--r- -1 root root 339017 Sep 12 07:27 pymongo-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl-rw-r--r-- 1 root root 368294 Sep 12 07:25 pymongo-3.5.1-cp27-cp27mu-manylinux1_x86_64.whl-rw-r--r-- 1 root root 799601 Sep 12 07:04 Routes-1.12.3.tar.gz-rw-r--r-- 1 root root 30409 Apr 23 2016 sasl-0.2 .1.tar.gzdrwxr-xr-x 11 root root 4096 Sep 12 08:06 simple-rw-r--r-- 1 root root 29630 Apr 21 2016 six-1.10.0.tar.gz-rw-r--r-- 1 root root 87832 Jan 14 2017 thrift-0.10.0.zip-rw-r--r-- 1 root root 3885 Jun 9 2016 thrift_sasl-0.2.1.tar.gz-rw-r--r-- 1 root root 140861 Dec 3 2016 ym-impyla-0.14.0.tar.gz [root@ip-172-31-6-148 python-packages] #

4. Verify whether the Http is accessed properly and the access address

Http://ip-172-31-6-148.fayson.com/python-packages/simple/

At this point, the private source server for Python has been built.

Configure pip commands to use private sources

1. Create a .pip directory in the current user's home directory (that is, "~")

[root@ip-172-31-6-14814] # mkdir-p. Pip [root@ip-172-31-6-14814] # pwd/ root [root @ ip-172-31-6-148i] #

two。 Create a new pip.conf file in the ~ / .pip directory with the following contents

[root@ip-172-31-6-148pp] # cd .pip / [root@ip-172-31-6-148pip] # pwd/root/ .pip [ip-172-31-6-148pip] # vim pip.conf [global] index-url = http://ip-172-31-6-148.fayson.com/python-packages/simple[install]trusted-host = ip-172-31-6-148.fayson.com

3. Test whether the pip command downloads dependent packages from a private source

[root@ip-172-31-6-148.pip] # pip install bit-arrayCollecting bit-array Downloading http://ip-172-31-6-148.fayson.com/python-packages/simple/bit-array/bit-array-0.1.0.tar.gzBuilding wheels for collected packages: bit-array Running setup.py bdist_wheel for bit-array. Done Stored in directory: / root/.cache/pip/wheels/7a/7f/04/3bdc776730b06ce5bedd46f9023d263a32f82941e1365a054fSuccessfully built bit-arrayInstalling collected packages: bit-arraySuccessfully installed bit-array-0.1.0 [root@ip-172-31-6-148,pip] #

You can see that the annotated section above shows downloading the dependency package from the source that has just been built.

4.pip usage

1. Specify the version to install package

[root@ip-172-31-6-148C] pip install SomePackage # latest version [root@ip-172-31-6-148C] pip install SomePackage==1.0.4 # specific version [root@ip-172-31-6-148C] pip install 'SomePackage > = 1.0.4' # minimum version

two。 Batch installation of packages

Create a text file requirements.txt with the following contents

[root@ip-172-31-6-148c] # vim requirements.txtsixbit_arraythrift==0.10.0

Batch installation using pip

[root@ip-172-31-6-148100] # pip install-r requirements.txt Collecting six (from-r requirements.txt (line 1)) Collecting bit_array (from-r requirements.txt (line 2)) Collecting thrift==0.10.0 (from-r requirements.txt (line 3)) Downloading http://ip-172-31-6-148.fayson.com/python-packages/simple/thrift/thrift-0.10.0.zip (87kB) 100% | █ ██ | 92kB 58.1MB/s Building wheels for collected packages: thrift Running setup.py bdist_wheel for thrift. Done Stored in directory: / root/.cache/pip/wheels/21/53/2d/121a953df877103ba4f2fe9041be0d28d6979438c6dd7b2b15Successfully built thriftInstalling collected packages: six, bit-array, thriftSuccessfully installed bit-array-0.1.0 six-1.10.0 thrift-0.10.0 [root@ip-172-31-6-148c] #

3. Exports all package and versions installed in the current python environment to the specified file

[root@ip-172-31-6-148] # pip freeze > requirements.txt

4. Uninstall package

[root@ip-172-31-6-148c] # pip uninstall six

5. Lists all package installed in the current Python environment

[root@ip-172-31-6-148] # pip list-- format=columnsPackage Version-alabaster 0.7.9 Anaconda-clean 1.0 anaconda-client 1.5.1 anaconda-navigator 1.3.1 argcomplete 1.0.0 astroid 1 . 4.7 astropy 1.2.1 Babel 2.3.4 backports-abc 0.4

6. View the details of the installed package

[root@ip-172-31-6-148C] # pip show sixName: sixVersion: 1.10.0Summary: Python 2 and 3 compatibility utilitiesHome-page: http://pypi.python.org/pypi/six/Author: Benjamin PetersonAuthor-email: benjamin@python.orgLicense: MITLocation: / opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/site-packagesRequires: [root@ip-172-31-6-148C] #

5. Remarks

Package download mode

Use pip2pi tool for batch download through the official website manual download, official website download address: https://pypi.python.org/pypi

Install pip2pi tools locally, and you can download package in bulk through the pip2tgz command

Create a requirements.txt file locally with the following sixbit_arraythrift==0.10.0

Because pip2tgz completes the package download by passing parameters to the pip call, you can specify any parameter format that pip can recognize.

two。 Use the pip2tgz command for bulk download

[root@ip-172-31-6-148.pip] # pip2tgz / root/python/-r / root/requirements.txt

Note: the pip command uses a foreign source download package by default. If the foreign source is unstable, you can refer to Section 3.3 to configure the domestic source.

6. common problem

After building the Python private source, pip installed package Times error [root@ip-172-31-6-148.pip] # pip install sixCollecting six The repository located at ip-172-31-6-148.fayson.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with'--trusted-host ip-172-31-6-148.fayson.com. Could not find a version that satisfies the requirement six (from versions:) No matching distribution found for contacts [root @ ip-172-31-6-148.pip] #

Solution:

Add the following configuration to the pip.conf file:

Trusted-host = ip-172-31-6-148.fayson.com

Drunken whips are famous horses, and teenagers are so pompous! Lingnan Huan Xisha, under the vomiting liquor store! The best friend refuses to let go, the flower of data play!

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.