Shulou(Shulou.com)06/03 Report--

Because you need to use many third-party packages, such as numpy,pandas, and the production environment does not have access to the Internet, it is too troublesome to install a single package, so consider replacing the previously installed independent environment of python with the integrated development environment of Anaconda. Here are the specific steps: 1 install Anaconda3-5.0.1-Linux-x86_64.sh script 2 to establish a soft link to / opt/modules/anaconda3/bin/python3.6

Ln-s / opt/modules/anaconda3/bin/python / usr/bin/python3

3 modify root environment variable

/ root/.bashrc

/ root/.bash_profile

Modify the hdfs user environment variable because most cluster operations are done by hdfs users.

Su-hdfs

~ / .bashrc

Export PATH= "/ usr/bin:$PATH"

4 modify pyspark2 command

Vi / usr/bin/pyspark2

Modify the spark2-submit command

Vi / usr/bin/spark2-submit

Modify the variable PYSPARK_PYTHON

Several nodes need to be modified, and then the spark will be replaced with the new python, and all the commonly used packages will be available.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.