Shulou(Shulou.com)06/03 Report--

Create a spark user group, group ID1000

Groupadd-g 1000 spark

For spark users who create user ID 2000 under the spark user group to obtain Chinese documents and complete videos, please add QQ group: 947967114

Useradd-u 2000-g spark spark

Set password

Passwd spark

Modify sudo permissions

Chmod Ubunw / etc/sudoers

Vi / etc/sudoers

find

Root ALL= (ALL) ALL

Add

Spark ALL= (ALL) ALL

Create an app directory to store the spark software environment (jdk, scala, spark)

Mkdir / app

Modify the group and owner of this file

Chown-R spark:spark / app

Create soft

Mkdir / app/soft

Create spark

Mkdir / app/spark

Create / spark/work

Mkdir-p / home/spark/work

Change / spark/work group and owner

Chown-R spark:spark / home/spark/work

Switch users

Su root

Decompress JDK

Cd / tmp/

Tar zxvf jdk-8u192-linux-x64.tar.gz-C / app/soft/

If you don't have permission, use chmod 777-R / tmp to modify permission first.

Cd / app/soft/

Ll-a

Configuration / etc/profile

Sudo vi / etc/profile, all required configurations have been added

JAVA_HOME=/app/soft/jdk1.8.0_192

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

Export PATH

To make the configuration effective, please add QQ group to get the documents and complete videos in the video. Please add QQ group: 947967114

Source / etc/profile

Install scala:

Tar zxvf / tmp/scala-2.11.12.tgz-C / app/soft/

Configure environment variables

Sudo vi / etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

Export PATH

Configure ssh unsecret login

Ssh-keygen-t rsa

Cd ~ /

Cd .ssh /

Modify the name of the public key

Master node: mv id_rsa.pub authorized_keys_master.pub

Slave1 node: mv id_rsa.pub authorized_keys_slave1.pub

Slave2 node: mv id_rsa.pub authorized_keys_slave2.pub

Give the public keys of slave1 and slave2 to master

Slave1 node: scp authorized_keys_slave1.pub spark@master:/home/spark/.ssh/

Slave2 node: scp authorized_keys_slave2.pub spark@master:/home/spark/.ssh/

Write the public keys of all three nodes in one file

Cat authorized_keys_master.pub > > authorized_keys

Cat authorized_keys_slave1.pub > > authorized_keys

Cat authorized_keys_slave2.pub > > authorized_keys

Check the total public key file

Vi authorized_keys

Give the total public key file authorized_keys to the slave1 and slave2 nodes

Scp authorized_keys spark@slave1:/home/spark/.ssh

Scp authorized_keys spark@slave2:/home/spark/.ssh

Modify the operation permission of authorized_keys. All three nodes need to be modified.

Chmod 400 authorized_keys

Verify that secret-free login is successful

Ssh master

Ssh slave1

Ssh slave2

Ssh master

Install spark:

Tar-zxf / tmp/spark-2.1.0-bin-hadoop2.6.gz-C / app/spark/

Cd / app/spark/

Ls

Cd spark-2.1.0-bin-hadoop2.6/

Configure environment variables:

Vi / etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

SPARK_HOME=/app/spark/spark-2.1.0-bin-hadoop2.6

PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

Export PATH

Configure the core file for spark:

Cd spark-2.1.0-bin-hadoop2.6/

Cd conf/

Configure slaves

Mv slaves.template slaves

Vi slaves add three nodes

Master

Slave1

Slave2

Configure spark-env.sh

Cp spark-env.sh.template spark-env.sh

Vi spark-env.sh

Export JAVA_HOME=/app/soft/jdk1.8.0_192

Export SCALA_HOME=/app/soft/scala-2.11.12

Export SPARK_MASTER_IP=master

Export SPARK_MASTER_PORT=7077

Export SPARK_EXECUTOR_INSTANCES=1

Export SPARK_WORKER_INSTANCES=1

Export SPARK_WORKER_CORES=1

Export SPARK_WORKER_MEMORY=1024M

Export SPARK_MASTER_WEBUI=8080

Export SPARK_CONF_DIR=/app/spark/spark-2.1.0-bin-hadoop2.6/conf/

Change the work and soft permissions of app on all nodes to 777: execute chmod 777-R / app/soft and chmod 777-R / app/spark on all nodes

Scp-r / app/spark/ spark@slave1:/app/

Scp-r / app/soft/ spark@slave1:/app/

At this point, the spark cluster has been built:

Open: start-all.sh to get the video documents and complete video partners, please add QQ group: 947967114

Jps can see the following process:

Master node:

3617 Worker

3507 Master

4156 Jps

Slave1 node:

3361 Worker

3702 Jps

Slave2 node:

3319 Worker

3647 Jps

Enable spark-shell authentication:

Spark-shell-- master spark://master:7077-- executor-memory 1024m-- driver-memory 1024m

After startup, the following content is displayed:

18-11-29 16:13:46 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

18-11-29 16:13:47 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as' sc' (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as' spark'.

Welcome to

_ /

/ /. / _, / _\ version 2.1.0

/ _ /

Using Scala version 2.11.8 (Java HotSpot (TM) 64-Bit Server VM, Java 1.8.0 / 192)

Type in expressions to have them evaluated.

Type: help for more information.

Scala >

You can write the spark code after >:

G NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as' sc' (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as' spark'.

Welcome to

_ /

/ /. / _, / _\ version 2.1.0

/ _ /

Using Scala version 2.11.8 (Java HotSpot (TM) 64-Bit Server VM, Java 1.8.0 / 192)

Type in expressions to have them evaluated.

Type: help for more information.

Scala > sc.textFile ("/ app/spark/spark-2.1.0-bin-hadoop2.6/README.md"). FlatMap (.split (")) .map (x = > (xmem1)). ReduceByKey (+ _) .map (x = > (x.Zong2jue x.train1)) .sortByKey (false) .map (x = > (x.split (") .take (10)

Res0: Array [(String, Int)] = Array ((", 71), (the,24), (to,17), (Spark,16), (for,12), (and,9), (# #, 8), (can,7), (run,7)

Scala > Please add QQ group to get the documents in the video and the complete video: 947967114.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.