Shulou(Shulou.com)06/03 Report--

Docker toolbox

Https://www.docker.com/products/docker-toolbox

Spark

Https://hub.docker.com/r/singularities/spark/~/dockerfile/

# start-hadoop-namenode

# hadoop fs-mkdir / user

# hadoop fs-mkdir / user/root/

# hadoop fs-put. / README.md / user/root

# start-spark

# start-spark worker [master]

# spark-shell

# spark-shell-master spark://a60b8c8f9653:7077

Scala > val lines = sc.textFile ("file:///usr/local/spark-2.1.0/README.md")

Scala > val lines = sc.textFile ("hdfs:///usr/local/spark-2.1.0/README.md")

Lines: org.apache.spark.rdd.RDD [String] = file:///usr/local/spark-2.1.0/README.md MapPartitionsRDD [1] at textFile at: 24

Scala > lines.count ()

Res0: Long = 104

Scala > lines.saveAsTextFile ("hdfs:///user/root/README2.md") / / Save to hdfs

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.