Shulou(Shulou.com)06/03 Report--

The main contents of this lesson are as follows:

1. Several ways to create RDD

2. RDD creates actual combat

3. Inside RDD

There are many ways to create RDD. Here are several ways to create RDD:

1. Use the collections in the program to create a RDD, which is actually used for testing.

2. Use the local file system to create RDD and test files with large amounts of data

3. Use HDFS to create RDD, the most commonly used way

4. Create RDD based on DB

5. Create RDD based on NoSQL, such as HBase

6. Create RDD based on S3

7. Create a RDD based on the data source

RDD actual combat:

/ / create a RDD through collection

Val conf = new SparkConf () .setAppName ("RDDDemo") .setMaster ("local")

Val sc = new SparkContext (conf)

/ / create a RDD

Val rdd = sc.parallelize (0 to 100)

/ / 1, 2, 3, 3, 3, 3, 3, 6, 4, 4, 10.

Val sum = rdd.reduce (_ + _)

Println (sum)

/ / create a RDD from a file on HDFS

Val conf = new SparkConf () .setAppName ("RDDDemo") .setMaster ("local")

Val sc = new SparkContext (conf)

/ / create a RDD

Val lines = sc.textFile ("hdfs://Master:9000/data/README.md")

Val words = lines.flatMap (line = > line.split ("")) .map (line = > (line, 1))

Val wordCount = words.reduceByKey (_ + _)

WordCount.collect () foreach (println)

Note:

Source: DT_ big data DreamWorks

For more private content, please follow the Wechat official account: DT_Spark

If you are interested in big data Spark, you can listen to the Spark permanent free open course offered by teacher Wang Jialin at 20:00 every evening, address YY room number: 68917580

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.