Lesson 15: thoroughly decrypting the inside story of RDD creation
The main contents of this lesson are as follows:
1. Several ways to create RDD
2. RDD creates actual combat
3. Inside RDD
There are many ways to create RDD. Here are several ways to create RDD:
1. Use the collections in the program to create a RDD, which is actually used for testing.
2. Use the local file system to create RDD and test files with large amounts of data
3. Use HDFS to create RDD, the most commonly used way
4. Create RDD based on DB
5. Create RDD based on NoSQL, such as HBase
6. Create RDD based on S3
7. Create a RDD based on the data source
RDD actual combat:
/ / create a RDD through collection
Val conf = new SparkConf () .setAppName ("RDDDemo") .setMaster ("local")
Val sc = new SparkContext (conf)
/ / create a RDD
Val rdd = sc.parallelize (0 to 100)
/ / 1, 2, 3, 3, 3, 3, 3, 6, 4, 4, 10.
Val sum = rdd.reduce (_ + _)
Println (sum)
/ / create a RDD from a file on HDFS
Val conf = new SparkConf () .setAppName ("RDDDemo") .setMaster ("local")
Val sc = new SparkContext (conf)
/ / create a RDD
Val lines = sc.textFile ("hdfs://Master:9000/data/README.md")
Val words = lines.flatMap (line = > line.split ("")) .map (line = > (line, 1))
Val wordCount = words.reduceByKey (_ + _)
WordCount.collect () foreach (println)
Note:
Source: DT_ big data DreamWorks
For more private content, please follow the Wechat official account: DT_Spark
If you are interested in big data Spark, you can listen to the Spark permanent free open course offered by teacher Wang Jialin at 20:00 every evening, address YY room number: 68917580
