Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to use spark Context to convert RDD". In daily operation, I believe many people have doubts about how to use spark Context to convert to RDD. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about how to use spark Context to convert to RDD! Next, please follow the editor to study!

one。 Background

Among the spark rdd transformation operators, join and cogroup are some operator transformations that need to be distinguished, which are illustrated by an example here.

two。 Example 1. Build List sample data List studentsList = Arrays.asList (new Tuple2 (1, "xufengnian"), new Tuple2 (2, "xuyao"), new Tuple2 (2, "wangchudong"), new Tuple2 (3, "laohuang")) List scoresList = Arrays.asList (new Tuple2 (1100), new Tuple2 (2jue 90), new Tuple2 (3jue 80), new Tuple2 (1101), new Tuple2 (2jue 91), new Tuple2 (3jue 81), new Tuple2 (3jue 71); 2. Use sparkContext to convert RDDJavaPairRDD studentsRDD = sc.parallelizePairs (studentsList); JavaPairRDD scoresRDD = sc.parallelizePairs (scoresList); / / studentsRDD is: List// (1meme xufengnian) (2meme xuyao) (2recordable Wangchudong) (3memogic), the following is printed to view studentsRDD.foreach (new VoidFunction () {public void call (Tuple2 tuple) {System.out.println (tuple._1); / 123 System.out.println (tuple._2); / / xufengnian xuyao laohuang}}); 3. After the data in front of join/*: (1, (xufengnian,100)) (1, (xufengnian,101)) (3, (laohuang,81)) (3, (laohuang,71)) (2, (laohuang,71)) (2, (xuyao,90)) (2, (xuyao,90)) (2, (xuyao,91)) (2, (wangchudong,90)) (2 (wangchudong,91)) * / JavaPairRDD studentScores = studentsRDD.join (scoresRDD) / join is the same join,key of key, and value becomes (string,integer) studentScores.foreach (new VoidFunction () {private static final long serialVersionUID = 1L; @ Override public void call (Tuple2 student) throws Exception {System.out.println ("student id:" + student._1); / / 11 3 System.out.println ("student name:" + student._2._1) / / xufengnian xufengnian laohuang System.out.println ("student score:" + student._2._2); / / 100 101 80 System.out.println ("=");}}); 4. After performing the data in front of the cogroup/* (1 xufengnian xufengnian) (2 meme xuyao) (2, "wangchudong") (3 recollections) (1100) (2 xuyao,wangchudong 90) (3) 80) (1101) (2 xufengnian 91) (3) 81) (3) cogroup: (1, ([xufengnian], [100101])) (3, ([laohuang], [801 # 71])) (2, ([xuyao,wangchudong], [90jue 91])) * / JavaPairRDD studentScores2 = studentsRDD.cogroup (scoresRDD) StudentScores2.foreach (new VoidFunction () {@ Override public void call (Tuple2 stu) throws Exception {System.out.println ("stu id:" + stu._1); / / 1 3 System.out.println ("stu name:" + stu._2._1); / / [xufengnian] [laohuang] System.out.println ("stu score:" + stu._2._2); / / [100101] [80 Magi 81J 71] Iterable integers = stu._2._2 For (Iterator iter = integers.iterator (); iter.hasNext ();) {Integer str = (Integer) iter.next (); System.out.println (str); / / 100101 8081 71} System.out.println ("= =");}}); at this point, the study on "how to use spark Context to RDD" is over, hoping to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.