Shulou(Shulou.com)06/01 Report--

This article introduces you how to achieve secondary sorting in spark, the content is very detailed, interested friends can refer to it, I hope it can help you.

import org.apache.spark._ import SparkContext._ object SecondarySort { def main(args: Array[String]) { val sparkConf = new SparkConf().setAppName(" Secondary Sort ") sparkConf.set("mapreduce.framework.name", "yarn"); sparkConf.set("spark.rdd.compress", "true"); sparkConf.set("spark.serializer","org.apache.spark.serializer.KryoSerializer"); sparkConf.set("spark.storage.memoryFraction", "0.5"); sparkConf.set("spark.akka.frameSize", "100"); sparkConf.set("spark.default.parallelism", "1"); val sc = new SparkContext(sparkConf) val file = sc.textFile("hdfs://namenode:9000/test/secsortdata") val rdd = file.map(line => line.split("\t")). map(x => (x(0),x(1))).groupByKey(). sortByKey(true).map(x => (x._ 1,x._ 2.toList.sortWith(_>_))) val rdd2 = rdd.flatMap{ x => val len = x._ 2.length val array = new Array[(String,String)](len) for(i

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.