Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to customize the partition in spark, many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

The function is to write to different files according to the last digit of the key value.

For example:

10 write to part-00000

11 write to part-00001

.

.

.

19 write to part-00009

Provide readers with the idea of a custom partition.

Import org.apache.spark. {Partitioner, SparkContext, SparkConf} / / Custom partition class To inherit the Partitioner class class UsridPartitioner (numParts:Int) extends Partitioner {/ / overlay partition number override def numPartitions: Int = numParts / / override partition number acquisition function override def getPartition (key: Any): Int = {key.toString.toInt}} object Test {def main (args: Array [String]) {val conf=new SparkConf () val sc=new SparkContext (conf) / / data simulating 5 partitions val data=sc.parallelize (1 to) 10) / / convert to 10 partitions according to the trailing number Divided into 10 files data.map ((_, 1)) .partitionBy (new UsridPartitioner (10)). SaveAsTextFile ("/ chenm/partition")}} after reading the above, do you have any further understanding of how to customize partitions in spark? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.