Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to understand the TextFile partition problem". In the daily operation, I believe many people have doubts about how to understand the TextFile partition problem. The editor consulted all kinds of data and sorted out a simple and easy-to-use operation method. I hope it will be helpful for you to answer the doubt of "how to understand the TextFile partition problem". Next, please follow the editor to study!

Val rdd1 = sc.parallelize (List (2, 3, 4, 1, 7, 5, 6, 9, 8)

Get the number of partitions: rdd1.partitions.length. If you don't specify the number of partitions in spark-shell, you can get the default number of partitions. In addition to this, the parallelize method can be used to specify how many partitions will appear.

Val rdd1 = sc.textFile ("hdfs://hadoop02:8020/word.txt", 3). FlatMap _ .split ('') .map ((_, 1)) .reduceByKey (_ + _)

The textFile method has a default value of 2 unless you change the default value in loacl, which is less than the default value as long as the default value is less than 2.

The default property is defaultMinPartitions with a value

If the number of partitions is passed in the textfile, then the number of partitions may be the same or different depending on the underlying calculation!

At this point, the study on "how to understand the TextFile partition problem" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.