Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the advantages of tungsten-sort". In daily operation, I believe many people have doubts about the advantages of tungsten-sort. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the advantages of tungsten-sort?" Next, please follow the editor to study!

Spark implements a variety of shuffle methods, which are determined by spark.shuffle.manager. For the time being, there are a total of three: hash shuffle, sort shuffle, and tungsten-sort shuffle, which defaults to sort shuffle since 1.2.0.

Spark can open Tungsten-sort shuffle via (spark.shuffle.manager = tungsten-sort) after 1. 4. If Tungsten-sort finds that it cannot handle it, it automatically uses Sort Based Shuffle for processing. The main optimization points of Tungsten-sort are:

Operate directly on serialized binary data, without deserialization, use the unsafe memory copy function to directly copy data.

Provide cache-efficient sorter ShuffleExternalSorter sort compressed record pointer and partition ids, using a 8bytes pointer to convert the sort into a pointer array sort.

There is no need for deserialization and serialization when spilling

The merge process of spill can be completed without deserialization, but it needs the support of shuffle.unsafe.fastMergeEnabled

The new Shuffle method is used only if and only if the following conditions are met:

Shuffle dependency cannot have aggregation or the output needs to be sorted

The serializer for Shuffle needs to be serialized by KryoSerializer or Spark SQL's. The number of * > Shuffle files cannot be greater than 16777216.

When serializing, a single record cannot be greater than 128 MB

# # advantages: many performance optimizations

# # disadvantages

Cannot sort on the mapper side

Unstable

No off-heap sort cache is provided

At this point, the study on "what are the advantages of tungsten-sort" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.