Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the stages of hash shuffle development". In daily operation, I believe many people have doubts about the development stage of hash shuffle. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the stages of hash shuffle development?" Next, please follow the editor to study!

Spark implements a variety of shuffle methods, which are determined by spark.shuffle.manager. For the time being, there are a total of three: hash shuffle, sort shuffle, and tungsten-sort shuffle, which defaults to sort shuffle since 1.2.0.

Spark defaults to hash shuffle (spark.shuffle.manager = hash) before 1.2, but hash shuffle has also gone through two stages of development.

# # Phase I

The figure above has four ShuffleMapTask to run on the same worker node. The number of CPU core is 2, and you can run two task at the same time. The execution result of each task (the records contained by a partition in the finalRDD of the stage) is written to the local disk one by one. Each task contains R buffers, R = the number of reducer (that is, the number of task in the next stage), the buffer is called bucket, its size is spark.shuffle.file.buffer.kb, and the default is 32KB (100KB before Spark 1.1).

# # the implementation of the second phase is very simple, but there are several problems:

1 produces too much FileSegment. Each ShuffleMapTask produces R (number of reducer) FileSegment,M ShuffleMapTask will produce M * R files. Generally speaking, the M and R of Spark job are very large, so there are a large number of data files on disk.

2 the buffer takes up a large amount of memory space. Each ShuffleMapTask needs R bucket,M ShuffleMapTask to generate M * R bucket. Although the corresponding buffer can be reclaimed after the end of a ShuffleMapTask, the number of bucket co-existing on a worker node can reach cores R (usually worker can run cores ShuffleMapTask at the same time), which takes up to cores * R * 32 KB. For 1000 reducer with 8 cores, memory footprint is 256MB.

Spark.shuffle.consolidateFiles defaults to false, and output files can be merged if it is true,shuffleMapTask. As shown in the picture

It is obvious that ShuffleMapTasks executed sequentially on a core can share a single output file, ShuffleFile. The ShuffleMapTask that is executed first forms the ShuffleBlock I, and then the ShuffleMapTask that executes can append the output data directly to the ShuffleBlock I to form a ShuffleBlock I, and each ShuffleBlock is called FileSegment. The reducer of the next stage only needs to fetch the entire ShuffleFile. In this way, the number of files held per worker is reduced to cores * R. However, the large amount of cache space has not been solved.

At this point, the study of "what are the stages of hash shuffle development" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.