Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the shuffling process of Shuffle". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The normal meaning of Shuffle is to shuffle or mess up.

Shuffle describes the process of data output from map task to reduce task input. .

Most map task and reduce task execution is on different nodes. Of course, in many cases, Reduce needs to pull map task results on other nodes across nodes.

Our expectations of the Shuffle process can be:

Pull the data completely from the map task side to the reduce side. When pulling data across nodes, reduce the unnecessary consumption of bandwidth as much as possible. Reduce the impact of disk IO on task execution.

Shuffle interpretation

Each map task has a memory buffer that stores the output of map. When the buffer is almost full, the data of the buffer needs to be stored on disk as a temporary file. When the whole map task ends, all temporary files generated by this map task in the disk will be merged to generate the final official output file, and then wait for reduce task to pull the data.

1. When map task executes, its input data comes from HDFS's block. Of course, in the concept of MapReduce, map task only reads split. The correspondence between Split and block may be many-to-one, and the default is one-to-one.

2. After running the mapper class, we know that the output of mapper is such a KBE key value.

Yes. Here, only do statistics, reduce do mergers.

3.Partitioner interface, its function is to decide which reduce task should handle the current pair of output data according to the number of key or value and reduce. By default, the model is modeled by the number of key hash after reduce task. The default mode is only to average the processing power of reduce, if users have their own needs for Partitioner, they can be customized and set to job.set (..).

4 (Memory Buffer) next we write the data to the memory buffer, which is used to collect map results in bulk, reducing the impact of disk IO. Both our key/value pair and the result of Partition are written to the buffer. Of course, both key and value values are serialized into a byte array before writing.

5 memory buffers are limited in size, and the default is 100MB. When the output of map task is greater than the threshold of this memory buffer, it is (buffer size * spill percent = 100MB * 0.8 = 80MB)

The overflow thread starts to write the 80m data in the buffer to disk, and Map task to the remaining 20MB in memory, complementing each other. This process of writing data from memory to disk is called Spill.

When the overflow thread starts, you need to Sort the key in this 80MB space. Sorting is the default behavior of the MapReduce model, and sorting here is also the sort of serialized bytes.

You can also see on the figure that the overflow files written to disk are merged with different values on the reduce side. So an important detail of the overflow process is that if there are many key/value pairs that need to be sent to a reduce, then these key/ values need to be spliced together to reduce the number of index records associated with partition.

6. If client has set up Combiner, now is the time to use Combiner. Add up the key/value pairs with the same key to reduce the amount of data overwritten to disk. Combiner optimizes the intermediate results of MapReduce, so it is used multiple times throughout the model. In which scenarios can Combiner be used? From this analysis, the output of Combiner is the input of Reducer, and Combiner must not change the final calculation result. So from my point of view, Combiner should only be used in scenarios where the input key/value of Reduce is exactly the same as the output key/value type and does not affect the final result. Such as accumulation, maximum, etc. The use of Combiner must be careful. If used well, it will help the efficiency of job execution, otherwise it will affect the final result of reduce.

7. Each overflow will generate an overwrite file on the disk. When the Map output result is very large, there will be several such overwrites to the disk. When the Map task is finished, the data of the memory buffer will also be overwritten to the disk. As a result, there will be one or more overflowed files on the disk and merge the overflowed files at the same time. (if map outputs few results, only one file will overflow when map is complete.) the process of merging is called Merge {Ink}.

Merge merges multiple overflow files into one file, so the same key may exist, and in the process, if client has set Combiner, it will also use Combiner to merge the same key.

At this point, the work on the map side is finished, and the resulting files are also stored in the Task Tracker local directory. Each reduce task constantly obtains whether the map task is completed from the JOBTracker through RPC. If the reduce task is notified, the map task execution on a Task Tracker is notified, and the reducece of the shuffle begins to pull the completed data from the map Task.

Before Reducer actually runs, all the time is pulling data, doing merge, and doing it over and over again. Describe the Shuffle details on the reduce side in segments as before

The 1.copy process is to pull data. The Reduce process starts some copythreads and requests the TaskTracker where map task is located to get the output file of map task through Http. Since the map task is finished, the file is managed by TaskTracker, which is managed on the local disk.

The data from 2.copy is first stored in the memory buffer (Memory Buffer). The buffer here is more flexible than the one on the map side. It is based on the heap size setting of JVM. Because the rduce does not run in the shuffle phase, most of the memory is used by shuffle.

Here, just like the memory overflow in map, when the data in memory reaches a certain threshold, the memory overflow to disk will be initiated. Merge Merge. This process we set up Combiner, will also enable, and then generate a lot of files on disk. The value ends only when there is no data on the map side. Then start the third disk-to-disk merge mode to generate the final file.

The input file of 3.Reduce, after continuous merging (merge), will eventually generate a "final file", which may be stored on disk or in memory (memory needs to be set and optimized). The default is on disk, when the input file of Reducer is fixed, the whole Shuffle finally ends. Then there is the Reducer execution, putting the results on the HDFS.

This is the end of the content of "what is the shuffling process of Shuffle"? thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.