Shulou(Shulou.com)06/03 Report--

Xiaobian to share with you what Shuffle is used for, I believe most people do not know much, so share this article for everyone's reference, I hope you have a lot of gains after reading this article, let's find out together!

Shuffle normally means shuffling or shuffling, and Shuffle describes the process of getting data from maptask output to reducetask input. partition, partition, partition

What does shuffle mean?

The main function of the shuffle phase is fetchOutputs(), which copies the output of the map phase locally to the reduce node.

Combine and partition are both functions, and the middle step should only be shuffle!

Combine is divided into map end and reduce end, the function is to combine the key-value pairs of the same key together, which can be customized.

Combine combines pairs (multiple keys, values) generated by a map function into a new. Take the new as input to the reduce function

This value2 can also be called values because there are multiple values. The purpose of this merger is to reduce network traffic.

What does shuffle mean?

partition is the result of partitioning each node of the map, mapped to different reductions according to the key, and can also be customized. This can actually be classified.

We classify complex data. For example, there are cattle, sheep, chickens, ducks and geese in the zoo. They are all mixed together, but at night they return to their cattle sheds, sheep pens and chickens. The function of partition is to classify the data. It's just that mapreduce uses hashPartitioner to help us classify when we write our programs. We can customize this as well.

Shuffle is the process between map and reduce, including the combination and partition at both ends.

The result of Map will be distributed to Reducer through partition. After Reducer completes the Reduce operation, it will output through OutputFormat.

The above is "Shuffle what to use" all the content of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.