Shulou(Shulou.com)05/31 Report--

What is the computing model and execution principle of mapreduce? aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Mapreduce Computing Model and execution principle

MapReduce is a distributed computing model. MR consists of two stages: Map and Reduce. When it comes to stages, there will be first and then, first execute map and then execute reduce. Map is the beginning of our entire computing phase, and reduce is the end of our entire computing phase. From this point of view, the input of our map is the input of the whole system, the output of reduce is the output of the whole system, and the output of map is the input of reduce.

two。 The formal parameters of these two functions are key and value pairs, which represent the input information of the function. There is a map.entry in our map that gets the key-value pair.

First, the calculation model of mapreduce

Mapreduce's computing model, in this computing model, the code we wrote on the client side will be submitted to jobtracker, why submit it to it? because jobtracker accepts the computing tasks submitted by users, and then assigns the computing tasks to taskTracker to execute. In taskTracker, there are many map tasks and some reduce tasks. In this diagram, here is running map, here is running reduce,map, receiving input data, and after map processing, there is output. The output, uh, is the input of reduce, and the reduce is processed as the output. In the whole computing model, our input is the input of map, and our output is the output of reduce.

Note: here, there are two mapper (or there can be more than one). There is one reduce here. The number of map represents the number of map tasks. Here, map tasks and reduce tasks are independent java processes.

What is the relationship between the number of map and the number of reduce? look at the schematic diagram of Mapreduce.

-> look at the following picture

What is the relationship between the number of map and the number of reduce? look at the schematic diagram of Mapreduce.

From this figure, we can see that the input file is in the lower left corner, and its data will be sent to the map task to execute. Here, there are five map tasks. These five map tasks take different input data. Then this data will be sent to reduce. The number of reduce is 3. After reduce processing, it will be sent to our disk.

It should be noted that how do the five map tasks correspond to the three reduce tasks? let's take a look at the output of map. There are three dotted lines with three branches, and these three branches are sent to different shuffle. Then three different branches of each of the five map tasks are scattered into the impassable shuffle, that is to say, a shuffle will accept the output of five map, and the shuffle will send the results to the reduce task. The output of 5 map when a reduce accepts it. Let's think about this. The number of branches output by map is the same as the number of reduce tasks.

The answer to the question about mapreduce computing model and implementation principle is shared here. I hope the above content can be of some help to everyone. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.