Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to achieve matrix multiplication in mapreduce. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Suppose, then

Matrix multiplication requires left matrix

Is equal to the number of rows of the right matrix, and

The matrix is multiplied by the matrix of the

Now let's analyze which operations are independent of each other (so that distributed computing can be done). Obviously, the calculation and the calculation do not interfere with each other; in fact, the calculations of the various elements in the are independent of each other. In this way, in the Map phase, we can centralize all the elements needed for calculation into the same key, and then, in the Reduce phase, we can parse out each element to calculate the other elements in the same way.

We also need to be aware that it will be,. If it is used in the calculation, it will be... Is used in the calculation of. That is to say, in the Map phase, when we fetch a row of records from HDFS, if the record is an element of, it needs to be stored as a pair, and the key is different from each other; if the record is an element of, it needs to be stored as a pair, similarly, the key should also be different from each other; but at the same time, it is used for calculation, storage,. And,. The correct key should all be the same so that it can be passed to the same Reduce.

After the above analysis, the whole calculation process is designed as follows:

(1) in the Map phase, the elements from the table are identified as strips. The elements from the table are identified as bars, where.

So, in the Map phase, we achieved this tactical goal: through key, we grouped the data involved in the calculation into one category. Through value, we can tell whether the element comes from or from, as well as the specific location.

(2) in the Shuffle phase, the value of the same key will be added to the same list, formed a pair, and passed to Reduce, which is done automatically by Hadoop.

(3) in the Reduce phase, there are two questions you need to ask yourself:

Which element is the current pair for calculation?

Where or where does each value in the list come from?

The first problem can be learned from key because we have constructed key into form during the Map phase. The second question can also be read directly in value, because we also marked it in the Map phase.

What we need to do next is to parse out the elements from list (value), put them separately in one array, and put the elements from them in another array, and then we calculate the dot product of the two arrays (each as a vector), which can be calculated.

The example matrix and multiplication are calculated as shown in the following figure:

Where the map phase:

Public static class MatrixMap extends Mapper {private int sum3 = 0; / / number of rows of matrix a private int sum4 = 0; / / number of rows of matrix b @ Override protected void map (Object key, Text value, Context context) throws IOException, InterruptedException {String [] lines = value.toString () .split (",") String url = ((FileSplit) context.getInputSplit ()) .getPath () .toString (); / / fetch the file path name and judge the data source to increase the number of rows if (url.contains ("matrix3")) sum3++; if (url.contains ("matrix4")) sum4++; for (int m = 0; m < lines.length) Data output if (url.contains ("matrix3") & & lines.length = = matrix_j) {for (int I = 1; I)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.