How to understand MapReduce by matrix multiplication

Today I will show you how to understand MapReduce by matrix multiplication. The content of the article is good. Now I would like to share it with you. Friends who feel in need can understand it. I hope it will be helpful to you. Let's read it along with the editor's ideas.

(1)

Matrix multiplication to understand MapReduce

Requires the use of a computer to calculate matrix multiplication (the size of two matrices exceeds the size of the computer memory)

two。 Because matrix multiplication refers to rows and columns, the first row of the first matrix can be recorded as A1 and the first column of another matrix as B1, and so on. Each is pushed to a server to perform row-column product (which corresponds to Map in MapReduce) if the size of this matrix is 100 rows * 100 columns, then we need 100 machines to perform the computational product of each row and column in parallel. As shown below:

3. After each computer has finished processing, it will summarize the calculation results of each machine, and finally output the results. (this corresponds to Reduce in MapReduce).

(2)

Usage scenario and analysis

From the above calculation process, MapReduce adopts a series of routines. First, a large calculation process is divided into a small calculation process and the calculation results of each process are calculated. Finally, the calculation results are combined to get a complete calculation result. It can not only solve the problem of single machine configuration, but also can be calculated in parallel to speed up the computing speed.

MapReduce plays a very important role in data analysis. For example, we need to analyze a user's access to a certain URL. Without MapReduce, we need to read continuously in series, and it will be very complicated to use multithreading after continuous cyclic analysis or optimization. But with the MapReduce model, we can split the data into separate parallel analysis and processing, and finally summarize. It is especially suitable for use scenarios with large amount of data, high repeatability and relatively small results after summary.

(III)

From this we can see that the truth of MapReduce is very simple, mainly using the idea of divide and conquer.

I believe the editor will add more knowledge to you. I hope you can support it!