Shulou(Shulou.com)06/03 Report--

Through the previous study, you have learned about the HDFS file system. With the data, the next step is to analyze and calculate the data to generate value. Next we introduce the Mapreduce computing framework to learn how data is used.

Mapreduce computing framework

If you compare Hadoop to an elephant, then MapReduce is the elephant's computer. MapReduce is the core programming model of Hadoop. In Hadoop, the core of data processing is MapReduce programming model.

The content of this chapter:

1) MapReduce programming model

2) MapReduce execution process

3) MapReduce data localization

4) how MapReduce works

5) MapReduce error handling mechanism

1. MapReduce programming model

The concepts of Map and Reduce are borrowed from functional language. The whole MapReduce computing process is divided into Map stage and Reduce stage, also known as mapping and reduction stage. These two independent stages are actually two independent processes, namely Map process and Reduce process. Data are read and preprocessed in Map, and then the preprocessed results are sent to Reduce for merging.

We use a code case to quickly familiarize you with how to quickly implement our own MapReduce through code.

Example: distributed calculation of the number of occurrences of each word in an article, that is, WordCount.

1) create a map.py file and write the following code:

#! / usr/bin/env python

Import sys

Word_list = []

For line in sys.stdin:

Word_list = line.strip () .split ('')

If len (word_list)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.