Shulou(Shulou.com)06/03 Report--

One: write the map class first

Import sysfor line in sys.stdin:line = line.strip () words = line.split () for word in words:print ('% s\ t% s'% (word, 1))

Two: write reduce class

Import syscurrent_word = Nonecurrent_count = 0word = Nonefor line in sys.stdin:line = line.strip () word, count = line.split try:count = int (count) except ValueError:continueif current_word = = word:current_count + = countelse:if current_word:print ('% s\ t% s'% (current_word,current_count)) current_count = countcurrent_word = wordif current_word = = word:print ('% s\ t% s'%))

Third, use hadoop Streaming to execute the content of Python.

Hadoop jar / home/hadoop/hadoop-2.6.0-cdh6.5.2/share/hadoop/tools/lib/hadoop-streaming-2.6.0-cdh6.5.2.jar-input / user/hadoop/aa.txt-output / user/hadoop/python_output-mapper "python mapper.py"-reducer "python reducer.py"-file mapper.py-file reducer.py

Description:

The input and output paths, which are themselves on the hdfs, do not require a special hdfs.

If you do not add quotation marks in the × × part, an error will be reported:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads (): subprocess failed with code 2

If you don't add the pink part, an error will be reported:

Error: java.lang.RuntimeException: Error in configuring object

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.