Shulou(Shulou.com)06/03 Report--

0. Preface

The previous article "initial experience of Hadoop: quickly building a pseudo-distributed environment for Hadoop" built a Hadoop environment, and now use the wordcount program that comes with Hadoop to do word statistics.

1. Use the sample program to realize word statistics

(1) wordcount program

The wordcount program is located in the share directory of hadoop as follows:

[root@leaf mapreduce] # pwd/usr/local/hadoop/share/hadoop/mapreduce [root@leaf mapreduce] # lshadoop-mapreduce-client-app-2.6.5.jar hadoop-mapreduce-client-jobclient-2.6.5-tests.jarhadoop-mapreduce-client-common-2.6.5.jar hadoop-mapreduce-client-shuffle-2.6.5.jarhadoop-mapreduce-client-core-2.6.5.jar hadoop-mapreduce-examples-2.6 .5. Jarhadoop-mapreduce-client-hs-2.6.5.jar libhadoop-mapreduce-client-hs-plugins-2.6.5.jar lib-exampleshadoop-mapreduce-client-jobclient-2.6.5.jar sources

This is the hadoop-mapreduce-examples-2.6.5.jar program.

(2) create HDFS data directory

Create a directory to save the input file for the MapReduce task:

[root@leaf] # hadoop fs-mkdir-p / data/wordcount

Create a directory to save the output file of the MapReduce task:

[root@leaf ~] # hadoop fs-mkdir / output

View the two directories you just created:

[root@leaf] # hadoop fs-ls / drwxr-xr-x-root supergroup 0 2017-09-01 20:34 / datadrwxr-xr-x-root supergroup 0 2017-09-01 20:35 / output

(3) create a word file and upload it to HDFS

The word file created is as follows:

[root@leaf ~] # cat myword.txt leaf yyhyyh xpleafkaty lingyeyonghao leafxpleaf katy

Upload the file to HDFS:

[root@leaf ~] # hadoop fs-put myword.txt / data/wordcount

View the newly uploaded file and its contents in HDFS:

[root@leaf ~] # hadoop fs-ls / data/wordcount-rw-r--r-- 1 root supergroup 57 2017-09-01 20:40 / data/wordcount/myword.txt [root@leaf ~] # hadoop fs-cat / data/wordcount/myword.txtleaf yyhyyh xpleafkaty lingyeyonghao leafxpleaf katy

(4) run wordcount program

Execute the following command:

[root@leaf ~] # hadoop jar / usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount / data/wordcount / output/wordcount...17/09/01 20:48:14 INFO mapreduce.Job: Job job_local1719603087_0001 completed successfully17/09/01 20:48:14 INFO mapreduce.Job: Counters: 38 File System Counters FILE: Number of bytes read=585940 FILE: Number of bytes written=1099502 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=114 HDFS: Number of bytes written=48 HDFS: Number of read operations=15 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map Input records=5 Map output records=10 Map output bytes=97 Map output materialized bytes=78 Input split bytes=112 Combine input records=10 Combine output records=6 Reduce input groups=6 Reduce shuffle bytes=78 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps = 1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms) = 92 CPU time spent (ms) = 0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes) = 241049600 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=57 File Output Format Counters Bytes Written=48

(5) check the statistical results

As follows:

[root@leaf ~] # hadoop fs-cat / output/wordcount/part-r-00000katy 2leaf 2ling 1xpleaf 2yeyonghao 1yyh 2

3. references

Http://www.aboutyun.com/thread-7713-1-1.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.