How to write the MapReduce Program of Hadoop with PHP and Shell

This article mainly explains "how to write Hadoop MapReduce program with PHP and Shell". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "how to write Hadoop MapReduce programs with PHP and Shell"!

Any executable program that supports standard IO (stdin, stdout) can become the mapper or reducer of hadoop. For example:

The copy code is as follows:

Hadoop jar hadoop-streaming.jar-input SOME_INPUT_DIR_OR_FILE-output SOME_OUTPUT_DIR-mapper / bin/cat-reducer / usr/bin/wc

In this example, isn't it amazing to use Unix/Linux 's native cat and wc tools as mapper / reducer?

If you are used to using some dynamic languages, use dynamic languages to write mapreduce, no different from previous programming, hadoop is just a framework to run it, let me demonstrate the use of PHP to implement Word Counter's mapreduce.

First, find Streaming jar

There is no hadoop-streaming.jar in the root directory of Hadoop. Because streaming is a contrib, you need to look under contrib. Take hadoop-0.20.2 as an example, it is here:

The copy code is as follows:

$HADOOP_HOME/contrib/streaming/hadoop-0.20.2-streaming.jar

Second, write Mapper

Create a new wc_mapper.php and write the following code:

The copy code is as follows:

#! / usr/bin/php