Shulou(Shulou.com)05/31 Report--

This article is to share with you about Hadoop Pipes programming C++ how to achieve WordCount, the editor feels very practical, so share with you to learn, I hope you can get something after reading this article, say no more, follow the editor to have a look.

Although Hadoop is implemented in java, it can also support Map and Reduce in other languages. Because I need to learn the Hadoop implementation of C++, I have been studying Hadoop Streaming programming recently, in which Hadoop pipes allows C++ programmers to write mapreduce programs, which allows users to mix C++ and Java's RecordReader, Mapper, Partitioner,Rducer and RecordWriter components. Here are my structures and the problems I have encountered.

Generally speaking, this kind of construction process is better seen from the official website, refer to http://wiki.apache.org/hadoop/C++WordCount, the process inside is simple and easy to understand. Enter the installation directory of hadoop and execute:

Ant-Dcompile.c++=yes examples

According to the error prompt, install ant:

Sudo apt-get install ant

Here are a steady stream of mistakes and solutions.

Error1:/build.xml:634: Execute failed: java.io.IOException: Cannot run program "autoreconf" (in directory "/ home/hadoop/hadoop-1.2.1/src/native"): error=2, No such file or directory

Solution1: execute because the automake tool is not installed according to the error:

Sudo apt-get install automake

Error2:build.xml:634: exec returned: 1

Solution2: http://stackoverflow.com/questions/23112074/building-hadoop-1-2-1-core-jar-using-ant-failed, execute:

Sudo apt-get install libtool

Error3:build.xml:2164: exec returned: 255

Solution3:forum.hadoop.tw/viewtopic.php?f=7&t=37970, refer to the implementation steps in it.

1) install glossy +

Sudo apt-get install gathers +

2) compile wordcount.cpp

Gmail +-I/home/hadoop/hadoop-1.2.1/c++/Linux-i386-32/include-c wordcount.cpp

3) generate executable file

Gmail + wordcount.o-o wordcount-L/home/hadoop/hadoop-1.2.1/c++/Linux-i386-32/lib-lnsl-lpthread-lhadooppipes-lhadooputils

Error4:In function `HadoopPipes::BinaryProtocol::createDigest (std::string&, std::string&)':

Solution4: http://blog.sina.com.cn/s/blog_605f5b4f010195w6.html

Step1:

Sudo apt-get install libssl-dev

Step2:

Gmail + wordcount.o-o wordcount-L/home/hadoop/hadoop-1.2.1/c++/Linux-i386-32/lib-lnsl-lpthread-lhadooppipes-lhadooputils-lcrypto

After a lot of problems above, the executable program WordCount is finally generated, and then you need to upload it to HDFS and execute it with an executable file. Here is the command:

Hadoop fs-mkdir wordcount/binhadoop fs-mkdir wordcount/inputhadoop fs-put wordcount.cpp / user/hadoop/wordcount/input/hadoop fs-put wordcount/ user/hadoop/wordcount/bin/hadoop pipes-D hadoop.pipes.java.recordreader=true-D hadoop.pipes.java.recordwriter=true-program / user/hadoop/wordcount/bin/wordcount-input/ user/hadoop/wordcount/input/-output / user/hadoop/wordcount/output

I take wordcount.cpp as the input file, upload the executable file to the bin directory, and the output directory is output.

Problems are encountered in the process of implementation:

Error5:java.io.IOException

At org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication (OutputHandler.java:188)

Solution5: http://blog.csdn.net/sigxxl/article/details/12293435, finally found a colleague. The main solution is to recompile the static libraries libhadooppipes.an and libhadooputils.a, and then overwrite the original static libraries.

1) enter the hadoop/src/c++/pipes directory and execute. / configure.

Error6:./configure: line 413: test: please: integer expression expected

. / configure: line 416: $4: Bad file descriptor

Configure: error: check

. / configure: line 302: return: please: numeric argument required

. / configure: line 312: exit: please: numeric argument required

Solution6: find this line in the configure file, which is usually specified when an error is reported. Comment out the as_fn_exit as follows:

Solution:as_fn_error () {as_status=$1; test $as_status-eq 0 & & as_status=1if test "$4"; thenas_lineno=$ {as_lineno- "$3"} as_lineno_stack=as_lineno_stack=$as_lineno_stack$as_echo "$as_me:$ {as_lineno-$LINENO}: error: $2" > & $4fi$as_echo "$as_me: error: $2" > & 2#as_fn_exit$as_status} # as_fn_error

2). / configure

Make install (modify the configure file again when you encounter the same error as above)

3) enter hadoop/src/c++/utils and perform the same steps

. / configure

Make install

After generating the new libhadooppipes.an and libhadooputils.a static libraries and related header files, overwrite these files to the include directory and lib directory in the ~ / hadoop/c++/Linux-i386-32 / folder. Restart hadoop, and then run the C++ program again.

Add: when I compiled again, I threw error:g++: error:-lcrypto: No such file or directory again

This is freaking me out. I checked it many times because I couldn't find libcrypto.so.

Solution: http://blog.csdn.net/yasi_xi/article/details/8658191

Take myself as an example, go to the / lib/i386-linux-gnu directory and ll libcrypto*, to check the current link. The result is that there is no libcrypto.so and there is no point to libcrypto.so.1.0.0, that is, it cannot be found in ld, so add a soft connection:

Ln-s / lib/i386-linux-gnu/libssl.so.1.0.0 / lib/libssl.so.1.0.0/libssl.so

Ln-s / usr/i386-linux-gnu/libcrypto.so.1.0.0 / lib/libssl.so.1.0.0/libcrypto.so

Of course, you have to find these libssl.so.1.0.0 files by yourself. if you can't find them, go to the next one. In theory, if you install openssl, you should have it. Ok, add that you finally got it right.

Run again, OK.

This is how C++ implements WordCount in Hadoop Pipes programming. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.