Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "the configuration process of hadoop-2.6.2 lzo". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Hadoop cluster description

The cluster has three hosts, and the host names are: bi10,bi12,bi13. All our operations are done on bi10.

Install dependency packages

Installing lzo requires some dependency packages, and if you have already installed it, you can skip this step. First, you need to switch to the root user.

Yum install gcc gcc-c++ kernel-develyum install git

In addition to the above two, you also need to configure the maven environment. After download, you can directly decompress and configure the environment variables to use.

Wget http://apache.fayea.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gztar-xzf apache-maven-3.3.9-bin.tar.gz

Configure the maven environment variable, and place the maven package in / home/hadoop/work/apache-maven-3.3.9

[hadoop@bi10 hadoop-2.6.2] $vim ~ / .bash_profile#init maven environmentexport MAVEN_HOME=/home/hadoop/work/apache-maven-3.3.9export PATH=$PATH:$MAVEN_HOME/binLZO installation

Download the lzo installation package

[hadoop@bi10 apps] $wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.09.tar.gz

Extract and compile and install lzo to: / usr/local/hadoop/lzo/, and switch to the root user during installation

[hadoop@bi10 apps] $tar-xzf lzo-2.09.tar.gz [hadoop@bi10 apps] $cd lzo-2.09 [hadoop@bi10 apps] $su root [root@bi10 lzo-2.09] $. / configure-enable-shared-prefix=/usr/local/hadoop/lzo/ [root@bi10 lzo-2.09] $make & & make test & & make install

View the installation directory

[hadoop@bi10 lzo-2.09] $ls / usr/local/hadoop/lzo/include lib shareHADOOP-LZO installation

Download hadoop-lzo

Git clone https://github.com/twitter/hadoop-lzo.git

Set environment variables and compile using maven

[hadoop@bi10 hadoop-lzo] $export CFLAGS=-m64 [hadoop@bi10 hadoop-lzo] $export CXXFLAGS=-m64 [hadoop@bi10 hadoop-lzo] $export C_INCLUDE_PATH=/usr/local/hadoop/lzo/ codes [Hadoop @ bi10 hadoop-lzo] $export LIBRARY_PATH=/usr/local/hadoop/lzo/ lib [Hadoop @ bi10 hadoop-lzo] $mvn clean package-Dmaven.test.skip=true

Copy the compiled files to the hadoop installation directory

[hadoop@bi10 hadoop-lzo] $tar-cBf-C target/native/Linux-amd64-64/lib. | | tar-xBvf-- C $HADOOP_HOME/lib/native/ [Hadoop @ bi10 hadoop-lzo] $cp target/hadoop-lzo-0.4.20-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/ [Hadoop @ bi10 hadoop-lzo] $scp target/hadoop-lzo-0.4.20-SNAPSHOT.jar bi12:$HADOOP_HOME/share/hadoop/common/ [Hadoop @ bi10 hadoop-lzo] $scp target/hadoop-lzo-0.4.20-SNAPSHOT.jar bi13:$HADOOP_HOME/share/hadoop/common/ |

Copy the compiled files to the corresponding directories of other machines in the cluster, in which the native directory needs to be packaged and copied to other machines in the cluster, and then decompressed.

Tar-czf hadoop-native.tar.gz / $HADOOP_HOME/lib/native/scp hadoop-native.tar.gz bi12:/$HADOOP_HOME/libscp hadoop-native.tar.gz bi13:/$HADOOP_HOME/lib modify hadoop configuration file

Modify hadoop-env.sh to add one

# The lzo libraryexport LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

Modify core-site.xml

Io.compression.codecs org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec io.compression.codec.lzo.class com.hadoop.compression.lzo.LzoCodec

Modify mapred-site.xml

Mapred.compress.map.output true mapred.map.output.compression.codec com.hadoop.compression.lzo.LzoCodec mapred.child.env LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

Copy three configuration files to other machines in the cluster

Scp etc/hadoop/hadoop-env.sh bi12:/home/hadoop/work/hadoop-2.6.2/etc/hadoop/scp etc/hadoop/hadoop-env.sh bi13:/home/hadoop/work/hadoop-2.6.2/etc/hadoop/scp etc/hadoop/core-site.xml bi12:/home/hadoop/work/hadoop-2.6.2/etc/hadoop/scp etc/hadoop/core-site.xml bi13:/home/hadoop/work/hadoop-2.6.2/ Etc/hadoop/scp etc/hadoop/mapred-site.xml bi12:/home/hadoop/work/hadoop-2.6.2/etc/hadoop/scp etc/hadoop/mapred-site.xml bi13:/home/hadoop/work/hadoop-2.6.2/etc/hadoop/ Test hadoop lzo

To install lzop, you need to switch to the root user

Yum install lzop

Entering the hadoop installation directory and performing lzo compression on LICENSE.txt will generate a lzo compressed file LICENSE.txt.lzo

Lzop LICENSE.txt

Upload compressed files to hdfs

[hadoop@bi10 hadoop-2.6.2] $hdfs dfs-mkdir / user/hadoop/wordcount/lzoinput [hadoop@bi10 hadoop-2.6.2] $hdfs dfs-put LICENSE.txt.lzo / user/hadoop/wordcount/lzoinput [hadoop@bi10 hadoop-2.6.2] $hdfs dfs-ls / user/hadoop/wordcount/lzoinputFound 1 items-rw-r--r-- 2 hadoop supergroup 7773 2016-02-16 20:59 / user/hadoop/wordcount/lzoinput/LICENSE.txt.lzo

Index lzo compressed files

Hadoop jar. / share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar com.hadoop.compression.lzo.DistributedLzoIndexer / user/hadoop/wordcount/lzoinput/ [hadoop@bi10 hadoop-2.6.2] $hdfs dfs-ls / user/hadoop/wordcount/lzoinput/Found 2 items-rw-r--r-- 2 hadoop supergroup 7773 2016-02-16 20:59 / user/hadoop/wordcount/lzoinput/LICENSE.txt.lzo-rw-r--r-- 2 Hadoop supergroup 8 2016-02-16 21:02 / user/hadoop/wordcount/lzoinput/LICENSE.txt.lzo.index

Wordcount the lzo zip file

This is the end of hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar wordcount/ user/hadoop/wordcount/lzoinput/ / user/hadoop/wordcount/output2 "hadoop-2.6.2 lzo configuration process". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.