Shulou(Shulou.com)06/01 Report--

How to understand how to install LZO compression program in CentOS7? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

1. Preparatory work:

Yum-y install lzo-devel zlib-devel gcc autoconf automake libtool

Second, install LZO

1. Decompress, compile, and install

Cd / opt/software

Wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.09.tar.gz

Tar-zxvf lzo-2.09.tar.gz

Cd lzo-2.09

. / configure-enable-shared-prefix=/usr/local/hadoop/lzo/

Make & & make test & & make install

2. Copy files

Copy / usr/local/hadoop/lzo/lib/* to / usr/lib/ and / usr/lib64/

Cp / usr/local/hadoop/lzo/lib/* / usr/lib/

Cp / usr/local/hadoop/lzo/lib/* / usr/lib64/

3. Modify the configuration environment variable (vi ~ / .bash_profile) by adding the following:

Export PATH=/usr/local/hadoop/lzo/:$PATH

Third, install LZOP

1. Download and decompress

Cd / opt/software

Wget http://www.lzop.org/download/lzop-1.04.tar.gz

Tar-zxvf lzop-1.04.tar.gz

2. Configure the following in the environment variable (~ / .bash_profile) required before compilation:

Export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include/

Note: if the variable is not configured, it will report: configure: error: LZO header files not found. Please check your installation or set the environment variable `CPPFLAGS'.

3. Enter the decompressed directory and compile and install it.

Cd cd / opt/software/lzop-1.04

. / configure-enable-shared-prefix=/usr/local/hadoop/lzop

Make & & make install

4. Copy lzop to / usr/bin/

Ln-s / usr/local/hadoop/lzop/bin/lzop / usr/bin/lzop

5. Test lzop

Input: lzop nohup.out

To generate a compressed file with the suffix of lzo: / home/hadoop/data/access_20131219.log.lzo indicates success

Note: you may encounter an error in the test: lzop: error while loading shared libraries: liblzo2.so.2: cannot open shared object file: No such file or directory

Solution: increase the environment variable (~ / .bash_profile) export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64

Fourth, install Hadoop-LZO

Note: maven is required when compiling. Configure maven by yourself.

1. Download media: https://github.com/twitter/hadoop-lzo

2. Decompress and compile:

Cd / opt/software/hadoop-lzo-release-0.4.19

Mvn clean package-Dmaven.test.skip=true

3. Complete the compilation and execute the following command:

Tar-cBf-C target/native/Linux-amd64-64/lib. | | tar-xBvf-C / app/hadoop-2.6.0-cdh6.7.0/lib/native |

Cp target/hadoop-lzo-0.4.19.jar / app/hadoop-2.6.0-cdh6.7.0/share/hadoop/common/

In the case of a clustered environment, the next step is to synchronize / app/hadoop-2.6.0-cdh6.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar and / app/hadoop-2.6.0-cdh6.7.0/lib/native/ to all other hadoop nodes.

Note that to ensure that the jar package under the directory / app/hadoop-2.6.0-cdh6.7.0/lib/native/, the user running hadoop has the right to execute.

5. Generate index files

Cd / app/hadoop-2.6.0-cdh6.7.0/share/hadoop/common

Hadoop jar hadoop-lzo-0.4.19.jar com.hadoop.compression.lzo.LzoIndexer / user/hive/warehouse/page_views_parquet1/page_views_parquet.lzo

Note: the lzo file must be in the hdfs file system.

This completes the installation of the LZO compressor in CentOS7

This is the answer to the question about how to understand the installation of LZO compression program in CentOS7. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.