Shulou(Shulou.com)05/31 Report--

This article mainly explains "hadoop streaming archives decompression jar, zip, tar.gz verification analysis", the explanation content in the article is simple and clear, easy to learn and understand, please follow the idea of small editor slowly in-depth, together to study and learn "hadoop streaming archives decompression jar, zip, tar.gz verification analysis" bar!

1. Archives Description:

One of the DistributedCache methods in Hadoop (see reference articles later in other reference articles). Its function is to distribute specified files to the working directory of each Task, and automatically decompress files with name suffixes of ".jar",".zip",".tar.gz" and ".tgz". By default, the decompressed contents are stored in the directory named as the file name before decompression under the working directory. For example, if the compressed package is dict.zip, the decompressed contents are stored in the directory dict.zip. To do this, you can give the file an alias/soft link, such as dict.zip#dict, so that the compressed package will be extracted into the directory dict.

2. Test jar file (basically directly extract reference document)

$ ls test_jar/file file1 file2 file = this is file1(this is a mistake during the experiment, it should be file1, which has no effect on the result, and does not modify it)file2 = this is file2$ jar cvf cache.jar -C test_jar/ .$ hdfs dfs -put cache. jar/user/work/cachefile#touch an input.txt file, then put to/user/work/cachefile$ hdfs dfs -cat /user/work/cachefile/input.txtcache/file (cache is the directory name after decompression, alias redefined with #, cache/file2HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh6.1.3 $HADOOP_HOME/bin/hadoop fs -rmr /cacheout/$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.3.0-cdh6.1.3.jar \ -archives /user/work/cachefile/cache.jar#cache \ -Dmapred.map.tasks=1 \ -Dmapred.reduce.tasks=1 \ -Dmapred.job.name="Experiment" \ -input "cachefile/input2.txt" \ -output "/cacheout/" \ -mapper "xargs cat" \ -reducer "cat" hadoop fs -cat /cacheout/*this is file 2this is file1

3. Test zip & tar.gz

Pack zip, tar.gz compression package respectively, put to hdfs to continue testing.

-archives /user/work/cachefile/cache.tar.gz#cache \ Only modify the suffix name, will report file not found error

Error: confirm whether it can be decompressed, change mapper to:

-mapper "ls cache" \

Found: jar file: The result is 4 files, namely META-INF, file, file1, file2

zip & tar.gz: There is only one, which is the directory name of test_jar

Then check the compressed files of the 3 packages, obviously the decompression was successful, the reason why the files could not be found was the directory problem, this is to study the packaging method in the next 3 in detail, and no longer repeat:

Thank you for reading, the above is the content of "hadoop streaming archives decompression jar, zip, tar.gz verification analysis", after the study of this article, I believe you have a deeper understanding of hadoop streaming archives decompression jar, zip, tar.gz verification analysis this problem, the specific use of the situation still needs to be verified. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.