Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to use Shell script to judge the existence of HDFS file / directory", in daily operation, I believe many people have doubts about how to use Shell script to judge whether HDFS file / directory exists. Xiaobian consulted all kinds of information and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the question of "how to use Shell script to judge whether HDFS file / directory exists". Next, please follow the editor to study!

1 the purpose of document writing

This article is mainly about how to use Shell scripts to determine whether HDFS files or directories exist, which is a little trick, which is quite useful when doing PoC these days. Because HDFS is designed to "write once, read multiple" and cannot modify data, it can only be appended or deleted, so if the target file or directory already exists on the HDFS, it cannot be created successfully.

Test environment:

The operating system version is Redhat7.2

Version 5.11.2 for CM and CDH

Article directory structure:

1. Purpose of document writing

two。 Testing principle

3. Shell script testing

3.1 Test whether the path exists

3.2 Test whether the directory exists

3.3 Test whether the file exists

4. Summary

2 testing principle

To determine whether a file or directory exists by hadoop fs-test, only one test parameter can be passed at a time.

Usage: hadoop fs-test-[defsz] URI

-test: Only one test flag is allowed

Options:

-d: if the path is a directory, return 0.

-e: if the path exists, return 0.

-f: if the path is a file, return 0.

-s: if the path is not empty, return 0.

-z: if the file is zero length, return 0.

For example, to determine whether a file directory on HDFS exists, you can execute the following command:

Hadoopfs-test-d $path # determines whether the directory exists

Hadoopfs-test-e $path # determines whether the path (file or directory) exists

3 Shell script Test 3.1 whether the test path exists

1. Prepare the test directory, including a file and a folder

[root@bigdata60 ~] # hadoop fs-ls / user/wdtest

Found 2 items

Drwxr-xr-x-fusionuser wdtest 0 2018-07-03 13:26 / user/wdtest/.fusion

-rw-r--r-- 3 fusionuser wdtest 908 2018-07-06 22:37 / user/wdtest/hosts

2. Write the test script as follows:

#! / bin/sh

Path=$1

Hadoop fs-test-e $path

If [$?-eq 0]; then

Echo "Path is exist!"

Else

Echo "Path is not exist!"

Fi

Execute the command ". / etest.sh / user/wdtest", run the test script, and verify the output

3.2 Test whether the directory exists

1. Write the test script as follows:

#! / bin/sh

Path=$1

Hadoop fs-test-d $path

If [$?-eq 0]; then

Echo "Directory is exist!"

Else

Echo "Directory is not exist!"

Fi

2. Execute the command ". / dtest.sh / user/wdtest" and ". / dtest.sh / user/wdtest/aa" respectively, run the test script, and verify the output.

3.3 Test whether the file exists

1. Write the test script as follows:

#! / bin/sh

Path=$1

Hadoop fs-test-f $path

If [$?-eq 0]; then

Echo "File is exist!"

Else

Echo "File is not exist!"

Fi

2. Execute the command ". / ftest.sh / user/wdtest/hosts" and ". / dtest.sh / user/wdtest/host" respectively, run the test script, and verify the output.

4 Summary

1. Hadoop fs-test-[defsz] URI can determine whether a HDFS file or directory exists.-test can only pass one test parameter at a time, and cannot execute commands such as hadoop fs-test-dfURI or hadoop fs-test-d-fURI.

2. The return value of hadoop fs-test-[defsz] URI is not output and needs to be passed through "$?" This special variable gets the return value

3. I have tested the-s and-z commands, but I don't think they are commonly used, so I didn't talk about them in the article. If you are interested, you can test them by yourself. It's relatively simple. Just change the script parameters.

4. After judging whether the HDFS file directory exists, you can perform operations such as creating or deleting directories or files, and of course, there are some more advanced ways to play, which will not be demonstrated here.

At this point, the study on "how to use Shell scripts to determine whether HDFS files / directories exist" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.