Shulou(Shulou.com)06/03 Report--

Using hdfs is one of the most common hadoop cluster management tasks, and although you can access HDFS in many ways, the command line is the most common way to manage HDFS storage. For example, webHDFS, using a HttpFS gateway to access HDFS through a firewall, and a file browser through Hue.

Use the hdfs: dfs command to manage HDFS

The syntax is as follows:

Hdfs dfs [GENERIC_OPTION] [COMMAND_OPTION]

1. List the files and directories of HDFS

Hdfs dfs-ls / files and directories for viewing files / paths

You can add-d after ls to list directories and view directory-related information-R-h

Hdfs dfs-cat / path/file.txt View the contents of file.txt

You can enter hdfs dfs-help for more options about hdfs dfs

2. Use the hdfs stat command to get the details of related files

% n returns a file or directory name

B returns the size of the file directory, byte. From the above, we can see that the directory does not take up space and its size is 0. 5%.

% g returns group

Mtime of inode in y format

% u user

3. Create a HDFS directory

Creating a hdfs directory is similar to creating a directory in the linux file system, you can use the mkdir command to create a hdfs creation

Hadfs dfs-mkdir / usr/hadoop/dir1 must exist / usr/hadoop path to create

If you want to create a parent directory along the path, specify the-p parameter

4. Delete HDFS files and directories

The command to delete hdfs files and directories is similar to the command in the linux file system. Use the-r | R option to recursively delete the directory and all contents under that directory.

The-skipTrash option is to delete the specified files and directories immediately, bypassing the hdfs Recycle Bin

Hdfs dfs-rm-R / usr/hadoop/dir/ deletes files and directories in the dir directory and dir directory

5. Change file directory ownership and groups

You can use the-chown command to change the owner and group names

$hdfs dfs-chown sam:produser / usr/data/name.txt

Use the chgrp command to change only the user's group

$hdfs dfs-chgrp supergroup / data/sales/makrer.txt

Change hdfs file permissions

You can use the chmod command to change the permissions of a file or directory hadoop uses standard linux file permissions.

$hdfs dfs-chmod-R / data/meta.txt

Only the superuser or the owner of files and directories can change permissions. Use the chgrp, chmod, and chown commands and specify the-R option to make recursive changes to the specified directory

6. Use the dfsadmin user program to perform HDFS operations

Use the dfsadmin command to manage HDFS from the command line, and although the hdfs dfs command can manage HDFS files and directories, the dfsadmin command can perform HDFS-specific administrative tasks.

The dfsadmin-report command displays basic statistics of the cluster, including useful information such as the status of DataNode and NaneNode, configured disk capacity, and the health of data blocks.

The dfsadmin-report command displays HDFS information for the entire cluster, as well as HDFS details for each node in the cluster. The DFS command displays the following information for each dataNode level of the cluster.

The hdfsadmin-report command checks the balance of HDFS data, as well as HDFS corruption.

The Dfsadmin-refreshNodes command updates the list of DataNode connected to the NameNode. Namenode reads the hostname of the DataNode from the dfs.hosts pointing to the dfs.hosts.ex-clude configuration parameter in the file and hdfs-site.xml. The Dfs.hosts file lists all hosts that are allowed to register to namenodede. The dfs.hosts.exclude file lists all the datanode that needs to be deactivated

The dfsadmin-metasave command provides more information than the dfsadmin-report command.

The file is saved under / hadoop/log/ and can be looked up locally by the find command

HDFS users and superusers

Creating a user in hadoop is a misnomer, because there is no way to create a user identity in HDFS as in the linux system. In the default authentication mode, hadoop relies on the underlying operating system to determine the client's identity. If the kerbers system is set to test kerberos, determine the client's identity.

Since there is no concept of users in hadoop, hadoop does not have a fixed super user. The system superuser of Hadoop is just the operating system user who starts NameNode. The superuser of Hdfs does not have to be a root user of the namenode host. You can assign a group of users to a separate superuser group.

Manage HDFS disk usage

You can use the df command to find available space

$hdfs dfs-df-h

Hdfs dfs-du-s-h gets the sum of used spaces

Check the current space quota

Use the dfs-count-Q command

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.