Shulou(Shulou.com)06/02 Report--

This article gives you an introduction to DFS and what is the Shell management command, the content is very detailed, interested friends can refer to, hope to be helpful to you.

1. Basic overview of HDFS 1. HDFS description

Big data field has been faced with the two core modules: data storage, data computing, HDFS as the most important big data storage technology, has a high degree of fault tolerance, stable and reliable. HDFS (Hadoop-Distributed-File-System), which is a distributed file system, is used to store files and locate files through the directory tree; the original design intention is to manage hundreds of servers and disks, so that applications store large-scale file data like an ordinary file system, suitable for writing once, reading multiple scenarios, and do not support file modification, suitable for data analysis.

2. Infrastructure

HDFS has a master / slave architecture and has two core components, NameNode and DataNode.

NameNode

Responsible for the metadata (MetaData) management of the file system, that is, file path name, block ID, storage location and other information, and configure copy policy to handle client read and write requests.

DataNode

Perform the actual storage and read and write operations of the file data, each DataNode stores part of the file data blocks, and the files are distributed and stored in the whole HDFS server cluster.

Client

On the client side, when the file is split and uploaded to HDFS, Client splits the file into a Block and then uploads it; obtains the location information of the file from NameNode; communicates with DataNode to read or write data; and Client accesses or manages HDFS through some commands.

Secondary-NameNode

It is not a hot backup for NameNode, but it shares the workload of NameNode, such as merging Fsimage and Edits on a regular basis and pushing it to NameNode; to assist in restoring NameNode in case of emergency.

3. High fault tolerance

For multiple copy storage of data blocks, file / users/sameerp/data/part-0, replication backup is set to 2, the stored block-ids is 1 and 3 respectively; file / users/sameerp/data/part-1, replication backup is set to 3, and the stored block-ids is 2, 4, 5, respectively After any single server goes down, at least one backup service for each data block survives, which will not affect the access to files and improve the overall fault tolerance.

The file in HDFS is physically Block, and the block size can be configured by the parameter dfs.blocksize. If the block setting is too small, it will increase the addressing time; if the block setting is too large, the time for transferring data from disk will be very slow, and the size setting of HDFS block mainly depends on the disk transfer rate.

2. Basic Shell command

1. Basic command

View the relevant Shell operation commands under Hadoop.

[root@hop01 hadoop2.7] # bin/hadoop fs [root@hop01 hadoop2.7] # bin/hdfs dfs

Dfs is the implementation class of fs

2. View the command description

[root@hop01 hadoop2.7] # hadoop fs-help ls

3. Recursively create a directory

[root@hop01 hadoop2.7] # hadoop fs-mkdir-p / hopdir/myfile

4. View the catalog

[root@hop01 hadoop2.7] # hadoop fs-ls / [root@hop01 hadoop2.7] # hadoop fs-ls / hopdir

5. Cut and paste files

Hadoop fs-moveFromLocal / opt/hopfile/java.txt / hopdir/myfile## View the file hadoop fs-ls / hopdir/myfile

6. View the contents of the file

# # View all hadoop fs-cat / hopdir/myfile/java.txt## and view hadoop fs-tail / hopdir/myfile/java.txt at the end

7. Append the contents of the file

Hadoop fs-appendToFile / opt/hopfile/c++.txt / hopdir/myfile/java.txt

8. Copy files

CopyFromLocal command is the same as put command

Hadoop fs-copyFromLocal / opt/hopfile/c++.txt / hopdir

9. Copy the HDFS file locally

Hadoop fs-copyToLocal / hopdir/myfile/java.txt / opt/hopfile/

10. Copy files within HDFS

Hadoop fs-cp / hopdir/myfile/java.txt / hopdir

11. Move files within HDFS

Hadoop fs-mv / hopdir/c++.txt / hopdir/myfile

12. Merge and download multiple files

The base commands get and copyToLocal have the same effect.

Hadoop fs-getmerge / hopdir/myfile/* / opt/merge.txt

13. Delete files

Hadoop fs-rm / hopdir/myfile/java.txt

14. View folder information

Hadoop fs-du-s-h / hopdir/myfile

15. Delete a folder

Bin/hdfs dfs-rm-r / hopdir/file0703 about DFS introduction and what Shell management commands are shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.