Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to operate HDFS, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

What ① must know.

Stolen picture: HDFS architecture

1 、 Client

In our case, this is the Linux command or Java code.

2 、 NameNode

There is only one NameNode for the entire HDFS cluster, which stores separate metadata information for the entire cluster file. This information is stored on the local disk as fsimage and editlog files, and Client can find the corresponding files through these metadata information. In addition, NameNode is responsible for monitoring the health of the DataNode, kicking out the DataNode if it is abnormal, and copying the data to other DataNode. Although Client communicates with NameNode, the final operation file is still connected to Datenode.

3 、 Secondary NameNode

Secondary NameNode is responsible for regularly merging NameNode's fsimage and editlog. Note here that it is not a hot backup for NameNode, so NameNode is still Single Point of Failure. The main purpose of its existence is to share some of the work of NameNode (especially the work of consuming memory, because memory resources are very precious to NameNode).

4 、 DataNode

DataNode is responsible for the actual storage of the data. When a file is uploaded to the HDFS cluster, it is distributed in each DataNode in the basic unit of Block. At the same time, to ensure the reliability of the data, each Block is written to multiple DataNode at the same time (the default is 3, we have made it 2). DataNode reports his status to node through his heartbeat and gets the command execution assigned by namedode.

5 、 Block

Default 64m one piece, one piece deposit three points, young and old are not bullied.

② Linux Operation HDFS

The HDFS command is basically the Linux command preceded by "Hadoop fs -", the official document entry.

The following are representative commands:

Hadoop dfsadmin-safemode leave / / exit safe mode, otherwise you can't put

Hadoop fs-mkdir / user/kitten / / create a folder to create the HDFS user

Hadoop fs-chown kitten / user/kitten / / change the ownership of the folder

Hadoop fs-ls / user / / listing file

Hadoop fs-lsr / user / / Recursive list file

Hadoop fs-put test.txt / user/kitten / / copy from Linux to HDFS

Hadoop fs-get / user/kitten / test.txt / / fetch Linux from HDFS

Hadoop fs-cat / user/kitten/test.txt / / build file

Hadoop fs-tail / user/kitten/test.txt / / View the last 1000 bytes

Hadoop fs-rm / user/kitten/test.txt / / Delete

Hadoop fs-help ls / / International practice, check the help documentation for the ls command

Thank you for reading this article carefully. I hope the article "how to operate HDFS" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.