Shulou(Shulou.com)06/02 Report--

Preface of HDFS

L design idea

Divide and conquer: large files and large quantities of files are distributed and stored on a large number of servers to facilitate the operation and analysis of massive data in a divide-and-conquer way.

L function in big data system:

For all kinds of distributed computing frameworks (such as mapreduce,spark,tez,... ) provide data storage services

Key concepts: file slicing, copy storage, metadata

The concept and characteristics of HDFS

First of all, it is a file system that stores files and locates them through a unified namespace, the directory tree.

Secondly, it is distributed, and many servers work together to achieve its functions, and the servers in the cluster have their own roles.

Important features are as follows:

(1) the file in HDFS is physically partitioned storage (block), and the size of the block can be specified by the configuration parameter (dfs.blocksize). The default size is 128m in hadoop2.x version and 64m in old version.

(2) the HDFS file system provides a unified abstract directory tree for the client. The client accesses the file through a path, such as hdfs://namenode:port/dir-a/dir-b/dir-c/file.data.

(3) the management of directory structure and file partition information (metadata) is undertaken by the namenode node.

-- namenode is the master node of the HDFS cluster, which is responsible for maintaining the directory tree of the entire hdfs file system and the block block information corresponding to each path (file) (block's id and the datanode server).

(4) the storage management of each block of the file is undertaken by the datanode node

-datanode is the slave node of HDFS cluster. Each block can store multiple replicas on multiple datanode (the number of replicas can also be set by parameter dfs.replication)

(5) HDFS is designed to adapt to the scenario of writing once and reading out multiple times, and does not support file modification.

(note: it is suitable for data analysis, but not suitable for network disk applications, because it is inconvenient to modify, large delay, high network overhead and high cost.)

Introduction of commonly used command parameters

-help

Function: output this command parameter manual

-ls

Function: display catalog information

Example: hadoop fs-ls hdfs://hadoop-server01:9000/

Note: among these parameters, all hdfs paths can be abbreviated.

-- > hadoop fs-ls / equivalent to the effect of the previous command

-mkdir

Function: create directories on hdfs

Example: hadoop fs-mkdir-p / aaa/bbb/cc/dd

-moveFromLocal

Function: cut and paste from local to hdfs

Example: hadoop fs-moveFromLocal / home/hadoop/a.txt / aaa/bbb/cc/dd

-moveToLocal

Function: cut and paste from hdfs to local

Example: hadoop fs-moveToLocal / aaa/bbb/cc/dd / home/hadoop/a.txt

-- appendToFile

Function: append a file to the end of an existing file

Example: hadoop fs-appendToFile. / hello.txt hdfs://hadoop-server01:9000/hello.txt

It can be abbreviated as:

Hadoop fs-appendToFile. / hello.txt / hello.txt

-cat

Function: display the contents of the file

Example: hadoop fs-cat / hello.txt

-tail

Function: displays the end of a file

Example: hadoop fs-tail / weblog/access_log.1

-text

Function: print the contents of a file in character form

Example: hadoop fs-text / weblog/access_log.1

-chgrp

-chmod

-chown

Function: the same usage in the linux file system, permissions to which the file belongs

Example:

Hadoop fs-chmod 666 / hello.txt

Hadoop fs-chown someuser:somegrp / hello.txt

-copyFromLocal

Function: copy files from the local file system to the hdfs path

Example: hadoop fs-copyFromLocal. / jdk.tar.gz / aaa/

-copyToLocal

Function: copy from hdfs to local

Example: hadoop fs-copyToLocal / aaa/jdk.tar.gz

-cp

Function: copy one path of hdfs from another path of hdfs

Example: hadoop fs-cp / aaa/jdk.tar.gz / bbb/jdk.tar.gz.2

-mv

Function: move files in the hdfs directory

Example: hadoop fs-mv / aaa/jdk.tar.gz /

-get

Function: equivalent to copyToLocal, which downloads files from hdfs to local

Example: hadoop fs-get / aaa/jdk.tar.gz

-getmerge

Function: merge and download multiple files

Example: for example, there are multiple files under the directory / aaa/ of hdfs: log.1, log.2,log.3,...

Hadoop fs-getmerge / aaa/log.*. / log.sum

-put

Function: equivalent to copyFromLocal

Example: hadoop fs-put / aaa/jdk.tar.gz / bbb/jdk.tar.gz.2

-rm

Function: delete files or folders

Example: hadoop fs-rm-r / aaa/bbb/

-rmdir

Function: delete empty directory

Example: hadoop fs-rmdir / aaa/bbb/ccc

-df

Function: statistics of the free space information of the file system

Example: hadoop fs-df-h /

-du

Function: statistics of folder size information

Example:

Hadoop fs-du-s-h / aaa/*

-count

Function: count the number of file nodes in a specified directory

Example: hadoop fs-count / aaa/

-setrep

Function: set the number of copies of files in hdfs

Example: hadoop fs-setrep 3 / aaa/jdk.tar.gz

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.