Shulou(Shulou.com)06/03 Report--

HDFS is a distributed file system in the Hadoop big data platform, providing data storage for upper applications or other big data components, such as Hive, Mapreduce, Spark, HBase, etc. All commands in this article are run in CentOS-6.4-x86_64, hadoop-2.5.2, jdk1.8.0_152, zookeeper-3.4.11. To reduce the impact of Linux privileges on beginners, all commands are operated under Linux root privileges. Hadoop technology itself includes HDFS, Map/Reduce. HDFS is used for mass data storage and M/R is used for mass data calculation. HDFS contains namenode and datanode. HDFS A namenode indexes the data in a datanode. Zookeeper acts as a coordination service, providing high availability for namenode. 3. Basic operation

Note: "# ./ The "#" preceding the hdfs dfs -ls /"command indicates that the currently logged in user is root. The run path is located under the bin folder in the hadoop directory. "..." in the command. "Represents the current directory. The info file involved in the command needs to be created in advance. As shown below:

3.1-ls function: Display directory information.# ./ hdfs dfs -ls /drwxr-xr-x - root supergroup 0 2018-07-30 00:09 /hbasedrwxr-xr-x - root supergroup 0 2018-06-23 15:22 /outputdrwx------ - root supergroup 0 2018-07-31 00:32 /tmpdrwxr-xr-x - root supergroup 0 2018-07-31 00:41 /user-rw-r--r-- 2 root supergroup 77 2018-04-22 02:34 /wordcount3.2 -mkdir Feature: Create directories on HDFS file systems.# ./ hdfs dfs -mkdir /wanhe# ./ hdfs dfs -ls /drwxr-xr-x - root supergroup 0 2018-07-30 00:09 /hbasedrwxr-xr-x - root supergroup 0 2018-06-23 15:22 /outputdrwx------ - root supergroup 0 2018-07-31 00:32 /tmpdrwxr-xr-x - root supergroup 0 2018-07-31 00:41 /userdrwxr-xr-x - root supergroup 0 2018-09-12 18:00 /wanhe-rw-r--r-- 2 root supergroup 77 2018-04-22 02:34 /wordcount3.3 -put function: Upload local files to HDFS specified directory.# ./ hdfs dfs -put info /wanhe# ./ hdfs dfs -ls /wanhe-rw-r--r-- 2 root supergroup 38 2018-09-12 18:10 /wanhe/info3.4 -get function: download files from hdfs to local.# rm -rf info# lscontainer-executor hadoop hadoop.cmd hdfs hdfs.cmd mapred mapred.cmd rcc test-container-executor yarn yarn.cmd# ./ hdfs dfs -get /wanhe/info ./# lscontainer-executor hadoop hadoop.cmd hdfs.cmd info mapred mapred.cmd rcc test-container-executor yarn yarn.cmd3.5 -rm Function: Delete files from HDFS.# ./ hdfs dfs -rm /wanhe/info# ./ hdfs dfs -ls /wanhe null 3.6 -moveFromLocal function: cut local files to HDFS# ./ hdfs dfs -moveFromLocal info /wanhe# ./ hdfs dfs -ls /wanhe-rw-r--r-- 2 root supergroup 38 2018-09-12 22:04 /wanhe/info# lscontainer-executor hadoop.cmd hdfs.cmd mapred mapred.cmd rcc test-container-executor yarn yarn.cmd3.7 -cat Features: Display file content. # ./ hdfs dfs -cat /wanhe/info jiangsuwanhe3.8 -appendToFile function: append data to the end of the file. # ./ hdfs dfs -appendToFile info /wanhe/info # ./ hdfs dfs -cat /wanhe/info jiangsuwanhe jiangsuwanhe3.9 -chmod Function: Change file ownership permissions.# ./ hdfs dfs -ls /wanhe-rw-r--r-- 2 root supergroup 51 2018-09-12 22:13 /wanhe/info# ./ hdfs dfs -chmod 777 /wanhe/info# ./ hdfs dfs -ls /wanhe-rwxrwxrwx 2 root supergroup 51 2018-09-12 22:13 /wanhe/info3.10 -cp function: realize file copy. Copy/wanhe/info to/tmp: # ./ hdfs dfs -cp /wanhe/info /tmp/# ./ hdfs dfs -ls /tmp-rw-r--r-- 2 root supergroup 51 2018-09-12 22:20 /tmp/info3.11 -mv Function: Move files. Move/wanhe/info to/user # ./ hdfs dfs -mv /wanhe/info /user/# ./ hdfs dfs -ls /wanhe null # ./ hdfs dfs -ls /user-rwxrwxrwx 2 root supergroup 51 2018-09-12 22:13 /user/info3.12 -df Function: Statistics available space information of file system.# ./ hdfs dfs -df -h /Filesystem Size Used Available Use%hdfs://master:9000 17.5 G 352 K 11.4 G 0%3.13 -du function: statistics folder size information.# ./ hdfs dfs -du /user51/user/info3.14 -count function: count the number of files in a specified directory.# ./ hdfs dfs -count /user 2 1 51 /user The first column 2 indicates the number of folders under/user/, and the second column 1 indicates the number of files under/user/. 51 represents the disk capacity occupied by all files in/user/directory (excluding the number of copies). 4. Summary

HDFS command line operations are similar to linux commands, skilled use of linux commands can be skilled use of command line operations HDFS. Later we will serialize big data series blog, welcome to pay attention to and exchange.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.