Shulou(Shulou.com)06/03 Report--

1. Hdfs startup process

1. Load fsimage, that is, load metadata

2. Load the edits file, because the above fsimage does not necessarily contain the latest metadata

3. Save checkpoints (checkpoint), mainly for use when merging edits to fsimage after snn

4. Enter safe mode and check whether the copy rate meets the requirement of redundancy, or whether it is complete. If not, it is in safe mode and cannot be written. Exit safe mode if satisfied.

Basic operating commands

We can use either hdfs dfs xxx or hadoop fs xxx, both are the same.

The complete commands are as follows:

Usage: hadoop fs [generic options] [- appendToFile...] [- cat [- ignoreCrc]...] [- checksum...] [- chgrp [- R] GROUP PATH...] [- chmod [- R] PATH...] [- chown [- R] [OWNER] [: [GROUP]] PATH...] [- copyFromLocal [- f] [- p] [- l] [- d]...] [- copyToLocal [- f] [- p] [- ignoreCrc] [- crc]...] [- count [- Q] [- h] [- v] [- t []] [- u] [- x]...] [- cp [- f] [- p |-p [topax]] [ -d]...] [- createSnapshot []] [- deleteSnapshot] [- df [- h] [...]] [- du [- s] [- h] [- x]...] [- expunge] [- find...] [- get [- f] [- p] [- ignoreCrc] [- crc]...] [- getfacl [- R]] [- getfattr [- R] {- n name |-d} [- e en]] [- getmerge [- nl] [- skip-empty-file]] [- help [cmd...]] [- ls [- C] [- d] [- h] [- Q] [- R] [ -t] [- S] [- r] [- u] [...] [- mkdir [- p]...] [- moveFromLocal...] [- moveToLocal] [- mv...] [- put [- f] [- p] [- d]...] [- renameSnapshot] [- rm [- f] [- r |-R] [- skipTrash] [- safely]...] [- rmdir [--ignore-fail-on-non-empty]...] [- setfacl [- R] [{- b |-k} {- m |-x}] | [--set]] [- setfattr {- n name [- v value] |-x name}] [ -setrep [- R] [- w]...] [- stat [format]...] [- tail [- f]] [- test-[defsz]] [- text [- ignoreCrc]...] [- touchz...] [- truncate [- w]...] [- usage [cmd...]]

1. Get command help

Hdfs dfs-help command such as: hdfs dfs-help ls

2. Create a directory

Hdfs dfs-mkdir HDFS path option:-p Recursive create directory

3. List directory information

Hdfs dfs-ls HDFS path option:-R recursively displays all directories and all subdirectories and files under the directory

4. Upload files or directories

Hdfs dfs-put local path HDFS path hdfs dfs-copyFromLocal local path hdfs path hdfs dfs-moveFromLocal local path hdfs path this will delete the local file

5. Download files or directories

Hdfs dfs-get HDFS path local path hdfs dfs-copyToLocal HDFS path local path

6. View the contents of the file

Hdfs dfs-cat HDFS path

7. Append the local file contents to the HDFS file

Hdfs dfs-appendToFile local path HDFS path

8. Monitor the contents of files

Hdfs dfs-tail-f HDFS path

9. Delete a file or directory

Hdfs dfs-rm HDFS path option:-r Recursive delete subdirectories Note: the default Recycle Bin is closed, so the deleted files are immediately gone

10. Merge the data and download it locally

Hdfs dfs-getmerge HDFS path 1 HDFS path 2. Local path

11. Statistical category

Hdfs dfs-count HDFS path count directory number of files total number of files hdfs dfs-du HDFS path shows directory or file size hdfs dfs-df HDFS path shows space usage

12. Copy and cut

Hdfs dfs-cp HDF***c HDFSdest HDFS internal replication hdfs dfs-mv HDF***c HDFSdest HDFS internal shear

13. File permissions

Hdfs dfs-chown user:group HDFS path modification subordinate group hdfs dfs-chmod permissions HDFS path modification permissions

14. Set the number of copies

Hdfs dfs-setrep n HDFS path III. Administrative commands

1. Balanced datanode

Hdfs balancer equalizes the data of multiple datanode nodes to prevent some nodes from storing too much data.

2. Print and run the report

Hdfs dfsadmin-report

3. Safety mode

Hdfs dfsadmin-safemode enter | leave | get enters safe mode | leaves safe mode | gets safe mode status

4. Snapshot

Hdfs dfsadmin-allowSnapshot HDFS directory take a snapshot of a directory hdfs dfsadmin-disableSnapshot HDFS directory unsnap a directory

5. Quota

/ / name quota, which is simply a limit on the total number of files and directories under this directory hdfs dfsadmin-setQuota N HDFS directory 1 HDFS directory 2. Set name quota hdfs dfsadmin-clrQuota HDFS directory 1 HDFS directory 2. Cancel the name quota / / space quota, which simply means that the storage space of the directory is limited by hdfs dfsadmin-setSpaceQuota N units (such as: 1GB) [- storageType] HDFS directory 1 HDFS directory 2. Set the space quota hdfs dfsadmin-clrSpaceQuota [- storageType] HDFS directory 1 HDFS directory 2. To cancel the space quota, there are the following types of storageType: RAW-DISK: raw physical disk DISK: partition space SSD: solid state disk ARCHIVE: 4. Hadoop start related commands function commands start / stop History Server mr-jobhistory-daemon.sh start | stop historyserver start / stop Total Resource Manager yarn-daemon.sh start | stop resourcemanager start / stop Node Manager yarn-daemon.sh start | stop nodemanager start / stop NN and DNstart | stop-dfs.sh start / stop RN and NMstart | stop-yarn.sh start / stop NN, DN, RN, NMstart | stop-all.sh start / stop NNhadoop-daemon.sh start | stop namenode start / stop DNhadoop-daemon.sh start | stop datanode

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.