Shulou(Shulou.com)06/03 Report--

Lu Chunli's work notes, who said that programmers should not have literary style?

1. Safety mode

The system will automatically enter safe mode when it starts, and if the DN node is normal, safe mode will automatically turn off after 30 seconds.

(1), parameter definition

Dfs.replication: sets the number of copies of data blocks that should be replicated

Dfs.replication.min: the minimum number of copies of the specified block

Dfs.safemode.threshold.pct: specify what percentage of blocks should meet the minimum number of replicas.

a. When it is less than this ratio, switch the system to safe mode and copy the data blocks.

b. When the ratio is greater than this ratio, it leaves the safe mode, indicating that the system has enough copies of data blocks to provide services.

c. Less than or equal to 0 means not entering safe mode, and an equal to 1 means being in safe mode all the time.

(2) the significance of the existence of dfs.replication.min

The number of copies is set according to dfs.replication. If the node fails, the number of block copies will be reduced. When it is lower than dfs.replication.min, the system will replicate new copies at other nodes. If a copy of the block is often lost, resulting in replication at too many nodes, then when the number of replicas of the dfs.replication.max is exceeded (the default is 512), it is no longer replicated.

(3) Safety mode

When NameNode starts, it first enters safe mode. If the block lost by datanode reaches a certain proportion (1-dfs.safemode.threshold.pct), the system will always be in safe mode state, that is, read-only state.

Dfs.safemode.threshold.pct (default value 0.999f) means that when HDFS starts, you can only leave safe mode if the number of block reported by DataNode reaches 0.999 times the number of block recorded in metadata, otherwise it will always be read-only. If set to 1, HDFS is always in SafeMode.

(4) operate by command

[hadoop@nnode hadoop2.6.0] $hdfs dfsadmin-help safemode-safemode: Safe mode maintenance command. Safe mode is a Namenode state in which it 1.does not accept changes to the name space (read-only) 2. Does not replicate or delete blocks. Safe mode is entered automatically at Namenode startup, and leaves safe mode automatically when the configured minimum percentage of blocks satisfies the minimum replication condition. Safe mode can also be entered manually, but then it can only be turned off manually as well.

Enter: enter safe mode

Leave: forces NN to leave safe mode

Get: returns the information whether safe mode is enabled or not

Wait: wait until the end of safe mode.

2. Disk quota

Hdfs dfsadmin [- setQuota...] [- clrQuota...] [- setSpaceQuota...] [- clrSpaceQuota...] Hdfs dfs [- count [- Q] [- h]...]

Example:

[hadoop@nnode hadoop2.6.0] $hdfs dfs-ls / dataFound 2 items-rw-r--r-- 2 hadoop hadoop 47 2015-06-09 17:59 / data/file1.txt-rw-r--r-- 2 hadoop hadoop 36 2015-06-09 17:59 / data/file2.txt [hadoop@nnode hadoop2.6.0] $hdfs dfsadmin-setQuota 4 / data [hadoop@nnode hadoop2.6.0] $hdfs dfs-count / data 1 2 83 / data # 47 + 36 = 83

View help information for count:

[hadoop@nnode hadoop2.6.0] $hdfs dfs-help count-count [- Q] [- h]...: Count the number of directories, files and bytes under the paths that match the specified file pattern. The output columns are: DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME or QUOTA REMAINING_QUOTA SPACE_QUOTA REMAINING_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME The-h option shows file sizes in human readable format.

Upload files:

[hadoop@nnode hadoop2.6.0] $hdfs dfs-put NOTICE.txt / data/ [hadoop@nnode hadoop2.6.0] $hdfs dfs-put README.txt / data/put: The NameSpace quota (directories and files) of directory / data is exceeded: quota=4 file count=5 [hadoop@nnode hadoop2.6.0] $# data directory itself occupies a limit [hadoop@nnode hadoop2.6.0] $hdfs dfs-ls / dataFound 3 items-rw-r--r-- 2 hadoop hadoop 101 2015-11-28 21:02 / data/NOTICE.txt-rw-r--r-- 2 hadoop hadoop 47 2015-06-09 17:59 / data/file1.txt-rw-r--r-- 2 hadoop hadoop 36 2015-06-09 17:59 / data/file2.txt [hadoop@nnode hadoop2.6.0] $hdfs dfs-count / data 13 184 / data [hadoop@ Nnode hadoop2.6.0] $hdfs dfsadmin-clrQuota / data [hadoop@nnode hadoop2.6.0] $hdfs dfs-rm / data/NOTICE.txt15/11/28 21:21:24 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes Emptier interval = 0 minutes.Deleted / data/NOTICE.txt [hadoop@nnode hadoop2.6.0] $[hadoop@nnode hadoop2.6.0] $hdfs dfsadmin-setSpaceQuota 200 / data [hadoop@nnode hadoop2.6.0] $hdfs dfs-ls / dataFound 2 items-rw-r--r-- 2 hadoop hadoop 47 2015-06-09 17:59 / data/file1.txt-rw-r--r-- 2 hadoop hadoop 36 2015-06-09 17:59 / Data/file2.txt [hadoop@nnode hadoop2.6.0] $hdfs dfs-put README.txt / data15/11/28 21:31:08 WARN hdfs.DFSClient: DataStreamer Exceptionorg.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of / data is exceeded: quota = 200B = 200B but diskspace consumed = 268435622 B = 256.00 MB [hadoop@nnode hadoop2.6.0] $hdfs dfsadmin-clrSpaceQuota / data

3. Dynamically modify replication factor

Hdfs dfs [- setrep [- R] [- w]...]

Check the number of copies, which is now 2:

-rw-r--r-- 2 hadoop hadoop 47 2015-06-09 17:59 / data/file1.txt-rw-r--r-- 2 hadoop hadoop 36 2015-06-09 17:59 / data/file2.txt

Set the number of copies of the file file1.txt to 1:

[hadoop@nnode hadoop2.6.0] $hdfs dfs-setrep 1 / data/file1.txtReplication 1 set: / data/file1.txt [hadoop@nnode hadoop2.6.0] $hdfs dfs-ls / dataFound 2 items-rw-r--r-- 1 hadoop hadoop 47 2015-06-09 17:59 / data/file1.txt-rw-r--r-- 2 hadoop hadoop 36 2015-06-09 17:59 / data/file2.txt [hadoop@nnode hadoop2.6.0] $

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.