Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to view the HDFS file system space usage, the article is very detailed, has a certain reference value, interested friends must read it!

1. View the space usage status of the cluster

[hduser@master1 bin] $. / hdfs dfsadmin-report

Configured Capacity: 8453768839168 (7.69TB)

Present Capacity: 7959274510394 (7.24 TB)

DFS Remaining: 4128882869718 (3.76TB)

DFS Used: 3830391640676 (3.48TB)

DFS Used%: 48.12%

Under replicated blocks: 18

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (5):

Name: 192.168.11.137 50010 (slave5)

Hostname: slave5

Decommission Status: Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 736041209958 (685.49 GB)

Non DFS Used: 66876043079 (62.28 GB)

DFS Remaining: 253700128851 (236.28 GB)

DFS Used%: 69.66%

DFS Remaining%: 24.01%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 30

Last contact: Tue Jul 03 15:06:51 CST 2018

Name: 192.168.11.134 50010 (slave2)

Hostname: slave2

Decommission Status: Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 749551308902 (698.07 GB)

Non DFS Used: 120793546401 (112.50 GB)

DFS Remaining: 1243166503161 (1.13 TB)

DFS Used%: 35.46%

DFS Remaining%: 58.82%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 23

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.133 50010 (slave1)

Hostname: slave1

Decommission Status: Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.135 Name 50010 (slave3)

Hostname: slave3

Decommission Status: Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 752023101542 (700.38 GB)

Non DFS Used: 121481747963 (113.14 GB)

DFS Remaining: 1240006508959 (1.13 TB)

DFS Used%: 35.58%

DFS Remaining%: 58.67%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 26

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.136 Name 50010 (slave4)

Hostname: slave4

Decommission Status: Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 800412266598 (745.44 GB)

Non DFS Used: 66495765928 (61.93 GB)

DFS Remaining: 189709349362 (176.68 GB)

DFS Used%: 75.75%

DFS Remaining%: 17.95%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 30

2. Analysis of 192.168.11.133

$df-h

Filesystem Size Used Avail Use% Mounted on

/ dev/mapper/VolGroup00-LVroot

17G 12G 4.9G 70% /

Tmpfs 7.8G 0 7.8G 0% / dev/shm

/ dev/sda1 190m 125m 56m 70% / boot

/ dev/mapper/VolGroup00-LVhome

477m 2.4m 449m 1% / home

/ dev/mapper/VolGroup00-LVcloud

2.9g 247m 2.5G 9% / opt/cloud

/ dev/mapper/VolGroup00-LVtmp

7.8G 19m 7.4G 1% / tmp

/ dev/mapper/VolGroup00-LVvar

25G 786m 23G 4% / var

/ dev/mapper/VolGroup1-LVdata1

985G 384G 551G 42% / data

/ dev/mapper/VolGroup2-LVdata2

985G 365G 570G 40% / data2

Data Statistics:

/ data and / data2 are the data storage directories

Total capacity = 985G + 985G = 1970G = 1.92T

Used capacity = 384G + 365G = 749G

Unused capacity = 551G + 570G = 1121G

Undisplayed capacity = 1970G-(749G + 1121G) = 100g

The unzipped files of hadoop and hbase in the / data directory occupy about 10G.

Dfs file system analysis:

Name: 192.168.11.133 50010 (slave1)

Hostname: slave1

Decommission Status: Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0B)

Cache Used: 0 (0B)

Cache Remaining: 0 (0B)

Cache Used%: 100.00%

Cache Remaining%: 0.005%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

Description parsing:

Configured Capacity: configuring capacity

Capacity used by DFS Used:DFS

Non DFS Used: capacity used by non-DFS

Remaining capacity of DFS Remaining:DFS

Data Statistics:

Configured Capacity = 1.92 TB = 1970 GB

DFS Used = 737.95 GB

Non DFS Used = 110.69 GB

DFS Remaining = 1.09TB = 1116.16 GB

DFS Used + Non DFS Used + DFS Remaining = 737.95 GB + 110.69 GB + 1116.16 GB = 1964.8 GB = Configured Capacity

So the result is:

The capacity correspondence shown by du-h and. / hdfs dfsadmin-report is:

Total capacity-> Configured Capacity

Used capacity-> DFS Used

Unused capacity-> DFS Remaining

Non-dfs files under capacity + / data are not displayed-> Non DFS Used

Non-dfs files refer to files that have been uploaded with the dfs file system and transferred through mv, cp, etc.

The statistics of 192.168.11.136 and 192.168.11.133 are similar.

3. Search for the missing 100g

Non DFS Used occupies 110.69GB 10GB for non-dfs files, so where is the rest of the 100GB?

View reserved space

# tune2fs-l / dev/mapper/VolGroup1-LVdata1

Reserved block count: 13107148

13107148 * 4 / 1024 / 1024 = 49.99

# tune2fs-l / dev/mapper/VolGroup2-LVdata2

Reserved block count: 13107148

13107148 * 4 / 1024 / 1024 = 49.99

The missing 100g is the sum of the reserved space of the two partitions.

The above is all the contents of the article "how to View HDFS File system Space usage". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.