Shulou(Shulou.com)05/31 Report--

This article mainly explains "Hadoop cluster how to manage Namenode directory data structure", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "Hadoop cluster how to manage Namenode directory data structure" bar!

I. dfs directory structure

[root@master hadoop] # pwd/opt/hadoop-2.7.1/etc/hadoop [root@master hadoop] # cat core-site.xml hadoop.tmp.dir / opt/hadoop-2.7.1/tmp fs.defaultFS hdfs://master.harry.com:9000 [root@master tmp] # pwd/opt/hadoop-2.7.1/tmp [root@master tmp] # tree dfs/-L 2dfs/ |-- data | `--current`-- name`-- current4 directories 0 files [root@master current] # pwd/opt/hadoop-2.7.1/tmp/dfs/name/current [root@master current] # lltotal 1076 Murray RW Murray. 1 root root 207 Oct 14 22:39 VERSION-rw-r--r--. 1 root root 14381 Oct 14 23:33 edits_0000000000000000001-000000000000127Murray RW Murray. 1 root root 42 Oct 15 00:33 edits_0000000000000000128-000000000000129Murray RW Murray. 1 root root 42 Oct 15 01:33 edits_0000000000000000130-000000000000131Murray RW Murray. 1 root root 42 Oct 15 02:33 edits_0000000000000000132-000000000000133 Murray RW Murray. 1 root root 1048576 Oct 15 02:33 edits_inprogress_0000000000000000134-rw-r--r--. 1 root root 1809 Oct 15 01:33 fsimage_0000000000000000131-rw-r--r--. 1 root root 62 Oct 15 01:33 fsimage_0000000000000000131.md5-rw-r--r--. 1 root root 1809 Oct 15 02:33 fsimage_0000000000000000133-rw-r--r--. 1 root root 62 Oct 15 02:33 fsimage_0000000000000000133.md5-rw-r--r--. 1 root root 4 Oct 15 02:33 seen_txid

2. Analysis

Edits_* file: when we modify the metadata, such as adding block, adding nodes, and so on, if we modify the namespace, the data will be temporarily stored in the edits file.

Fsimage file: stores the data managed by the actual namenode. Periodically formed by secondary namenode and parallel edits files and old fsimage files.

Seen_txid: record the number of the last edits file as in this example. Because the edits file ends up to 133. When the system restarts, the last digits of the seen_txid and edits files are checked to indicate whether there is a missing metadata.

[root@master current] # cat seen_txid 134

VERSION file:

[root@master current] # cat VERSION # Wed Oct 14 22:39:48 PDT 2015namespaceID=1916027777clusterID=CID-f0ee4204-f7a1-4e8d-9799-bdd5e23259f1cTime=0storageType=NAME_NODEblockpoolID=BP-1637593683-172.16.114.202-1444887588098layoutVersion=-63 Thank you for reading. This is the content of "how the Hadoop cluster manages the directory data structure of Namenode". After studying this article, I believe you have a deeper understanding of how the Hadoop cluster manages the directory data structure of Namenode. The specific use situation still needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.