Shulou(Shulou.com)06/03 Report--

1. HDFS

Permanent data structure

1.1 directory structure of namde

[root@datanode1 name] # cd / data0/hadoop/dfs/name/current/ [root@datanode1 current] # lsedits edits.new fsp_w_picpath fstime VERSION [root@datanode1 current] # ls-l total consumption 56 RW RW Murray. 1 hadoop hadoop 789 January 15 16:59 edits-rw-rw-r--. 1 hadoop hadoop 1049088 January 15 18:00 edits.new-rw-rw-r--. 1 hadoop hadoop 14557 January 14 18:47 fsp_w_picpath-rw-rw-r--. 1 hadoop hadoop 8 January 14 18:47 fstime-rw-rw-r--. 1 hadoop hadoop 14 January 18:47 VERSION

1.1.2 the VERSION file is a JAVA attribute that contains the version information of the running HDFS

[root@datanode1 current] # cat VERSION # Thu Jan 14 18:47:15 CST 2016namespaceID=688384215cTime=0storageType=NAME_NODElayoutVersion=-32

LayoutVersion is a negative integer that describes the version of the HDFS persistent data structure (also known as layout), but the version number is independent of the version number of the hadoop release package. Whenever the layout changes, the version number is incremented (for example, the version number-18 is followed by-19), and HDFS also needs to be upgraded. Otherwise, the disk will still use the old version layout, and the new version of namenode or datanode will not work properly.

NamespaceIT: is the unique identifier of the file system and is set when the file system is first formatted.

CTime: marks the creation time of the namenode storage system. For the storage system that has just been formatted, the property value is 0

StorageType: indicates that the storage directory contains the data structure of namenode

1.1.3 File system images and editing logs

When the file system client performs write operations, such as creating and moving files, these operations are first recorded in the edit log. Namenode maintains file system metadata in memory; when the edit log is modified, the relevant metadata information also needs to be updated. The read request of the client for the number of elements in memory.

The editing log needs to be updated and synchronized after each write operation and before the successful code is sent to the client. When namenode writes to multiple directories, the code is returned only after all writes have been performed to ensure that no operation is lost due to machine failure.

Fsp_w_picpath: is a permanent checkpoint of file system metadata. If namenode fails, you can first load the fsp_w_picpath file into memory to reconstruct the recent metadata, and then perform various operations of editing log records.

Fsp_w_picpath contains serialization information for all directories and file inode in the file system. Each inode is a memory description of the metadata of a file or directory. For files, it contains information such as "copy level" (replication level), modification time and access time, access permission, block size, forming a file block, etc.; for directories, it contains information such as modification time, access permission and quota metadata.

Blocks are stored in datanode, but the fsp_w_picpath file does not describe datanode. Instead, namenode keeps this block mapping in memory. When datanode joins the cluster, namenode asks datanode for a list of blocks to establish a mapping relationship; namnode also periodically consults datanode to ensure that it has the latest block mapping.

Run the secondary namenode to create a checkpoint of the number of file system elements in the memory of the primary namenode

(1) the secondary namenode requests the primary namenode to stop using the edits file and temporarily record the new write operation in a new file.

(2) Auxiliary namenode obtains fsp_w_picpath and edits files from the main namenode (using HTTP GET)

(3) assist namenode to load fsp_w_picpath files into memory, perform operations in edits files one by one, and create new fsp_w_picpaths files.

(4) Secondary namenode sends the new fsp_w_picpath file back to the primary namenode (using HTTP POST)

(5) the main namenode replaces the old fsp_w_picpath file with the fsp_w_picpath file received from the secondary namenode; the old edits file is replaced with the edits file generated by Buju 1. At the same time, the ftime file is updated to record the checkpoint execution time.

The trigger condition for creating a checkpoint is controlled by two configuration parameters.

(1) Secondary namenode every hour (set by the fs.checkpoint.period property in seconds)

(2) when the edit log size reaches 64MB (set by the fs.checkpoint.size property, in bytes), a checkpoint is created even if it is less than an hour. The system checks the edit log size every 5 minutes.

1.2 the directory structure of the secondary namenode

[root@slave-two current] # pwd/data0/hadoop/dfs/data/current [root@slave-two current] # cat VERSION # Fri Jan 1515: 34:22 CST 2016namespaceID=688384215storageID=DS-1030151558-10.1.2.216-50010-1452481280886cTime=0storageType=DATA_NODElayoutVersion=-32

In the event of a failure of the primary NameNode (assuming there is no timely backup, not even on the NFS), data can be restored from the secondary namenode. Two implementation methods

(1) copy the relevant storage directory to the new namenode

(2) use the-importCheckpoint option to start the namenode daemon, and the slave namenode is used as the new master namenode. With this option, when there is no metadata in the dfs.name.dir attribute definition directory, the secondary namenode intercepts the latest checkpoint data from the fs.checkpoint.dir directory, otherwise execution fails

1.3 directory structure of datanode

The storage directory of datanode is created automatically in the initial phase and no additional formatting is required.

[root@slave-one current] # lsblk_-1342046564177101301 blk_3255346014128987307 blk_-4378222930931288631 blk_7478159877522346339 blk_-8475713792677154223blk_-1342046564177101301_1004.meta blk_3255346014128987307_1010.meta blk_-4378222930931288631_1065.meta blk_7478159877522346339_1002.meta blk_-8475713792677154223_1063.metablk_-1859875086242295767 blk_3484901243420393976 blk_5202437766650751967 blk_7579826132350507903 Blk_-9058686418693604829blk_-1859875086242295767_1061.meta blk_3484901243420393976_1067.meta blk_5202437766650751967_1072.meta blk_7579826132350507903_1080.meta blk_-9058686418693604829_1062.metablk_253660519371394588 blk_-350256639016866731 blk_5450455005443823908 blk_774901497839428573 dncp_block_verification.log.currblk_253660519371394588_1014.meta blk_-350256639016866731_1077.meta blk_5450455005443823908_1076.meta blk_774901497839428573_1068.meta VERSIONblk _ 2653614491429524571 blk_-4332403947618992681 blk_6996247191717220870 blk_7996063171811697628blk_2653614491429524571_1066.meta blk_-4332403947618992681_1012.meta blk_6996247191717220870_1064.meta blk_7996063171811697628_ 1013.metastatic [root @ slave-one current] # pwd/data0/hadoop/dfs/data/current [root@slave-one current] # cat VERSION # Fri Jan 15 15:34:16 CST 2016namespaceID=688384215storageID=DS-444750413-10.1.2.215-50010-1452481260852cTime=0storageType=DATA_NODElayoutVersion=-32

Other files in datanode's current directory have the blk_ prefix, including two file types: HDFS block files (raw data only) and block metadata (with the .meta suffix). The block file contains a portion of the raw data in the stored file; the metadata file includes the header (with version and type information) and a series of checksums for each section of the block

When the amount of directory data increases to a certain size, datanode creates a subdirectory to hold new blocks and metadata information. If you store 64 blocks (set by the dfs.datanode.numblocks attribute), create a subdirectory

If the dfs.data.dir attribute specifies multiple directories on different disks, the blocks are written to each directory in a round-robin manner. Note that blocks on each disk on the same datanode are not duplicated, and blocks between different datanode can be duplicated.

two。 Safety mode

When namenode starts, first load fsp_w_picpath into memory and perform various operations in edits. Once the image of the file system metadata is successfully established in memory, a new fsp_w_picpath file (which does not require a secondary namenode) and an empty edit log are created. At this point, namenode starts listening for RPC and HTTP requests. At the moment, however, namenode is in safe mode, meaning that namenode's file system is read-only to the client.

Enter and leave safe mode

[hadoop@slave-one current] $hadoop dfsadmin-safemode getSafe mode is ON

HDFS's web page can also show whether namenode is in safe mode.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.