Shulou(Shulou.com)06/02 Report--

This article shows you what the working mechanism of NameNode in the Hadoop framework is, the content is concise and easy to understand, and it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

1. Storage mechanism 1. Basic description

The NameNode runtime metadata needs to be stored in memory, and the fsImage of the metadata is backed up on disk. When the metadata is updated or added, modifying the metadata in memory will append the operation record to the edits log file. Query operations are not included here. If the NameNode node fails, the metadata can be reloaded into memory through the merge of FsImage and Edits, where SecondaryNameNode is dedicated to the merge of fsImage and edits.

2. Workflow

NameNode mechanism

After NameNode formatting starts, Fsimage and Edits files are created for the first time

Load FsImage image files and Edits logs into memory directly without first startup

The addition, deletion and modification of metadata by the client will be recorded in the Edits file.

Then the requested operation modifies the metadata in memory

SecondaryNameNode mechanism

Ask NameNode if CheckPoint,NameNode return information is required.

If SecondaryNameNode is required, request to execute CheckPoint

NameNode cuts the existing log file, and the new record scrolls into the new Edits file

Copy the edit log and image files before scrolling to SecondaryNameNode

SecondaryNameNode loads Edits logs and FsImage image files into memory merge

Generate a new image file fsimage.chkpoint and copy it to NameNode

NameNode renamed fsimage.chkpoint to fsimage

3. CheckPoint settings

Set up some SecondaryNameNode mechanisms by modifying the relevant configuration of the hdfs-default.xml file, such as every other hour.

Dfs.namenode.checkpoint.period 3600 dfs.namenode.checkpoint.txns 1000000 files full 1000000 records dfs.namenode.checkpoint.check.period 60 1 minute check the number of file records 2, file information 1, FsImage files

Metadata serialization backup information in NameNode memory

Generation path: based on NameNode node

Cd / opt/hadoop2.7/data/tmp/dfs/name/current/

View Fil

# basic syntax hdfs oiv-p convert file type-I image file-o file output path after conversion

Based on the syntax format, manipulate the file in the figure above:

# convert file hdfs oiv-p XML-I fsimage_0000000000000000019-o / data/fsimage.xml# to view cat / data/fsimage.xml

So you can see some information about the metadata.

2. Edits file

The path of all additions, deletions and modifications to the HDFS file will be recorded in the Edits file.

Basic grammar

Hdfs oev-p converted file type-I log file-o converted file output path

View Fil

# convert file hdfs oev-p XML-I edits_0000000000000000020-00000000000000020-o / data/edits.xml# to view cat / data/edits.xml III, failure recovery 1, copy SecondaryNameNode data

First end the NameNode process

Delete data stored in NameNode

[root@hop01 /] rm-rf / opt/hadoop2.7/data/tmp/dfs/name/*

Copy the data from SecondaryNameNode to the NameNode data storage directory

# Note that the SecondaryNameNode service is configured on hop03 [root@hop01 /] scp-r root@hop03:/opt/hadoop2.7/data/tmp/dfs/namesecondary/* / opt/hadoop2.7/data/tmp/dfs/name/

Restart the NameNode process

2. Based on Checkpoint mechanism

Modify hdfs-site.xml configuration, synchronize to cluster-related services, and restart the HDFS process

Dfs.namenode.checkpoint.period 120 dfs.namenode.name.dir / opt/hadoop2.7/data/tmp/dfs/name

End the NameNode process

Delete data stored in NameNode

[root@hop01 /] rm-rf / opt/hadoop2.7/data/tmp/dfs/name/*

Since SecondaryNameNode (in hop03) and NameNode (in hop01) in the cluster are not on the same host node, you need to copy the directory where SecondaryNameNode stores data to the level directory where NameNode stores data, and delete the in_use.lock file

[root@hop01 /] scp-r root@hop03:/opt/hadoop2.7/data/tmp/dfs/namesecondary / opt/hadoop2.7/data/tmp/dfs/ [root@hop01 namesecondary/] rm-rf in_ use.lock [root @ hop01 dfs] $lsdata name namesecondary

Import checkpoint data

[root@hop01 hadoop2.7] bin/hdfs namenode-importCheckpoint

Restart NameNode

[root@hop01 hadoop2.7] sbin/hadoop-daemon.sh start namenode IV. Multiple directory configuration

NameNode can configure multiple local directories, and each directory stores the same content, which increases the reliability of operation.

1. Add configuration # vim / opt/hadoop2.7/etc/hadoop/hdfs-site.xml# add the following content: dfs.namenode.name.dir file:///${hadoop.tmp.dir}/dfs/name01,file:///${hadoop.tmp.dir}/dfs/name02

This configuration needs to synchronize all services under the cluster.

2. Delete the original data

All services under the cluster need to perform this operation.

[root@hop01 hadoop2.7] # rm-rf data/ logs/

Restart the cluster service after formatting NameNode.

Safety model 1. Basic description

When NameNode starts, it loads the image of file system metadata in memory based on the image file and editing log, and then starts listening for DataNode requests. During this process, the client cannot upload files in a read-only security mode. In this safe mode, DataNode will send the latest block list information to NameNode. If the minimum copy condition is met, NameNode will exit safe mode after a specified time.

2. Safety mode

Safe mode status

/ opt/hadoop2.7/bin/hdfs dfsadmin-safemode get

Enter safe mode

/ opt/hadoop2.7/bin/hdfs dfsadmin-safemode enter

Exit safe mode

/ opt/hadoop2.7/bin/hdfs dfsadmin-safemode leave

Wait for safe mode

/ opt/hadoop2.7/bin/hdfs dfsadmin-safemode wait the above is how NameNode works in the Hadoop framework. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.