Shulou(Shulou.com)05/31 Report--

This article is to share with you about the basics and goals of HDFS design. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Design fundamentals:

(1) because hardware errors are normal. Therefore, redundancy is needed.

(2) streaming data access, that is, batch reading of data, rather than random reading. Hadoop is good at data analysis rather than transaction processing.

(3) large-scale data set

(4) simple consistency model. In order to reduce the complexity of the system, the logical design of the file is written and read many times at one time, that is, once the file is written and closed, it can no longer be modified.

(5) the program uses the principle of "data nearest" to assign nodes to execute.

Architecture:

Hdfs adopts master-slave structure, Namenode belongs to the master end and Datanode belongs to the slave end.

Namenode:

1) manage the namespace of the file system.

2) record the location and copy information of each file block on the Datanode.

3) coordinate client access to files.

4) record changes in the namespace or changes to the attributes of the space itself.

5) Namenode uses transaction logs to record changes in HDFS metadata. Use image files to store the namespace of the file system, including file mappings, file attributes, and so on.

From a sociological point of view, Namenode is the manager of HDFS, playing the role of management, coordination and control.

Datanode:

1) responsible for the storage management of the physical node.

2) write once, read many times, and cannot be modified.

3) the file consists of a database. In general, the size of the data block is 64MB.

4) the data is distributed to each node as much as possible.

From a sociological point of view, Datanode is a worker of HDFS, who works according to the orders of Namenode and feeds back the progress and problems of work to Namenode.

SecondaryNameNode:

1) A solution for HA. But hot backup is not supported. Just configure it.

2) execution process: download metadata information (fsimage,edits) from NameNode, then merge the two to generate a new fsimage, save it locally, and push it to NameNode, while resetting the edits of NameNode.

3) it is installed on the NameNode node by default, but this way. Unsafe!

Reliability:

(1) redundant copy strategy

(2) Rack strategy

(3) heartbeat mechanism

(4) Safety mode

(5) checksum

(6) Recycle Bin

(7) metadata protection

(8) Snapshot mechanism

Thank you for reading! This is the end of the article on "what is the basis and goal of HDFS design". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.