Shulou(Shulou.com)06/02 Report--

This article focuses on "HDFS Architecture and Design principles". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "HDFS Architecture and Design principles"!

1 preface

HDFS (Hadoop Distributed File System) is a well-known Hadoop distributed file system, which is a highly fault-tolerant system that can provide high-throughput data access and is very suitable for applications on large-scale data sets.

2 characteristics of HDFS

Streaming data access: this may be difficult for beginners to understand. Streaming data access mode is one of the characteristics of HDFS data access, which means that the data is processed as soon as part of the data is collected, rather than all the data is collected and then processed, otherwise it will lead to a large delay and consume a lot of memory.

Large datasets: HDFS is friendly to large file storage, and a typical file size on HDFS is generally at the GB to TB level.

Write once read multiple read: one of the HDFS data access features that cannot be changed after a file has been created, written, and closed. This also simplifies data consistency issues and makes high-throughput data access possible.

Low-latency data access is not supported: this is also the characteristic of HDFS data access. HDFS is related to high throughput and is not suitable for applications with low-latency data access.

Single-user write, does not support arbitrary modification: the data of HDFS is mainly read, only a single writer is supported, and the write operation is always appended at the end of the text in the form of addition, and modification is not supported at any location.

3 HDFS architecture

HDFS uses the Master/Slave architecture. A HDFS cluster has two important roles, Namenode and Datanode. Namenode is the management node that manages the namespace (namespace) of the file system and client access to files. The Datanode is the node where the data is actually stored. HDFS exposes the namespace of the file system on which users can manipulate data in the form of manipulation files. The HDFS architecture diagram is as follows:

Files on the HDFS are stored in blocks, which are usually stored on a set of Datanode. Namenode performs namespace operations on the file system, such as opening, closing, and renaming files or directories. It is also responsible for determining the mapping of data blocks to specific Datanode nodes. Datanode handles read and write requests from file system clients and performs block creation, deletion and replication under the unified scheduling of Namenode.

4 data block and replica mechanism

HDFS is designed to store very large files reliably across machines in a large cluster. It stores each file as a series of blocks, all of which are of the same size except the last one. For fault tolerance, all blocks of the file will have copies. The block size and replica factor for each file are configurable. An application can specify the number of copies of a file. The copy factor can be specified when the file is created or can be changed later. Files in HDFS are written at once, and it is strictly required that there can only be one writer at any one time.

Namenode oversees block replication and periodically receives heartbeats and block status reports (Blockreport) from each Datanode in the cluster. Receiving a heartbeat signal means that the Datanode node is working properly. The block status report contains a list of all the blocks on the Datanode.

5 rack awareness and replica storage

Replica storage is the key to the reliability and performance of HDFS. HDFS optimizes this aspect by using a strategy called rack awareness (Rack Awareness) to improve data reliability, availability and network bandwidth utilization.

Large HDFS instances typically run on a cluster of computers that span multiple racks, and communication between two machines on different racks needs to go through a switch. In most cases, the bandwidth between two machines in the same rack is greater than that between two machines in different racks.

Through a rack-aware process, Namenode can determine the rack id to which each Datanode belongs. A simple but unoptimized strategy is to store copies on different racks. This effectively prevents the loss of data when the entire rack fails, and allows you to make full use of the bandwidth of multiple racks when reading data. This policy setting can distribute replicas evenly in the cluster, which is beneficial to load balancing in the case of component failure. However, because one write operation of this strategy requires the transfer of data blocks to multiple racks, this increases the cost of writing.

In most cases, the replica factor is 3, and the HDFS storage strategy is to store one copy on a node in the local rack, one copy on another node on the same rack, and the last copy on a different rack node. This strategy reduces data transfer between racks, which improves the efficiency of write operations. Rack errors are far fewer than node errors, so this strategy does not affect the reliability and availability of data. At the same time, because the data blocks are placed on only two (not three) different racks, this strategy reduces the total network transmission bandwidth required to read the data. This strategy improves write performance without compromising data reliability and read performance.

6. HDFS metadata management

The namespace of HDFS is saved on Namenode. Namenode records any changes to the file system metadata using a transaction log called EditLog. For example, if you create a file in HDFS, Namenode will insert a record in Editlog to represent it; similarly, changing the copy factor of the file will also insert a record into Editlog. Namenode stores this Editlog in the file system of the local operating system. The namespace of the entire file system, including block-to-file mapping, file attributes, and so on, is stored in a file called FsImage, which is also placed on the local file system where Namenode resides.

Namenode holds the namespace and file block mapping (Blockmap) image of the entire file system in memory. When Namenode starts, or when checkpoints are triggered periodically, it reads Editlog and FsImage from the hard disk, acts all transactions in Editlog on the in-memory FsImage, and persists this new version of FsImage from memory to the local disk. HDFS then deletes the old Editlog because the transactions of the old Editlog are already working on the FsImage. This process is called a checkpoint. Changes to Editlog are applied to FsImage during the checkpoint. The checkpoint trigger time can be triggered at a given time interval (dfs.namenode.checkpoint.period, in seconds) or after a given number of file system transactions have accumulated (dfs.namenode.checkpoint.txns). If these two properties are set, the first threshold to be reached triggers the checkpoint.

At this point, I believe you have a deeper understanding of "HDFS architecture and design principles". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.