Shulou(Shulou.com)06/02 Report--

This article mainly explains "what are the architecture and design points in the Hadoop distributed file system". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's ideas to study and learn "what are the architecture and design points in the Hadoop distributed file system".

Hadoop distributed File system: architecture and Design Essentials

I. premise and design objectives

1. Hardware errors are normal, not abnormal. HDFS may be composed of hundreds of server, and any component may fail all the time, so error detection and fast and automatic recovery are the core architectural goals of HDFS.

2. The application of running on HDFS is different from the general application, they mainly focus on streaming reading and batch processing, and the high throughput of data access is more important than paying attention to the low latency of data access.

3. HDFS aims to support big data collections. A typical file size stored above is generally between gigabytes and T bytes. A single HDFS instance should support tens of millions of files.

4. HDFS applications require write-one-read-many access model for files. A file does not need to be changed after it has been created, written, and closed. This assumption simplifies the problem of data consistency and makes high-throughput data access possible. A typical MapReduce framework, or a webcrawler application, fits this model very well.

5. The cost of mobile computing is lower than that of mobile data. The calculation of an application request, the closer to the data it operates, the more efficient it is, especially when the data reaches a massive level. Moving the computation near the data is obviously better than moving the data to the application, which HDFS provides an interface to.

6. Portability between heterogeneous hardware and software platforms.

II. Namenode and Datanode

Hadoop distributed file system HDFS adopts master/slave architecture. A HDFS cluster consists of a Namenode and a certain number of Datanode. Namenode is a central server that manages the namespace of the file system and client access to files. Datanode is typically a node in a cluster and is responsible for managing the storage that accompanies them on the nodes. Internally, a file is actually divided into one or more block, and these block are stored in the Datanode collection. Namenode performs namespace operations on the file system, such as opening, closing, and renaming files and directories, while determining the mapping of block to specific Datanode nodes. Datanode creates, deletes and replicates block under the command of Namenode. Both Namenode and Datanode are designed to run on ordinary, cheap machines that run linux. HDFS is developed in the Java language, so it can be deployed on a wide range of machines. A typical deployment scenario is that one machine runs a single Namenode node, and the other machines in the cluster run a Datanode instance each. This architecture does not rule out running multiple Datanode on one machine, but this is relatively rare.

Single-node Namenode greatly simplifies the architecture of the system. Namenode is responsible for keeping and managing all HDFS metadata, so user data does not need to go through Namenode (that is, file data is read and written directly on the Datanode).

3. Namespace of the file system

HDFS supports traditional hierarchical file organization, and like most other file systems, users can create directories and create, delete, move, and rename files in between. HDFS does not support userquotas and access rights, nor does it support link, but the current architecture does not rule out the implementation of these features. Namenode maintains the namespace of the file system, and any changes to the file system namespace and file properties will be recorded by Namenode. The application can set the number of copies of files saved by HDFS. The number of copies of files is called the replication factor of files, and this information is also saved by Namenode.

IV. Data replication

How to implement data replication in Hadoop distributed file system? HDFS is designed to store large amounts of files reliably across machines in a large cluster. It stores each file as a block sequence, with all block being the same size except for one block. All block of the file is copied for fault tolerance. The block size and replication factor of each file are configurable. The Replication factor can be configured when the file is created or can be changed later. The file in HDFS is write-one, and it is strictly required that there is only one writer at any time. Namenode has full management of block replication, periodically receiving heartbeats and a Blockreport from each Datanode in the cluster. The reception of the heartbeat indicates that the Datanode node is working properly, and the Blockreport includes a list of all the block on the Datanode.

1. Copy storage, copy storage is the key to the reliability and performance of HDFS. HDFS uses a strategy called rack-aware to improve the reliability and validity of data and the utilization of network bandwidth. The short-term goal of this strategy is to verify its performance in a production environment, observe its behavior, and build the basis for testing and research in order to achieve a more advanced strategy. Large HDFS instances generally run on clusters of computers with multiple racks, and two machines between different racks need to communicate through a switch. obviously, usually, the bandwidth between two nodes in the same rack is greater than that between two machines in different racks.

Through a process called RackAwareness, Namenode determines the rackid to which each Datanode belongs. A simple but unoptimized strategy is to store copies on a separate rack. This prevents the failure of the entire rack (non-copy storage) and allows data to be read from multiple racks. This simple policy setting can distribute replicas in the cluster, which is beneficial to load balancing in the event of component failure. However, this simple strategy increases the cost of writing, because a write operation requires the transfer of block to multiple racks.

In most cases, the replication factor is 3JI HDFS. The storage strategy of HDFS is to store a copy on a node on a local rack, a copy on another node on the same rack, and * * a copy on a node on a different rack. Rack errors are far less than node errors, and this strategy will not affect the reliability and validity of the data. 1/3 copies are on one node, 2/3 are on one rack, and the rest are stored in the rest of the rack, which improves write performance.

2. Replica selection. In order to reduce the overall bandwidth consumption and read delay, HDFS will try its best to let reader read the most recent copy. If you have a copy on the same rack in reader, read that copy. If a HDFS cluster spans multiple data centers, reader will also first try to read a copy of the local data center.

3 、 SafeMode

After Namenode starts, it enters a special state called SafeMode, in which Namenode does not copy data blocks. Namenode receives heartbeats and Blockreport from all Datanode. Blockreport includes a list of all the blocks of a Datanode. Each block has a specified minimum number of copies. When the Namenode test confirms the minimum number of block copies of a Datanode, then the Datanode is considered secure; if a certain percentage of block detection (this parameter is configurable) confirms that it is secure, then Namenode exits the SafeMode state, and then it determines which other block copies do not reach the specified number of copies and replicates those block to other Datanode.

V. persistence of file system metadata

Persistence of Hadoop distributed file system file system metadata. Namenode stores metadata for HDFS. Namenode uses a transaction log called Editlog to record any changes to the file metadata. For example, if you create a file in HDFS, Namenode will insert a record in Editlog to represent it; similarly, modifying the replication factor of the file will also insert a record into Editlog. Namenode stores the Editlog in the file system of the local OS. The namespace of the entire file system, including the block-to-file mapping and file attributes, is stored in a file called FsImage, which is also placed on the file system of the Namenode system.

Namenode holds images of the entire file system namespace and the file Blockmap in memory. This critical metadata is designed to be compact, so a Namenode with 4G memory is sufficient to support a large number of files and directories. When Namenode starts, it reads Editlog and FsImage from the hard disk, apply all the transactions in Editlog in the FsImage in memory, and flush this new version of FsImage from memory to the hard disk, and then truncate the old Editlog, because the transactions of the old Editlog are already on the FsImage. This process is called checkpoint. In the current implementation, checkpoint occurs only when Namenode is started, and we will implement periodic checkpoint support in the near future.

Datanode doesn't know anything about the file, except to save the data in the file on the local file system. It stores each HDFS block in an isolated file on the local file system. Instead of creating all the files in the same directory, Datanode uses heuristics to determine the number of files in each directory and creates subdirectories when appropriate. Creating all files in the same directory is not an option, because the local file system may not be able to efficiently support a large number of files in a single directory. When a Datanode starts, it scans the local file system, generates a corresponding list of all HDFS blocks for these local files, and sends a report to Namenode, which is Blockreport.

VI. Communication protocol

All HDFS communication protocols are built on the TCP/IP protocol. The client connects to Namenode through a configurable port and interacts with Namenode through ClientProtocol. Datanode, on the other hand, uses DatanodeProtocol to interact with Namenode. Abstracting a remote call (RPC) from ClientProtocol and Datanodeprotocol, by design, Namenode does not initiate RPC actively, but responds to RPC requests from the client and Datanode.

Thank you for reading, the above is the content of "what are the architecture and design points in the Hadoop distributed file system". After the study of this article, I believe you have a deeper understanding of what the architecture and design points in the Hadoop distributed file system are, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.