Shulou(Shulou.com)05/31 Report--

Editor to share with you what is the use of distributed file system HDFS, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

The design of HDFS is based on the following six considerations:

(1) Fault tolerance

The hardware error of a stand-alone computer cannot be handled as an exception, but is in a normal state. There are many ordinary computer nodes in the HDFS file system, and any node may fail at any time, so HDFS should be designed to automatically recover and quickly detect errors, which should be the core goal of keeping HDFS running reliably.

(2) streaming access to datasets

Applications running on HDFS need streaming access to the stored dataset. These applications use parallel batch processing to calculate data, which is different from the applications used for data processing on ordinary systems. Improving data access throughput is the focus of HDFS research, while response time and data access latency are not considered too much.

(3) big data storage

The most basic goal of HDFS is to support big data storage. An ordinary file size stored on a HDFS system ranges from gigabytes to T bytes. The most basic requirement of a HDFS application is to support a large number of files.

(4) data consistency

The HDFS application handles files by writing once and reading multiple times. A single file does not need to be changed after it is written to HDFS. This way of dealing with files simplifies the problem of data consistency and can greatly improve the throughput of HDFS file access.

(5) Speed

The cost of mobile computing is lower than that of moving data. For a request that needs to be calculated, the closer the calculation is to the operational data, the more efficient the result will be, especially in the calculation of massive data.

(6) portability

Provide portability between heterogeneous software and hardware platforms. HDFS is developed by Java language, and the cross-platform feature of Java language perfectly solves this requirement.

HDFS uses the master/slave architecture. A HDFS cluster consists of a Namenode and a certain number of Datanode, these Datanode periodically communicate with Namenode, like Namenode feedback status and accept Namenode instructions [p. In order to reduce the burden on Namenode, it is not necessary to keep information about which blocks are contained on all Datanode permanently on Namenode, but to update the mapping table on Namenode through the block information reported by Datanode at startup. HDFS exposes the namespace of the file system on which users can store data in the form of files. Internally, a file is actually divided into one or more blocks (or at least one block), which are usually stored on multiple Datanode to ensure reliability and speed up later reads through redundancy. Datanode is responsible for handling the actual read and write data requests from distributed file system clients. Create, delete and copy data blocks under the unified scheduling of Namenode.

HDF S is a system that can store very large files in a large cluster, supports cross-machine storage, and runs reliably. Each file is divided into a series of data blocks with the size of 64MB (initially defined as 64, which can actually be set in the project, and related actions will be done later in the report), with the exception of the last data block. For fault tolerance, copies are stored when the data blocks of the file are stored in DataNode. In general, there will be three copies and placed in different places, the first copy will be placed on the local node, while the other node on the local rack will place the second copy, and the third copy will be copied and placed on different rack nodes. The advantage of this approach is that it can reduce the write traffic in the rack, improve the write performance, at the same time, you can freely configure the block size and replica factor of each file, and the application can specify the number of copies of a file. The copy factor can be established before the file is created or modified after the file is created.

Namenode is the master server of the whole distributed file system and the core of the whole system. From naming, we can clearly see that it is the name server of the whole system. Namenode, as the manager of file directory and file allocation in HDFS, the most important information it saves is the following two mappings: file name to data block and data block to Datanode list. Information from filenames to blocks is saved on disk (persisted); however, blocks are not saved to Datanode lists on Namenode. In the process of saving the file name to the data block, HDFS uses the action log to save the update to ensure that the entire structure does not need to be re-saved for each change. Now you can get the information that Namenode needs to store on Disk, including: the functionality of in_use.lock,fsimage and edits.in_use.lock is consistent with that of Datanode. Fsimage saves the directory tree of the file system, edits is the operation log on the file tree, and fstime is the last time the operation log was opened (long). In addition, Namenode performs file system namespace operations, such as opening, closing, and renaming files or directories. It is also responsible for determining the mapping of data blocks to specific Datanode nodes. Main functions of NAMENODE

(1) manage metadata and file blocks

Managing metadata is to manage the name space, file-to-file block mapping, and file-to-data node mapping it contains. Among them, the mapping of file names to data blocks is required not only to be retained in memory, but also to be persisted to disk. The management of file blocks includes not only the creation and replication of file blocks, but also the removal of invalid file blocks and the collection of isolated file blocks.

(2) simplify metadata update operations

To avoid having to re-save the entire structure every time, NameNode records each modification to the metadata of the file system and represents it as a transaction log (Editlog). Similarly, the operation of modifying the copy coefficient of the file is recorded in Editlog, while the Editlog is stored in the file system of the local operating system by NameNode. NameNode manages the namespace of the entire file system stored in FsImage files, including file attributes and block-to-file mapping. FsImage files, like Editlog, are stored in the local file system where NameNode resides. When NameNode starts, Editlog and FsImage are read from the hard disk, all transactions recorded by Editlog are applied to FsImage, and the newly generated Fslmage replaces the old Editlog to the hard disk.

(3) Monitoring and processing requests

Unlike the client and DataNode, NameNode only listens for client events and DataNode events. Instead of initiating requests, client events usually include directory and file creation, read and write, rename and delete, and file list information acquisition. DataNode events mainly include the reporting of block information, heartbeat response, error information and so on. When NameNode monitors these requests, it responds to them and returns the corresponding processing results to the requestor.

(4) heartbeat detection

After the connection is established, the heartbeat is maintained between DataNode and NameNode, so that DataNode will receive instructions from NameNode while reporting its load to NameNode. If any DataNode does not respond to NameNode's regular ping within a certain period of time, it will be considered a failure, and at that point, NameNode will readjust the entire file system.

Datanode is responsible for storing data, a block has backups in multiple Datanode, and a Datanode contains at most one backup for a block. So you can simply think of Datanode as storing block ID and block content, as well as their mapping. A HDFS cluster may contain thousands of Datanode nodes that regularly communicate with Namenode and receive instructions from Namenode. In order to reduce the burden on Namenode, the information about those data blocks on that Datanode is not permanently saved on Namenode, but the mapping table on Namenode is updated through the escalation when Datanode starts.

After Datanode and Namenode establish a connection, they keep their heartbeats with Namenode. When the heartbeat returns, it contains some Namenode commands to Datanode, such as deleting a block or copying a block from one Datanode to another Datanode. It should be noted that Namenode does not initiate requests to Datanode, and in this communication process, they are strict client / server architecture. Datanode, of course, also accepts access from the client as a server, handling block read / write requests. Datanode will also communicate with each other to perform block replication tasks. At the same time, when the client does write operations, Datanode needs to cooperate with each other to ensure the consistency of write operations.

Main functions of DATANODE:

(1) read and write data blocks.

The data blocks of all files are stored in DataNode, but the client does not know the specific location information of a data block, so the operation of the data block can not be done directly through DataNode. All this location information is stored in NameNode. Therefore, when the system client needs to create, copy and delete data blocks, it needs to first access NameNode to obtain the location information of the data blocks, and then access the specified DataNode to perform related operations. The specific file operations are finally completed by the client process rather than DataNode.

(2) report status to NameNode.

Since the mapping of data blocks to DataNode is not persisted to the local disk of NameNode, NameNode can only grasp the state of file blocks through the heartbeat signal between NameNode and DataNode, and then grasp the overall layout of the state of all DataNode nodes in the entire work cluster. If there is an abnormal state in DataNode, make adjustments in time. When a DataNode fails and causes failure, in order to ensure that the number of copies of data blocks reaches the specified range. NameNode will schedule the relevant DataNode to perform the replication of data blocks on the failed node.

(3) perform pipelined replication of data.

The system performs a block copy operation when a file is created or some DataNode fails, resulting in less than the specified number of block copies. After getting the list of data blocks to be replicated from NameNode, the client will first copy the data blocks cached in the client to the first DataNode, and then take the way of pipeline replication to copy the data on the first DataNode to the second DataNode at the same time, and so on, until all the data blocks are copied. Pipeline replication can effectively improve the running speed of the system, especially for some hot data need to copy a large number of Block, often can achieve very good results.

HDFS metadata caching strategy:

All metadata in HDFS is managed and maintained by Namenode. There are three types of metadata saved in Namenode: file and block namespaces, filename-to-block mapping, and the location of each block copy. Where the filename-to-block mapping is persisted in the disk soil, not just in memory. All updates to the directory tree and changes to the relationship between file names and data blocks must be persistent. To ensure that the entire structure does not need to be re-saved for each change, HDFS uses the action log to save updates. Any changes made to the file metadata are recorded by Namenode using a transaction log called Editlog. For example, if you delete a file in HDFS, Namenode will insert a record in Editlog to represent it; similarly, modify the copy of the file (the replication factor will also insert a record in Editlog. Namenode stores this Editlog in the file system of the local operating system. The namespace of the entire file system, including block-to-file mapping and file attributes, is stored in a file called FsImage, which is also placed on the file system of the operating system where Namenode resides.

Namenode holds the namespace and file name-to-block mapping of the entire file system in memory. This critical metadata is designed to be compact, and a Namenode with 4G memory is sufficient to support a large number of files and directories when storing large files. When Namenode starts, it reads Editlog and FsImage from the hard disk, applies all transactions in Editlog to the in-memory FsImage, flushes the new FsImage from memory to the hard disk, and then discards the old Editlog.

Data integrity policy:

Due to the complexity of the network, the data read by DataNode nodes may be damaged for a variety of reasons, such as network communication errors, program problems, DataNode node storage and so on. At this time, a set of data verification mechanism is needed to ensure the integrity of the data. When the client writes a new file, it calculates the checksum of each file block, saves the checksum as a separate hidden file, and stores the checksum file in the directory space at the same time. If the client wants to read the file, it will also read the check file in the directory space and match the check of the file block on the DataNode node. If there is an error in the match, then there is something wrong with the data of the file block, and the client will select the DataNode node where the copy of the file block resides to read the file block.

Copy placement policy:

In HDFS, 3 backups are saved for each data block by default (you can also modify the number of backups by yourself). In order to make later reading more efficient and ensure the reliability of the data, HDFS designs a corresponding replica placement strategy. In general, the first block copy is dumped with the node where the client is located (if the client is not within the cluster, the first node is selected at random. Of course, the system will try not to choose which node is too full or too busy. The second copy is placed in the node in the same rack as the first node (randomly selected). The third copy is randomly placed in a different node in a rack that is different from the first two copies. If there are more copies, they are randomly placed anywhere in the cluster. HDFS's replica placement strategy fully takes into account the reliability of the system (block in different racks) and bandwidth (a pipe only needs to cross one network node), and makes a good balance between them.

These are all the contents of the article "what is the use of distributed File system HDFS". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.