Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what is the composition of HDFS architecture". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Start with RAID.

The main problem to be solved by big data technology is the calculation and processing of large-scale data, so the first thing to solve is the storage of large-scale data. There are three core issues to be solved for large-scale data storage:

The problem of data storage capacity, since big data wants to solve the data calculation problem of several PB, and the general server disk capacity is usually 1-2TB, then how to store such a large scale of data.

For the problem of data reading and writing speed, generally the continuous reading and writing speed of disk is dozens of MB. At this speed, dozens of PB of data may have to be read and written forever.

For the problem of data reliability, the disk is probably the most easily damaged hardware in the computer equipment. The service life of a disk on the website is about one year. If the disk is damaged, what about the data?

Before the emergence of big data technology, people need to face these storage problems, and the corresponding solution is RAID technology.

RAID (redundant Array of Independent disks) technology is mainly to improve the storage capacity, read and write speed, and enhance the availability and fault tolerance of disks. At present, server-level computers support the insertion of multiple disks (8 or more). By using RAID technology, data can be read, written and backed up concurrently on multiple disks.

There are several common RAID technologies, as shown in the figure.

Schematic diagram of commonly used RAID technology

Suppose the server has N disks.

RAID0

When the data is written to the disk from the memory buffer, the data is divided into N parts according to the number of disks, and these data are simultaneously written to N blocks of disk, so that the overall data writing speed is N times that of a disk. It is the same when reading, so RAID0 has extremely fast data read and write speed, but RAID0 does not do data backup, as long as one of the N disks is damaged, the data integrity will be destroyed, and the data of all disks will be damaged.

RAID1

When data is written to disk, one piece of data is written to two disks at the same time, so that any disk damage will not lead to data loss. Inserting a new disk can be automatically repaired by copying data, with high reliability.

RAID10

Combined with the two schemes of RAID0 and RAID1, all disks are divided into two parts averagely, and the data is written on two disks at the same time, which is equivalent to RAID1. But in each disk, RAID0 technology is used to read and write concurrently, which not only improves the reliability but also improves the performance. However, the disk utilization of RAID10 is low, and half of the disks are used to write backup data.

RAID3

In general, two disks will not be damaged on one server at the same time. In the case of damaging only one disk, if the data of other disks can be used to recover the data of damaged disks, this ensures reliability and performance, and disk utilization is also greatly improved.

When the data is written to the disk, the data is divided into NMY 1, and the NMY 1 disk is written concurrently, and the check data is recorded on the Nth disk. Any disk damage (including the parity data disk) can be repaired by using the data of other NMY 1 disks.

However, in the scenario with more data modification, any disk modification will cause the Nth disk to rewrite the check data. The consequence of frequent writing is that the Nth disk is easier to be damaged than other disks and needs to be replaced frequently, so RAID3 is rarely used in practice.

RAID5

The more commonly used solution is RAID5 than RAID3.

RAID5 and RAID3 are very similar, but the parity data is not written to the Nth disk, but spirally written to all disks. In this way, the modification of the check data is also averaged on all disks, avoiding the frequent write-down of a disk by RAID3.

RAID6

If the data needs high reliability, in the case that two disks are damaged at the same time (or the operation and maintenance level is relatively backward, one disk is broken but has not been replaced for a long time, resulting in another disk being broken), you still need to repair the data, and you can use RAID6 at this time.

RAID6 is similar to RAID5, but the data is written to only NMel 2 disks and writes parity information spirally on both disks (generated using different algorithms).

With the same number of disks (N), the comparison of the various RAID technologies is shown in the table below.

Comparison of several RAID techniques

RAID technology has hardware implementation, such as special raid card or motherboard direct support, can also be implemented through software, in the operating system level, multiple disks constitute RAID, in logic as an access directory. RAID technology is widely used in traditional relational database and file system, and it is an important means to improve computer storage characteristics.

RAID technology only forms an array on multiple disks on a single server, and big data needs larger storage space and access speed. Applying the principle of RAID technology to the distributed server cluster, the architecture idea of Hadoop distributed file system HDFS is formed.

Principle of HDFS Architecture

Like RAID for file storage and parallel reading and writing on multiple disks, HDFS reads and writes data in parallel and redundant storage on a large-scale distributed server cluster. Because HDFS can be deployed on a large server cluster, and the disks of all servers in the cluster can be used by HDFS, the storage space of the entire HDFS can reach PB-level capacity. The HDFS architecture is shown in the figure.

HDFS architecture

There are two key components in HDFS, one is NameNode and the other is DataNode.

DataNode is responsible for the storage, reading and writing of file data. HDFS divides the file data into several block, and each DataNode stores part of the block, so that the files are distributed in the entire HDFS server cluster. Application client (Client) can access these data blocks in parallel, so that HDFS can access data in parallel on the scale of server cluster, which greatly improves the access speed. In practice, there are many DataNode servers in a HDFS cluster, usually hundreds to thousands of servers. Each server is equipped with several disks, and the storage capacity of the entire cluster ranges from a few PB to hundreds of PB.

NameNode is responsible for the MetaData management of the whole distributed file system, that is, the file path name, the ID of the data block and the storage location and other information, and undertakes the role of the file allocation table (FAT) in the operating system. In order to ensure the high availability of data, HDFS copies a block into multiple copies (the default is 3 copies) and stores three copies of the same block on different servers. In this way, when a disk is damaged or a DataNode server goes down so that its stored block cannot be accessed, Client will find its backup block for access.

Block multiple copy storage is shown in the following figure. For file / users/sameerp/data/part-0, the number of replicated backups is set to 2, and the stored block id is 1pm 3. Two backups of block1 are stored on DataNode0 and DataNode2 servers, and two backup stores of block3 are stored on DataNode4 and DataNode6 servers. After any of these servers are down, there is at least one backup for each block, which does not affect access to files / users/sameerp/data/part-0.

HDFS's block replication backup strategy

In fact, DataNode will communicate with NameNode through heartbeat. If DataNode timeout does not send heartbeat, NameNode will think that the DataNode has failed, immediately find out which block is stored on this DataNode and on which other servers these block are stored, and then tell these servers to copy another block to other servers to ensure that the number of block backups stored in HDFS meets the number set by users, even if the server is down again. And no data will be lost.

HDFS application

Hadoop distributed file systems can be accessed like normal file systems: read and write files using the command line or the programming language API. Let's take HDFS writing as an example to see the HDFS process, as shown in the following figure.

HDFS write file operation

The application Client calls HDFS API, requesting the creation of a file, and the HDFS API is included in the Client process.

HDFS API sends the request parameters to the NameNode server, and NameNode creates a file path in the meta information and looks for free block in the DataNode. Then the id of the idle block and the corresponding DataNode server information are returned to Client. Because the data block requires multiple backups, even if Client only needs the amount of data of one block, NameNode will return multiple NameNode information.

Client calls HDFS API and requests that the data flow be written out.

The HDFS API connects to the first DataNode server and sends the Client data stream to the DataNode, which writes the data to the local disk while sending it to the second DataNode. Similarly, the second DataNode records the data and sends it to the third DataNode.

Client informs NameNode that the file has been written, and NameNode marks the file as normal and ready to read.

Although HDFS provides API, in practice, we seldom program to read data in HDFS directly. As mentioned at the beginning, mobile computing is more cost-effective than mobile data in big data scenario. Instead of writing a program to read data distributed on so many DataNode, it is better to distribute the program to DataNode to access the block data on it. But how to distribute the program? How do distributed programs access data on HDFS? How to deal with the results of the calculation, and if the results need to be merged, how to merge?

Hadoop provides a framework for parallel computing of large-scale data stored on HDFS, which is what we mentioned earlier as MapReduce.

This is the end of the content of "what is the composition of HDFS architecture"? thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.