Shulou(Shulou.com)05/31 Report--

This article shows you what the motivation and basic principles of Federation design are. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

HDFS Federation is a namenode horizontal expansion scheme proposed in Hadoop-0.23.0, the latest release of Hadoop, to solve the single point of failure of HDFS. This scheme allows HDFS to create multiple namespace to improve the scalability and isolation of the cluster. The following mainly introduces the design motivation and basic principles of HDFS Federation.

1. Current HDFS Overview 1.1 current HDFS architecture

The current HDFS consists of two layers:

(1) Namespace manages directories, files and data blocks. It supports common file system operations, such as creating files, modifying files, deleting files, etc.

(2) Block Storage consists of two parts:

Block Management maintains the basic relationship of datanode in the cluster. It supports block-related operations, such as creating data blocks, deleting data blocks, and so on. At the same time, it also manages replica replication and storage.

Physical Storage stores actual data blocks and provides read and write services for data blocks.

[these two parts of Block Storage are implemented on namenode and datanode respectively, so the module is completed by namenode and datanode]

The current HDFS architecture allows only one namespace to exist in the entire cluster, and that namespace is managed by only one namenode. This architecture makes HDFS very easy to implement, but it (see figure above) will have some ambiguities in the implementation process, which will lead to a lot of limitations (which will be described in detail below). Of course, these limitations only appear in companies with large clusters, such as baidu, Tencent and so on.

1.2 current HDFS limitations [Block Storage and namespace high coupling]

The combination of namespace and block management in the current namenode makes the two-tier architecture coupled together, making it difficult for other possible namenode implementations to use block storage directly.

[namenode Extensibility]

The underlying storage of HDFS can be scaled horizontally (explain: the underlying storage refers to datanode. When there is not enough storage space in the cluster, you can simply add machines to scale horizontally), but namespace cannot. The current namespace can only be stored on a single namenode, while namenode stores metadata information in the entire distributed file system in memory, which limits the number of blocks, files and directories in the cluster.

[performance]

The performance of file operations is limited by the throughput of a single namenode. Currently, a single namenode only supports about 60K task, while the next generation Apache MapReduce will support more than 100K concurrent tasks, implying support for multiple namenode.

[isolation]

Nowadays, most company clusters are shared, and different users from different group submit jobs every day. It is difficult for a single namenode to provide isolation, that is, the heavily loaded job submitted by one user will slow down the job of other users, and it is difficult for a single namenode to assign different jobs to different namenode according to application category like HBase.

2. Why does HDFS Federation2.1 adopt Federation

The main reason for adopting Federation is simplicity. Federation can quickly solve most of the problems of single Namenode.

The entire core design implementation of Federation took about 4 months. Most of the changes are in Datanode, Config, and Tools, while Namenode itself has very few changes, so that the original robustness of Namenode will not be affected. This makes the scheme compatible with previous versions of HDFS.

2.2 Federation architecture

Multiple independent namenode/namespace are used to scale namenode,federation horizontally. These namenode are United, that is, they are independent of each other and do not need to coordinate with each other, divide the labor and manage their own areas. Distributed datanode is used as a general-purpose block storage device. Each datanode registers with all the namenode in the cluster, periodically sends heartbeats and block reports to all namenode, and executes commands from all namenode.

A block pool consists of blocks belonging to the same namespace, and each datanode may store blocks of all block pool in the cluster.

Each block pool has internal autonomy, that is, it manages its own block and does not communicate with other block pool. One namenode is dead and will not affect other namenode.

The namespace on a namenode and its corresponding block pool are called namespace volume. It is the basic unit of management. When a namenode/nodespace is deleted, the corresponding block pool on all its datanode is also deleted. When a cluster is upgraded, each namespace volume is upgraded as a base unit.

2.3 key technical points of Federation [Namespace Management]

There are multiple namespaces in Federation, and how to divide and manage these namespaces is critical. In Federation and use the "file name hash" method, because the locality of this method is very poor, for example: look at the files under a directory, if you use the file name hash method to store files, then these files may be placed in different namespace, HDFS needs to access all namespace, the cost is too high. To facilitate the management of multiple namespaces, HDFS Federation uses the classic Client Side Mount Table.

As shown in the figure above, the following four dark triangles represent a separate namespace, and the light triangles above represent subnamespaces accessed from the customer's point of view. Each dark namespace Mount to a light-colored table, customers can access different mount points to access different namespaces, just as they can access different mount points in Linux systems. This is the basic principle of namespace management in HDFS Federation: by mounting each namespace into a global mount-table, you can do the data to global sharing; the same namespace is mounted to a personal mount-table, which becomes a view of the namespace visible to the application.

2.4 key benefits [scalability and isolation]

Supports multiple namenode to horizontally extend the namespace of the entire file system. Namespace volume can be separated according to the user and type of application, thereby enhancing isolation.

[general storage service]

The Block Pool abstraction layer opens the door to innovation for the architecture of HDFS. Separating the block storage layer results in:

The new file system (non-HDFS) can be built on block storage

New applications, such as HBase, can use the block storage layer directly

Separate block storage layers lay the foundation for fully distributed namespace in the future

[design is simple]

The entire core design implementation of Federation took about 4 months. Most of the changes are in Datanode, Config, and Tools, while the changes to Namenode itself are very small, so that the original robustness of Namenode is not affected. Although the scalability of this implementation is smaller than that of a real distributed Namenode, it can meet the requirements quickly. In addition, Federation has good backward compatibility, and the existing single Namenode deployment configuration can continue to work without any change.

3. Insufficient HDFS Federation [single point of failure]

HDFS Federation does not completely solve the problem of single point of failure. Although there are multiple namenode/namespace, from a single namenode/namespace point of view, there is still a single point of failure: if a namenode is down, the corresponding files it manages cannot be accessed. Each namenode in Federation is still equipped with a secondary namenode, as implemented on the previous HDFS, so that the main namenode can hang up and be used to restore metadata information.

[load balancing problem]

HDFS Federation uses Client Side Mount Table to share files and load, which requires more manual intervention to achieve the ideal load balancing.

The above content is what are the motivation and basic principles of Federation design. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.