Shulou(Shulou.com)06/01 Report--

This article is about how to use Router-Based HDFS Federation in Didi big data. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

I. background

HDFS's Master/Slave architecture makes it have a single bottleneck, that is, with the large-scale expansion of business data, Master nodes will have bottlenecks in metadata storage and services. In order to overcome the scalability, performance and isolation problems of HDFS single point bottleneck, the community put forward a Federation (https://issues.apache.org/jira/browse/HDFS-1052) scheme to solve it.

However, after using this scheme, the problem exposed to the customer is that there are multiple namespaces (namespace) in the same cluster, and the customer needs to know under which namespace the read and write data can operate. In order to solve the problem of unified namespace, the community has put forward a client-side-based solution ViewFS (https://issues.apache.org/jira/browse/HADOOP-7257), which will be configured on the client side, and the user directory will be mounted to a specific namespace directory one-to-one. Didi uses this solution to solve the Federation problem.

There are also some problems with the ViewFS scheme:

It is difficult to upgrade the client that has been released.

For the new catalog, you need to increase the mount configuration, and it is difficult to connect with the product and maintain it.

In versions 2.9 and 3.0, the community released a new solution to the problem of unified namespace, Router-Based Federation (https://issues.apache.org/jira/browse/HDFS-10467), which is implemented on the server side and is easy to maintain in terms of upgrade management. Didi recently introduced this scheme and made some improvements.

II. Introduction of Router-Based Federation scheme

Router-Based Federation provides Router services externally, which are included in Federation layer, as shown in the following figure. This Router service will allow users to access any subcluster transparently, allowing the subcluster to manage its own Blockpool independently. To achieve these goals, Federation layer must direct Block access to the appropriate subcluster. At the same time, it has scalability, high availability and fault tolerance.

Federation layer contains multiple components. Router is a component with the same interface as Namenode that forwards client requests to the correct subcluster based on the metadata information of State Store. The State Store component contains the remotely mounted table (which has the ViewFS feature but is shared between clients) and load / spatial information about SubCluster.

The architecture below shows the addition of Router (marked "R") and logically centralized (but physically distributed) state storage (State Store) for each subcluster, as well as Namenodes ("NN") and Datanodes ("DN") for each SubCluster. This approach has the same architecture as YARN Federation (YARN-2915).

2.1 Router components

There can be multiple Router in the system, and each Router has two roles:

Provides a global Namenode interface to the client and is responsible for forwarding the Active Namenode in the correct subcluster

Maintain information about Namenode in State Store.

After receiving the client request, Router finds the correct subcluster based on the information in mount-table, and then forwards the request for that cluster to the corresponding subcluster Active Namenode. After receiving the response result from Active Namenode, the result is returned to the client. To improve performance, Router can cache remotely mounted table entries and the status of subclusters.

For the maintenance of Namenode information, Router periodically checks the status of an Namenode and reports its high availability (HA) status and load / space status to State Store. To improve the performance of Namenode HA, Router uses high availability status information from State Store to forward requests to the Namenode that is most likely to be active.

2.1.1 availability and fault tolerance

Router is stateless, and all Router provide services at the same time. If a Router becomes unavailable, it does not affect any other Router services.

Clients configure their DFS HA clients, such as ConfiguredFailoverProvider or RequestHedgingProxyProvider, to work with all Router in Federation.

For high availability and flexibility, multiple Router can monitor the same Namenode and send heartbeats to the State Store. If the Router fails, this increases the resilience of the information.

2.1.2 Safe Mode

If the Router cannot connect to the State Store, it may mistakenly provide access to the expired locations, leaving the Federation in an inconsistent state.

To prevent this from happening, when Router cannot connect to the State Store for a period of time, it goes into safe mode (similar to Namenode's safe mode). When the client tries to access safe mode's Router, an exception is thrown, and after the client's Proxy is captured, it attempts to connect to another Router. Similar to Namenode,Router, it remains in this safe mode until it determines that State Store is available.

This prevents inconsistencies when Router starts. Suppose an Router is dead or in safe mode if it does not have a heartbeat for a period of time (for example, five times the heartbeat interval).

2.1.3 Interactive interface

In order to interact with users and administrators, Router exposes multiple interfaces. Including RPC, Admin, WebUI.

RPC implements the most common interface for clients to interact with HDFS. Currently, only normal MapReduce,Spark and Hive (on Tez,Spark and MapReduce) are supported. Some advanced features, such as snapshots, encryption, and tiered storage, are implemented in future releases. All unimplemented functions throw exceptions.

Admin implements a RPC interface for administrators, including getting information from subclusters and adding / deleting entries to mout table. Federation information can also be obtained and modified from the command line. WebUI implements a visual Federation state that mimics the current Namenode UI, in addition to the mout table, the membership information of each subcluster, and the status of the Router.

2.2 State Store components

The information maintained by State Store includes:

Block access load, available disk space, HA status and other states of the subcluster

Mapping between folders / files and subclusters, that is, remote Mount Table

The status of the Router. State Store's back-end storage is configurable. It can be stored in a file or stored in ZooKeeper.

2.2.1 Membership

Membership reflects the state of Namenode in Federation. Includes information about subclusters, such as the amount of storage and the number of nodes. Router periodically detects information for one or more Namenode.

2.2.2 Mount Table

Manage the mapping between folders and subclusters. It is similar to Mount Table in ViewFS: hdfs://tmp → hdfs://C0-1/tmp / * Folder tmp is mapped to folder tmp in subcluster C0-1 * /

2.2.3 Router State

In order to track the status of caches in Router, Router stores its version information, status information, and so on in State Store.

2.3 Future plans

At present, RBF only implements some basic Namenode interfaces, some interfaces are not supported, and some unsupported protocol interfaces will be implemented in HDFS-13655 (https://issues.apache.org/jira/browse/HDFS-13655). At present, there are still some problems in the stability of RBF, and HDFS-13891 (https://issues.apache.org/jira/browse/HDFS-13891) will track some stability problems and solve them.

III. Deployment of Router-Based Federation Application 3.1 in Didi

The community Hadoop released RBF this Feature in 2.9,3.0. Didi's current Hadoop version is 2.7.2. Our approach is to port all the RBF submissions in the branch-2 branch to our code and make some necessary modifications.

In the big data cluster of Didi, Federation was split into five groups of Namenode. After performance testing, we have come to the conclusion that there is no pressure on one Router to serve a set of Namenode, so we choose to deploy 5 Router to serve the entire cluster. At present, the Router-Based Federation program has been running stably in Didi for more than 2 months.

3.2 compatibility

Introducing RBF directly will cause some errors when running Hive tasks, such as Wrong FS, and so on. To do this, we modified the Hive client code to make it compatible with RBF. In the metadata storage of Hive, location information stores absolute path information with HDFS Schema. When dealing with move logic in Hive code, we will make a resolve of the path to get the actual HDFS path, and then deal with it, so as to avoid this problem.

3.3 RBF Community contribution

In the actual test, we also found some performance problems and BUG of RBF, including Quota problems, improper use of mount-table cache, Null problems when mount-table created znode, and so on. After solving these problems, the patch is contributed to the community, and most of it is accepted by the community. The specific fixes and optimizations are as follows:

HDFS-13710: https://issues.apache.org/jira/browse/HDFS-13710

HDFS-13821: https://issues.apache.org/jira/browse/HDFS-13821

HDFS-13836: https://issues.apache.org/jira/browse/HDFS-13836

HDFS-13844: https://issues.apache.org/jira/browse/HDFS-13844

HDFS-13845: https://issues.apache.org/jira/browse/HDFS-13845

HDFS-13854: https://issues.apache.org/jira/browse/HDFS-13854

HDFS-13856: https://issues.apache.org/jira/browse/HDFS-13856

HDFS-13857: https://issues.apache.org/jira/browse/HDFS-13857

HDFS-13802: https://issues.apache.org/jira/browse/HDFS-13802

HDFS-13852: https://issues.apache.org/jira/browse/HDFS-13852

HDFS-14114: https://issues.apache.org/jira/browse/HDFS-14114

Thank you for reading! This is the end of the article on "how to apply Router-Based HDFS Federation in Didi big data". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.