Shulou(Shulou.com)05/31 Report--

This article mainly shows you how "HDFS Namenode works". The content is simple and easy to understand, and the organization is clear. I hope it can help you solve your doubts. Let Xiaobian lead you to study and learn this article "How HDFS Namenode works".

HDFS (Hadoop Distributed Filesystem) clients perform file system raw data operations through a single server node called Namenode, while DataNodes communicate with other DataNodes and replicate data blocks for redundancy, so that a single DataNode corruption does not cause data loss in the cluster.

However, the loss of NameNode failure is intolerable. The main responsibility of NameNode is to track how files are divided into file blocks, which nodes store file blocks, and whether the overall running state of the distributed file system is normal. If NameNode stops running, it will cause data nodes to fail to communicate, and clients cannot read and write data to HDFS. In fact, this will also cause the entire system to stop working.

The HDFS Namenode is a single point of failure (SPOF)

Facebook also understands the seriousness of the problem caused by "Namenode-as-SPOF," so Facebook hopes to build a system to eliminate the hidden dangers caused by "Namenode-as-SPOF." But before we get into the system, let's look at the problems Facebook has encountered in using and deploying HDFS.

Facebook Data Warehouse Usage

The largest HDFS cluster is deployed in Facebook's data warehouse, which is used for traditional Hadoop MapReduce workloads-running MapReduce batch jobs on a small fraction of the large cluster

Because the cluster is very large, the client and many DataNode nodes and NameNode nodes transmit a large amount of raw data, which causes the NameNode load to be very heavy. High NameNode loads are common in data warehouse clusters due to stress from CPU, memory, disk, and network. Facebook found that HDFS accounted for 41% of all failures in its data warehouse.

HDFS NameNode is an important part of HDFS and also an important part of the whole data warehouse. While a highly available NameNode can only prevent 10% of unplanned downtime in a data warehouse, eliminating NameNodes is a major victory for SPOF because it allows Facebook to perform scheduled hardware and software recovery. In fact, Facebook estimates that it could eliminate 50% of the cluster's planned downtime if NameNode is addressed.

So what does a high availability NameNode look like? How will it work? Let's take a look at the graph of the highly available NameNode.

Avatarnode Client

That's how HDFS Namenode works. Thanks for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.