Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the initialization process of DFSClient under hadoop2.x HA". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian to study and learn "what is the initialization process of DFSClient under hadoop2.x HA" together.

Hadoop2.X NameNode adds autofileover mechanism. For namenode with HA, how to automatically identify active NN when HDFS Client accesses it, and automatically switch to new NN when active NN is down?

1. DFSClient initialization:

The main classes involved are the following:NameNodeProxies, RetryProxy, RetryInvocationHandler, ConfiguredFailoverProxyProvider.

NameNodeProxies

Creation of objects communicating with NN provides classes

RetryProxy

Factory Method of RetryProxy

RetryInvocationHandler

Method Implementation of Connection Exception Proxy

ConfiguredFailoverProxyProvider

Get classes of available NN

The specific code implementation is relatively simple, but the logic is slightly complex.

2. How to switch automatically when Active NN is down.

You can see the implementation code for handling connection exceptions:

if (action.action == RetryAction.RetryDecision.FAILOVER_AND_RETRY) {//Continue selecting available NN nodes // Make sure that concurrent failed method invocations only cause a // single actual fail over. synchronized (proxyProvider) { if (invocationAttemptFailoverCount == proxyProviderFailoverCount) { proxyProvider. performFail (currentProxy.proxy); //This method retrieves the next available NN proxyProviderFailoverCount++; } else { LOG.warn("A failover has occurred since the start of this method" + " invocation attempt. "); } currentProxy = proxyProvider.getProxy(); //Returns the currently available NN } invocationFailoverCount++; }

Implementation of performFail:

@Override public synchronized void performFailover(T currentProxy) { currentProxyIndex = (currentProxyIndex + 1) % proxies.size(); }

In fact, for obtaining the available NN, it is to put all NN node methods into the List, and then get NN from index 0. If there is an exception Index++, continue to get it until it is available.

Thank you for reading, the above is the content of "what is the DFSClient initialization process under hadoop2.x HA". After studying this article, I believe that everyone has a deeper understanding of what is the DFSClient initialization process under hadoop2.x HA. The specific use situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.