Shulou(Shulou.com)06/03 Report--

This article mainly shows you the "Hadoop startup file namenode principle example analysis", the content is easy to understand, clear, hope to help you solve doubts, the following let the editor lead you to study and learn "Hadoop startup file namenode principle example analysis" this article.

1. Start namenode

In the third article of this series, we analyzed the startup file of hadoop, which mentioned that the class called when namenode starts is

Org.apache.hadoop.hdfs.server.namenode.NameNode

Its main method is as follows: public static void main (String argv []) throws Exception {if (DFSUtil.parseHelpArgument (argv, NameNode.USAGE, System.out, true)) {System.exit (0);} try {StringUtils.startupShutdownMessage (NameNode.class, argv, LOG); NameNode namenode = createNameNode (argv, null); if (namenode! = null) {namenode.join () }} catch (Throwable e) {LOG.error ("Failed to start namenode.", e); terminate (1, e);}}

The focus of this code is on line 8, where the createNameNode method creates a namenode object and then calls its join method to block waiting for the request.

The content of the createNameNode method is as follows: public static NameNode createNameNode (String argv [], Configuration conf) throws IOException {LOG.info ("createNameNode" + Arrays.asList (argv)); if (conf = = null) conf = new HdfsConfiguration (); / / Parse out some generic args into Configuration. GenericOptionsParser hParser = new GenericOptionsParser (conf, argv); argv = hParser.getRemainingArgs (); / / Parse the rest, NN specific args. StartupOption startOpt = parseArguments (argv); if (startOpt = = null) {printUsage (System.err); return null;} setStartupOption (conf, startOpt); switch (startOpt) {case FORMAT: {boolean aborted = format (conf, startOpt.getForceFormat (), startOpt.getInteractiveFormat ()); terminate (aborted? 1: 0); return null / / avoid javac warning} case GENCLUSTERID: {System.err.println ("Generating new cluster id:"); System.out.println (NNStorage.newClusterID ()); terminate (0); return null;} case FINALIZE: {System.err.println ("Use of the argument'" + StartupOption.FINALIZE + "'is no longer supported. To finalize an upgrade, start the NN "+" and then run `hdfs dfsadmin-finalizeUpgrade' "); terminate (1); return null; / / avoid javac warning} case ROLLBACK: {boolean aborted = doRollback (conf, true); terminate (aborted? 1: 0); return null / avoid warning} case BOOTSTRAPSTANDBY: {String toolArgs [] = Arrays.copyOfRange (argv, 1, argv.length); int rc = BootstrapStandby.run (toolArgs, conf); terminate (rc); return null; / / avoid warning} case INITIALIZESHAREDEDITS: {boolean aborted = initializeSharedEdits (conf, startOpt.getForceFormat (), startOpt.getInteractiveFormat ()) Terminate (aborted? 1: 0); return null; / / avoid warning} case BACKUP: case CHECKPOINT: {NamenodeRole role = startOpt.toNodeRole (); DefaultMetricsSystem.initialize (role.toString (). Replace (",")); return new BackupNode (conf, role);} case RECOVER: {NameNode.doRecovery (startOpt, conf); return null } case METADATAVERSION: {printMetadataVersion (conf); terminate (0); return null; / / avoid javac warning} case UPGRADEONLY: {DefaultMetricsSystem.initialize ("NameNode"); new NameNode (conf); terminate (0); return null;} default: {DefaultMetricsSystem.initialize ("NameNode"); return new NameNode (conf) }}}

This code is very simple. There are three main operations to do:

1. Create a profile object

2. Parse the parameters of the command line

3. Execute the corresponding method according to the parameters (switch block)

The configuration file created is HdfsConfiguration (line 5), where HdfsConfiguration inherits from the Configuration class, which loads the hadoop configuration file into memory. Then parse the parameters passed into the main method and execute the specific method based on that parameter. The contents of the default that are executed at normal startup. The content of default is also very simple, which is to create a Namenode object.

Here, let's start with the analysis of HdfsConfiguration and explain the configuration file processing of hdfs in detail.

First, the initialization method of HdfsConfiguration is as follows: public HdfsConfiguration () {super ();}

Here is the initialization method that calls its parent class.

Its parent class is the Configuration class, and its initialization method is as follows: / * * A new configuration. * / public Configuration () {this (true);}

You can see here that he called an overloaded method, passing in a parameter: true.

Then take a closer look at the overloaded method: public Configuration (boolean loadDefaults) {this.loadDefaults = loadDefaults; updatingResource = new ConcurrentHashMap (); synchronized (Configuration.class) {REGISTRY.put (this, null);}}

It's also simple here, mainly assigning values to two parameters and adding the newly created Configuration to the REGISTRY.

At this point, a configuration file is created. However, the initialization parsing of the configuration file has not been completed. In java, you can use the static keyword to declare a block of code that is executed when the class is loaded. There are static blocks of code in both Configuration and HdfsConfiguration.

First, in the Configuration class, there is a static block of code on line 682, which reads as follows:

The focus of this code is on lines 695 and 696, where an addDefaultResource method is called, passing in two parameters, core-default.xml and core-site.xml. Where core-site.xml is the configuration file that is set when you install hadoop. Core-default.xml is the configuration file that comes with hadoop, which can be found in the official documentation of hadoop. The document link is as follows: https://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-common/core-default.xml

This file is also available in the source code of hadoop, which is in hadoop-common-XX.jar.

Then continue to analyze the called addDefaultResource method

Its contents are as follows:

Public static synchronized void addDefaultResource (String name) {if (! defaultResources.contains (name)) {defaultResources.add (name); for (Configuration conf: REGISTRY.keySet ()) {if (conf.loadDefaults) {conf.reloadConfiguration ();}

This code is also very simple. First, the second line determines from the defaultResources whether the configuration file already exists.

The defaultResources here is a list.

It is defined as follows:

Private static final CopyOnWriteArrayList defaultResources = new CopyOnWriteArrayList ()

If the configuration file does not exist in defaultResources, proceed to add the configuration file to defaultResources (line 3). Then iterate through the key in REGISTRY (line 4), where key is the Configuration object mentioned above. It is then determined whether the reloadConfiguration method is executed according to the value of its loadDefaults.

The value of loadDefaults here is the value of the incoming overloaded method analyzed above, and the one passed in above is true, so the Configuration object it creates executes the reloadConfiguration method here.

The reloadConfiguration method is as follows: public synchronized void reloadConfiguration () {properties = null; / / trigger reload finalParameters.clear (); / / clear site-limits}

Here you can see that the reloadConfiguration method does not actually reload the configuration file but sets the value of properties to null.

There is also a similar static code block in HdfsConfiguration, on line 30, which reads as follows:

Here you first call an addDeprecatedKeys method and then call an addDefaultResource. The addDefaultResource here sends two files, hdfs-default.xml and hdfs-site.xml. Where hdfs-site.xml is the configuration file for installation, and hdfs-default.xml is the default file that comes with it, just like core-default.xml above. The official website link is: https://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml. The file is located at: hadoop-hdfs-2.7.6.jar.

The content of addDeprecatedKeys method is as follows: private static void addDeprecatedKeys () {Configuration.addDeprecations (new DeprecationDelta [] {new DeprecationDelta ("dfs.backup.address", DFSConfigKeys.DFS_NAMENODE_BACKUP_ADDRESS_KEY), new DeprecationDelta ("dfs.backup.http.address", DFSConfigKeys.DFS_NAMENODE_BACKUP_HTTP_ADDRESS_KEY), new DeprecationDelta ("dfs.balance.bandwidthPerSec", DFSConfigKeys.DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_KEY) New DeprecationDelta ("dfs.data.dir", DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY), new DeprecationDelta ("dfs.http.address", DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_KEY), new DeprecationDelta ("dfs.https.address", DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY), new DeprecationDelta ("dfs.max.objects", DFSConfigKeys.DFS_NAMENODE_MAX_OBJECTS_KEY) New DeprecationDelta ("dfs.name.dir", DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY), new DeprecationDelta ("dfs.name.dir.restore", DFSConfigKeys.DFS_NAMENODE_NAME_DIR_RESTORE_KEY), new DeprecationDelta ("dfs.name.edits.dir", DFSConfigKeys.DFS_NAMENODE_EDITS_DIR_KEY), new DeprecationDelta ("dfs.read.prefetch.size") DFSConfigKeys.DFS_CLIENT_READ_PREFETCH_SIZE_KEY), new DeprecationDelta ("dfs.safemode.extension", DFSConfigKeys.DFS_NAMENODE_SAFEMODE_EXTENSION_KEY), new DeprecationDelta ("dfs.safemode.threshold.pct", DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY), new DeprecationDelta ("dfs.secondary.http.address", DFSConfigKeys.DFS_NAMENODE_SECONDARY_HTTP_ADDRESS_KEY) New DeprecationDelta ("dfs.socket.timeout", DFSConfigKeys.DFS_CLIENT_SOCKET_TIMEOUT_KEY), new DeprecationDelta ("fs.checkpoint.dir", DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_KEY), new DeprecationDelta ("fs.checkpoint.edits.dir", DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY), new DeprecationDelta ("fs.checkpoint.period", DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_PERIOD_KEY) New DeprecationDelta ("heartbeat.recheck.interval", DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY), new DeprecationDelta ("dfs.https.client.keystore.resource", DFSConfigKeys.DFS_CLIENT_HTTPS_KEYSTORE_RESOURCE_KEY), new DeprecationDelta ("dfs.https.need.client.auth", DFSConfigKeys.DFS_CLIENT_HTTPS_NEED_AUTH_KEY), new DeprecationDelta ("slave.host.name") DFSConfigKeys.DFS_DATANODE_HOST_NAME_KEY), new DeprecationDelta ("session.id", DFSConfigKeys.DFS_METRICS_SESSION_ID_KEY), new DeprecationDelta ("dfs.access.time.precision", DFSConfigKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY), new DeprecationDelta ("dfs.replication.considerLoad", DFSConfigKeys.DFS_NAMENODE_REPLICATION_CONSIDERLOAD_KEY), new DeprecationDelta ("dfs.replication.interval") DFSConfigKeys.DFS_NAMENODE_REPLICATION_INTERVAL_KEY), new DeprecationDelta ("dfs.replication.min", DFSConfigKeys.DFS_NAMENODE_REPLICATION_MIN_KEY), new DeprecationDelta ("dfs.replication.pending.timeout.sec", DFSConfigKeys.DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY), new DeprecationDelta ("dfs.max-repl-streams", DFSConfigKeys.DFS_NAMENODE_REPLICATION_MAX_STREAMS_KEY) New DeprecationDelta ("dfs.permissions", DFSConfigKeys.DFS_PERMISSIONS_ENABLED_KEY), new DeprecationDelta ("dfs.permissions.supergroup", DFSConfigKeys.DFS_PERMISSIONS_SUPERUSERGROUP_KEY), new DeprecationDelta ("dfs.write.packet.size", DFSConfigKeys.DFS_CLIENT_WRITE_PACKET_SIZE_KEY), new DeprecationDelta ("dfs.block.size", DFSConfigKeys.DFS_BLOCK_SIZE_KEY) New DeprecationDelta ("dfs.datanode.max.xcievers", DFSConfigKeys.DFS_DATANODE_MAX_RECEIVER_THREADS_KEY), new DeprecationDelta ("io.bytes.per.checksum", DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY), new DeprecationDelta ("dfs.federation.nameservices", DFSConfigKeys.DFS_NAMESERVICES), new DeprecationDelta ("dfs.federation.nameservice.id", DFSConfigKeys.DFS_NAMESERVICE_ID) New DeprecationDelta ("dfs.client.file-block-storage-locations.timeout", DFSConfigKeys.DFS_CLIENT_FILE_BLOCK_STORAGE_LOCATIONS_TIMEOUT_MS),} }

This code is very simple, only one sentence. The static method addDeprecations of Configuration is called, passing in an array of type DeprecationDelta, and assigning a value to the data in the array.

The above is all the contents of the article "sample Analysis of the principle of namenode in the startup file of Hadoop". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.