Shulou(Shulou.com)05/31 Report--

This article mainly shows you the "sample code of Configuration in hadoop-common", which is easy to understand and clear. I hope it can help you solve your doubts. Let the editor lead you to study and study the "sample code of Configuration in hadoop-common".

The Configuration class implements the Iterable and Writable interfaces so that it can be traversed and serialized (hadoop serializes itself)

Configuration file format lala$ {user.home} / hadoopdatatrue

Hadoop is configured through xml and supports attribute extension. When user.home calls get, it will first determine whether it is a system parameter by System.getProperty (). In this case, ${user.home} is replaced with the current user's path. Therefore, when we configure hadoop, we can directly use some system properties to enhance portability.

When a property is final, the configuration is added later, and the configuration added first is not overwritten.

At the same time, because the DOM parsing of java is used, the inclusion of XML is supported and can be used for classified management in the configuration file.

Code analysis private inner class Resource private static class Resource {/ / private inner class, mark resource name and resource object private final Object resource;private final String name;.} private inner class DeprecatedKeyInfo, DeprecationDelta, DeprecationContext private static class DeprecatedKeyInfo {private final String [] newKeys;private final String customMessage;private final AtomicBoolean accessed = new AtomicBoolean (false); private final String getWarningMessage (String key) {}} public static class DeprecationDelta {private final String key;private final String [] newKeys;private final String customMessage } private static class DeprecationContext {/ / store oldkey-newkeysprivate final Map deprecatedKeyMap;//, store newkeys-oldkey, provide reverse search function private final Map reverseDeprecatedKeyMap; DeprecationContext (DeprecationContext other, DeprecationDelta [] deltas) {... this.deprecatedKeyMap = UnmodifiableMap.decorate (newDeprecatedKeyMap); this.reverseDeprecatedKeyMap = UnmodifiableMap.decorate (newReverseDeprecatedKeyMap);}}

DeprecatedKeyInfo saves the new key and information, and if the customMessage is empty, the default information is automatically generated when calling getWarningMessage.

DeprecationDelta preserves the abandoned key and the proposed new key.

DeprecationContext encapsulation encapsulates abandoned key with recommended keys and prompts.

Private static AtomicReference deprecationContext = new AtomicReference (new DeprecationContext (null, defaultDeprecations))

A global DeprecationContext object, atomic, and loads the key that is abandoned by default.

Static addDeprecations method

It is worth mentioning that this method skillfully uses a lock-free method, but it ensures the security of the data, see the specific code:

Public static void addDeprecations (DeprecationDelta [] deltas) {DeprecationContext prev, next; do {prev = deprecationContext.get (); next = new DeprecationContext (prev, deltas);} while (! deprecationContext.compareAndSet (prev, next);}

The compareAndSet method updates the current object to next when the current object is equal to prev (= =)

SetDeprecatedProperties

After analyzing the source code, we find that the purpose of setDeprecatedProperties is to update overlay and properties, so that we can get the latest status when we get key. See the following example:

Configuration.addDeprecation ("xx", new String [] {"xx1", "xx2", "xx3"}); / / configuration.setDeprecatedProperties (); System.out.println (configuration.get ("xx"))

When I comment out the configuration.setDeprecatedProperties, I get the null value when I get, so when we want to traverse the abandoned key, we need to update the setDeprecatedProperties so that the abandoned key can still be used.

HandleDeprecation

First determine whether the key is abandoned, and if so, you will get the recommended key, otherwise update overlay, properties, and return the recommended key array.

The sample handleDeprecation method is to perform a refresh operation. Specifically used in asXmlDocument.

Static {} static code block

By analyzing the code, we can get the following points:

If hadoop-site.xml exists under classpath, log4j prints a warning message and is not loaded into defaultResources.

Two core configuration files, core-default.xml and core-site.xml, are loaded by default

AddResourceObject and several methods

No matter which addResource you use, you end up calling addResourceObject (Resource resource), which first adds the resource to a global List collection, and then calls reloadConfiguration to trigger the refresh properties and the key marked as final expires.

FindSubVariable substituteVars

Before hadoop-2.7, there was only one substituteVars method that used java's own regular expression to match the value in the middle of ${user.home} (user.home).

After the hadoop-2.7 version, in order to improve performance, I implemented the matching method to obtain the intermediate value (findSubVariable) ps: probably because the regular expression of java itself consumes too much performance, so it reduces the consumption of performance by matching manually.

/ / this method gets the interval of the position in the middle of ${} in the string. See the code private static int [] findSubVariable (String eval) {.} / / 1. Will get key to replace, if System.getProperty () exists, replace / / 2. Does not exist, find properties, if there is a replacement, does not exist, leave private String substituteVars (String expr) {...} set method / / allow key-value,source to be empty through the program, when the user does not set the source, the program automatically sets the programatically to source, / / when the value is abandoned, this method will first send the new key to And set source to because old key is deprecated public void set (String name, String value, String source) {.} loadResource

The method is to parse xml, using DOM parsing, analysis code we know, xml format and the format written above, while DOM parsing, support the introduction of xml files.

Compared with previous versions, in the xml configuration file, you can declare the source tag in property and declare the information of the resource?

If ("source" .equals (field.getTagName ()) & & field.hasChildNodes ()) source.add (StringInterner.weakIntern (Text) field.getFirstChild ()). GetData ()); the above is all the content of the article "sample Code of Configuration in hadoop-common". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.