Shulou(Shulou.com)06/02 Report--

This article mainly introduces what are the key classes in the Hadoop-0.20.0 source code, which are introduced in great detail and have certain reference value. Friends who are interested must finish reading them.

Hadoop-0.20.0 source code analysis

Hadoop framework is an organic integration of two models, that is, Hadoop distributed file system (HDFS) and MapReduce parallel programming model, that is, the basic function of Hadoop framework is to perform MapReduce parallel computing on storage system HDFS, so if you want to understand the working principle and running mechanism of Hadoop framework, we should start from these two aspects.

In fact, MapReduce parallel computing in Hadoop should be implemented in HDFS, so understanding that the computing is based on HDFS should be the entry point. When you have a certain understanding of HDFS, you can know what basic elements such a parallel computing platform can provide for computing.

Of course, before you learn about HDFS, you should familiarize yourself with how Hadoop implements the file system FS and what operations it provides. In the org.apache.hadoop.fs package, a high-level abstraction of the file system (FileSystem class) is provided. Based on this abstract file system, you can implement file system implementation classes that meet practical needs, such as a file system (RawLocalFileSystem) used to store native files locally. For example, some other types of file systems can exist on top of a file system (such as the checksum-based file system ChecksumFileSystem class, ChecksumFileSystemextendsFilterFileSystem). FilterFileSystem is the most basic file system implementation.

As I read the source code, I started with the other packages, org.apache.hadoop.security, related to the file system in the org.apache.hadoop.fs package.

Some of the classes in the org.apache.hadoop.security package relate to information about users in the file system, such as user permissions, and so on. Therefore, in order to gain an in-depth understanding of the file system and facilitate reading the source code, you should understand the security support associated with the file system, which is actually implemented in the org.apache.hadoop.security package in the Hadoop source code.

The following is to read and analyze the source code of the key classes:

Group class and User class

First of all, both the org.apache.hadoop.security.Group class and the org.apache.hadoop.security.User class are entity classes that represent a class of entities that exist in the HDFS file system. Their definitions are very similar, and only the Group class is used to illustrate them.

The Group class represents the conceptual implementation of a group, which implements the java.security.Principal interface, that is, the group class represents an entity used to hold objects, for example, a group can contain multiple different users, a group can contain multiple different permissions, a group can also contain multiple authorized certificates, and so on. This class is relatively easy and contains a field of final-decorated group names, which means that once a group is created, you cannot modify the group name, and the Group class does not provide a method to modify the group name. When you create a new group, you need to specify a group name. More important in this class is the equals method, which is used to compare whether an Object object (group object) matches that group (this).

UserGroupInformation Abstract Class

Let's take a look at the UserGroupInformation abstract class in the Hadoop-0.20.0 source code. In the Hadoop framework, the distributed file system framework HDFS has an implementation for storing user and group information. It abstracts these information through a UserGroupInformation abstract class located in the org.apache.hadoop.security package. If a specific operating system based on user and group can inherit from this abstract class, it can be used to implement entities that represent some information about users and groups, and some simple operations.

The UserGroupInformation class implements the org.apache.hadoop.io.Writable interface, which is a serialization protocol defined by the Hadoop framework based on DataInput and DataOutput. The classes that implement the interface support serialization operations.

To sum up, the main operations defined by the UserGroupInformation abstract class are as follows:

1. Get the usage and group information (UGI) of the current user thread, which is realized by the getCurrentUGI () method.

2. Get the user name and group name, which are realized by the abstract methods getUserName () and getGroupNames (), respectively.

3. According to the configuration class Configuration instance of Hadoop, log in to the system and return an instance of UserGroupInformation class, which is realized by the method login (Configurationconf).

4. Read the Configuration instance of the configuration class of Hadoop and return an instance of UserGroupInformation class, which is realized by the method readFrom (Configurationconf).

These are all the contents of this article entitled "what are the key classes in the Hadoop-0.20.0 source code?" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.