Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the inner classes and auxiliary classes of Task". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what are the inner classes and auxiliary classes of Task?"

Task has many inner classes and a large number of class member variables that work with Task, as shown in the following figure.

MapOutputFile manages the output files of Mapper and provides a series of get methods to get the various files needed by Mapper, all of which are stored in a directory.

We assume that the JobID passed into MapOutputFile is job_200707121733_0003,TaskID and task_200707121733_0003_m_000005. The root of MapOutputFile is

{mapred.local.dir} / taskTracker/jobcache/ {jobid} / {taskid} / output

In the following discussion, we write down the above path as {MapOutputFileRoot}

Taking the above JogID and TaskID as an example, we have:

{mapred.local.dir} / taskTracker/jobcache/job_200707121733_0003/task_200707121733_0003_m_000005/output

It is important to note that {mapred.local.dir} can contain a series of paths, so Hadoop will find a directory under these root paths to meet the requirements and create the required files. There are two methods for MapOutputFile, ending with ForWrite and without ForWrite, and with ForWrite for creating files, which requires a file size as a parameter to check disk space. The file that is used to get to build without ForWrite.

GetOutputFile: file name is {MapOutputFileRoot} / file.out

GetOutputIndexFile: file name is {MapOutputFileRoot} / file.out.index

GetSpillFile: file name is {MapOutputFileRoot} / spill {spillNumber} .out

GetSpillIndexFile: file name is {MapOutputFileRoot} / spill {spillNumber} .out.index

The above four methods are used in the Task subclass MapTask

GetInputFile: file name is {MapOutputFileRoot} / map_ {mapId} .out

Used in ReduceTask. We will introduce the corresponding application scenarios where they are used.

After introducing temporary file management, let's take a look at Task.CombineOutputCollector, which inherits from org.apache.hadoop.mapred.OutputCollector. It's very simple, it's just an OutputCollector to IFile.Writer Adapter, and IFile.Writer does all the work.

ValuesIterator is used as an iterator to get RawComparatorcomparator-compliant values from RawKeyValueIterator (Key,Value requires that the input has been sorted by DataInputBuffer,ValuesIterator). It has a simple subclass in Task, CombineValuesIterator.

Task.TaskReporter is used to submit counter reports and status reports to JobTracker, which implements counter report Reporter and status report StatusReporter. In order not to affect the work of the main thread, TaskReporter has a separate thread that reports Task execution to JobTracker through the TaskUmbilicalProtocol interface and using Hadoop's RPC mechanism.

FileSystemStatisticUpdater is a simple utility class for recording the number of bytes of pair / write operations on the file system.

At this point, I believe you have a deeper understanding of "what are the internal classes and auxiliary classes of Task". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.