How to read getSplits by FileInputFormat
This article mainly shows you "FileInputFormat how to read getSplits", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "FileInputFormat how to read getSplits" this article.
/ * *
* Generate the list of files and make them into FileSplits.
* @ param job the job context
* @ throws IOException
, /
Public List getSplits (JobContext job) throws IOException {
Stopwatch sw = new Stopwatch () .start ()
/ / get a minimum value that an InputSplit can contain
Long minSize = Math.max (getFormatMinSplitSize (), getMinSplitSize (job))
/ / get the maximum value that an InputSplit can contain
Long maxSize = getMaxSplitSize (job)
/ / generate splits
List splits = new ArrayList ()
List files = listStatus (job)
/ *
* it can be seen that if there are a million small files, it will be looped a million times and at least a million InputSplit will be generated, which will contain at least a million map tasks
* if the default size of an InputSplit is a block size, that is, 64m
* A 20m file will generate an InputSplit, a Map task
* an 80m file will generate two InputSplit and two Map tasks
* two 20m files generate a total of two InputSplit and two Map tasks
* A 20m and 70m file will generate a total of three InputSplit and three Map tasks
, /
For (FileStatus file: files) {
Path path = file.getPath ()
Long length = file.getLen ()
If (length! = 0) {
BlockLocation [] blkLocations
If (file instanceof LocatedFileStatus) {
BlkLocations = (LocatedFileStatus) file) .getBlockLocations ()
} else {
FileSystem fs = path.getFileSystem (job.getConfiguration ())
BlkLocations = fs.getFileBlockLocations (file, 0, length)
} if (isSplitable (job, path)) {
/ / get the default block block size of hdfs
Long blockSize = file.getBlockSize ()
/ / calculate the size of an InputSplit
Long splitSize = computeSplitSize (blockSize, minSize, maxSize)
Long bytesRemaining = length
While ((double) bytesRemaining) / splitSize > SPLIT_SLOP) {
Int blkIndex = getBlockIndex (blkLocations, length-bytesRemaining)
Splits.add (makeSplit (path, length-bytesRemaining, splitSize
BlkLocations [blkIndex] .getHosts ()
BlkLocations [blkIndex]. GetCachedHosts ())
BytesRemaining-= splitSize
}
If (bytesRemaining! = 0) {
Int blkIndex = getBlockIndex (blkLocations, length-bytesRemaining)
Splits.add (makeSplit (path, length-bytesRemaining, bytesRemaining
BlkLocations [blkIndex] .getHosts ()
BlkLocations [blkIndex]. GetCachedHosts ())
}
} else {/ / not splitable
Splits.add (makeSplit (path, 0, length, blkLocations [0] .getHosts ()
BlkLocations [0] .getCachedHosts ())
}
} else {
/ / Create empty hosts array for zero length files
Splits.add (makeSplit (path, 0, length, new String [0]))
}
}
/ / Save the number of input files for metrics/loadgen
Job.getConfiguration () .setLong (NUM_INPUT_FILES, files.size ()
Sw.stop ()
If (LOG.isDebugEnabled ()) {
LOG.debug ("Total # of splits generated by getSplits:" + splits.size () + ", TimeTaken:" + sw.elapsedMillis ())
}
Return splits
}
The above is all the content of the article "how to read getSplits by FileInputFormat". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!
