Shulou(Shulou.com)05/31 Report--

This article introduces the "Hadoop SequenceFile BLOCK compression type file data loss how to do" related knowledge, in the actual case of the operation process, many people will encounter such a dilemma, then let the editor lead you to learn how to deal with these situations! I hope you can read it carefully and be able to achieve something!

Let's first learn about the data writing mechanism of the SequenceFile BLOCK compression type:

SequenceFile data structure diagram of BLOCK compression type

The SequenceFile.Writer implementation class of BLOCK compression type is SequenceFile.BlockCompressWriter, and the writing process is as follows:

1. Write header information: version information, compression type information, compression algorithm class information, keyClass/valueClass class name, Metadata, etc.

two。 Write Sync tags

3. The key and value are serialized and written to the cache. When the cache size reaches the threshold (default io.seqfile.compress.blocksize=1000000 bytes), the sync () operation is triggered, and the sync operation: write the sync flag first, and then compress the key and value in the cache to FSDataOutputStream. The format is like the Block compression in the figure above. Thus, a block was successfully written.

4. The subsequent data writing process is the same as 3.

5. When the final data is written, there will be the last block whose data is smaller than io.seqfile.compress.blocksize, so the sync () operation will not be triggered, so the close () method of BlockCompressWriter must be called, in which the sync () operation will be called to write the last remaining data to FSDataOutputStream and close () to FSDataOutputStream, thus completing the whole writing process.

‍ solves my problem: ‍

My problem is that only FSDataOutputStream is close () after the data is written, and writes based on the BLOCK compression type must call Writer's close () operation to trigger the sync () operation to compress the remaining data to FSDataOutputStream. In my implementation, when io.seqfile.compression.type is set to NONE and RECORD, there is no data loss. Because of these two types of compression, a piece of data is written to outputStream, and there is no cache write mechanism such as BLOCK.

This is the end of the content of "what to do if the file data of Hadoop SequenceFile BLOCK compression type is lost". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.