Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "what is the method of buffering the output stream of java". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The construction of BFOS requires passing in a byte output stream.

Basic usage

The function of this stream can be seen by name, is a buffer output stream, can be used as a buffer, the output bytes will be temporarily stored in the buffer, so what to do when the buffer is full? Then you need to brush out all the buffered data and brush it to another stream, so you need to pass in a stream during construction.

It's a bit like a swimming pool, where when the water is full, you release all the water and fill it with new water. In order to release the water, you need to give an output destination, which is the downstream output stream that needs to be passed in the constructor. Its default buffer size is 8KB.

Pay attention to this process. When writing to 8KB, the data is still stored in BFOS. As long as you write one more byte, all the previous buffers will be brushed out. If you want to write out the data before the buffer is full, you need to flush manually.

The complete example is as follows, here our downstream stream uses a memory array stream and checks the amount of data of the memory array flow at any time.

Running result:

0

8192

8195

I have previously introduced the column storage file format: big data's column storage format: Parquet

Each column of data needs to be managed by a buffer in memory, one buffer if the file has only one column, and multiple buffers if the file has multiple columns. Here we distinguish between single buffers and multiple buffers. Previously, our buffer used BAOS, refer to the java byte stream portal (memory array stream-> file stream). Is it possible to use BFOS?

Single buffer

Suppose we only need to maintain the data of one buffer in memory, that is, a byte array. When we want to write to the disk, we need to wrap the FOS with a layer of BAOS or BFOS. In order to prevent too many small data writes, we do a buffer in memory to convert a lot of small data writes into very few large data writes, making full use of the IO of the disk.

First compare the speed, BFOS+FOS vs BAOS + FOS (only one BAOS or one BFOS is maintained in memory, and then a file output stream) which is faster?

An experiment was conducted to write 800m data to a file in two ways:

BufferedOutputStream + FileOutputStream

ByteArrayOutputStream + FileOutputStream

The result: BFOS + FOS wins.

Another advantage of BFOS over BAOS is that it can control memory usage and does not consume memory indefinitely. This is when there is only one buffer of data in memory when there is a single data buffer.

Multiple buffers

If you need to maintain multiple buffers in memory, each buffer is responsible for different data, corresponding to two columns in column storage, both of which need to be written to one file, what is the difference between BFOS+FOS and BAOS+FOS?

Since BFOS's file brushing is triggered by buffer fullness, we use orange and red to distinguish the two buffered streams. If the buffer size of the buffer stream is 8KB, each buffer stream receives 24KB data, but the data is not uniform, so the order in which it is brushed into the file is not fixed, that is, each stream will trigger flush 3 times at random. Each segment of the File in the picture is a 8KB. The most memory needed for the two streams is 16KB.

With BAOS + FOS, if you need to store red and yellow data separately and cannot cross each other, you need to cache 48KB data in memory at most, and wait for a stream to cache the 24KB before writing the file. In this way, the red and yellow data can be separated, but the order is uncertain.

If separating red and yellow is a functional requirement, in this scenario, BAOS must be used.

This is the end of the content of "what is the method of buffering the output stream by java". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.