Shulou(Shulou.com)06/02 Report--

This article shows you the role of DirectByteBuffer and file IO respectively, the content is concise and easy to understand, absolutely can make your eyes bright, through the detailed introduction of this article, I hope you can get something.

a. The traditional IO operation (that is, the api using the java.io package) accesses the disk file, and the number of times copy is required for the data:

1. Data of disk files copy kernel page cache

two。 Kernel data copy application space (i.e. jvm out-of-heap memory)

3. Jvm out-of-heap memory copy jvm in-heap memory

Why don't you merge 2, 3 and 3 to copy jvm the kernel data in the heap memory. Because gc occurs when jvm makes a system call to read the file, then the corresponding address of the heap memory will move, so it is a problem to copy directly into the heap.

b. Using DirectByteBuffer to access disk files, the number of times copy is required for data:

1. Data of disk files copy kernel page cache

two。 Kernel data copy application space (i.e. DirectByteBuffer)

So DirectByteBuffer reduces the number of memory copy.

1. IO parsing of traditional files

Example of file reading:

FileInputStream input = new FileInputStream ("/ data")

Byte [] b = new byte [SIZE]

Input.read (b)

The byte array shows heap memory objects, where the data is copy to jvm heap memory. Let's look at the internal implementation of the read function

Public int read (byte b []) throws IOException {

Return readBytes (b, 0, b.length)

}

Private native int readBytes (byte b [], int off, int len) throws IOException

We see that the read function finally calls the native function readBytes.

JintreadBytes (JNIEnv * env, jobject this, jbyteArray bytes, jint off, jint len, jfieldID fid) {

Jint nread

Char stackBuf[BUF _ SIZE]

Char * buf = NULL

FD fd

If (IS_NULL (bytes)) {

JNU_ThrowNullPointerException (env, NULL)

Return-1

}

If (outOfBounds (env, off, len, bytes)) {

JNU_ThrowByName (env, "java/lang/IndexOutOfBoundsException", NULL)

Return-1

}

If (len = = 0) {

Return 0

} else if (len > BUF_SIZE) {

Buf = malloc (len)

If (buf = = NULL) {

JNU_ThrowOutOfMemoryError (env, NULL)

Return 0

}

} else {

Buf = stackBuf

}

Fd = GET_FD (this, fid)

If (fd =-1) {

JNU_ThrowIOException (env, "Stream Closed")

Nread =-1

} else {

Nread = IO_Read (fd, buf, len)

If (nread > 0) {

(env)-> SetByteArrayRegion (env, bytes, off, nread, (jbyte *) buf)

} else if (nread =-1) {

JNU_ThrowIOExceptionWithLastError (env, "Read error")

} else {/ * EOF * /

Nread =-1

}

}

If (buf! = stackBuf) {

Free (buf)

}

Return nread

}

We see that the buffered data is eventually read into buf through IO_Read, which is actually a macro definition:

# define IO_Read handleRead

The handleRead function is implemented as follows. Here you can see that the read system call is made here:

Ssize_t

HandleRead (FD fd, void * buf, jint len)

{

Ssize_t result

RESTARTABLE (read (fd, buf, len), result)

Return result

}

After buf is returned, the JNI function SetByteArrayRegion is copied to bytes. Its implementation is as follows (a general macro function is defined below to represent the setting of the array area of various data types, which can be understood by replacing the Result macro with Byte:

JNI_ENTRY (void,\

Jni_Set##Result##ArrayRegion (JNIEnv * env, ElementType##Array array, jsize start,\

Jsize len, const ElementType * buf))\

JNIWrapper ("Set" XSTR (Result) "ArrayRegion");\

DTRACE_PROBE5 (hotspot_jni, Set##Result##ArrayRegion__entry, env, array, start, len, buf);\

DT_VOID_RETURN_MARK (Set##Result##ArrayRegion);\

TypeArrayOop dst = typeArrayOop (JNIHandles::resolve_non_null (array));\

If (start

(unsigned int) dst- > length ()) {\

THROW (vmSymbols::java_lang_ArrayIndexOutOfBoundsException ());\

} else {\

If (len > 0) {\

Int sc = TypeArrayKlass::cast (dst- > klass ())-> log2_element_size ();\

Memcpy ((uprichar*) dst- > Tag##_at_addr (start),\

(uprichar *) buf,\

Len-1L) {

DirectMemory = var2

}

}

/ /...

}

The "sun.nio.MaxDirectMemorySize" property is set through the-XX:MaxDirectMemorySize parameter. If we do not specify this jvm parameter, the author tests it in jdk8, which defaults to-1, which results in the directBufffer memory being limited to the maximum process memory. Of course, this is also a potential risk.

Risk case:

The author once ran an application online. The application consumes data from the message queue and then stores the data in Hbase after processing. However, when the application runs for about 2 weeks each time, the machine will take up too much swap. After analysis, the jvm process takes up too much memory, but the analysis of jvm-related parameters (heap, thread size) is not set very large. Finally, it is found that the directBuffer occupies up to 10G. Later, the problem was solved by limiting directbuffer usage through-XX:MaxDirectMemorySize=2048m. Every time the directBuffer occupies 2G, a fullgc is triggered to recycle the previously useless directbuffer. Hbase is a pit. I will sort out this case when I have time.

3.DirectByteBuffer file IO

Example of file reading:

FileChannel filechannel=new RandomAccessFile ("/ data/appdatas/cat/mmm", "rw") .getChannel ()

ByteBuffer byteBuffer = ByteBuffer.allocateDirect (SIZE)

Filechannel.read (byteBuffer)

Let's take a look at the read function

Public int read (ByteBuffer var1) throws IOException {

/ / .

Var3 = IOUtil.read (this.fd, var1,-1L ·this.nd)

/ / .

}

The main logic calls IOUtil.read. Let's take a look at this function.

Static int read (FileDescriptor var0, ByteBuffer var1,long var2, NativeDispatcher var4) throws IOException {

If (var1.isReadOnly ()) {

Throw new IllegalArgumentException ("Read-only buffer")

} else if (var1instanceof DirectBuffer) {

Return readIntoNativeBuffer (var0, var1, var2, var4)

} else {

ByteBuffer var5 = Util.getTemporaryDirectBuffer (var1.remaining ())

Int var7

Try {

Int var6 = readIntoNativeBuffer (var0, var5, var2, var4)

Var5.flip ()

If (var6 > 0) {

Var1.put (var5)

}

Var7 = var6

} finally {

Util.offerFirstTemporaryDirectBuffer (var5)

}

Return var7

}

}

The main method is to read the data into directBuffer through the function readIntoNativeBuffer, where readIntoNativeBuffer also calls a native method.

From the above code, we will see that if fielchannel.read (ByteBuffer) can also pass in a HeapByteBuffer, this class is in the heap. If it is this class, then when reading internally, the data will be read into DirectByteBuffer first, and then in copy to HeapByteBuffer. Util.getTemporaryDirectBuffer (var1.remaining ()); is to get a DirectBuffer object. Because DirectBuffer is expensive when it is created, it is usually managed by a pool when it is used. If you are interested, you can take a look at the implementation in the Util class.

What is the function of DirectByteBuffer and file IO respectively? have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.