Shulou(Shulou.com)06/01 Report--

Native environment:

Linux 4.4.0-21-generic # 37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86 "64 GNU/Linux

Buffer

The class diagram of Buffer is as follows:

Except for Boolean, all basic data types have corresponding Buffer, but only ByteBuffer can interact with Channel. Only ByteBuffer can generate buffer for Direct, while Buffer of other data types can only produce Buffer of type Heap. ByteBuffer can produce view Buffer of other data types, and if the ByteBuffer itself is Direct, the resulting view Buffer is also Direct.

The nature of Direct and Heap type Buffer

The first choice is to talk about how JVM operates IO.

JVM needs to complete IO operations through operating system calls, such as reading files through read system calls. The prototype of read is ssize_t read (int fd,void * buf,size_t nbytes), which, like other IO system calls, generally requires a buffer as one of the parameters, which is required to be continuous.

Buffer is divided into two categories: Direct and Heap, which are described below.

Heap

Buffer of type Heap exists on the heap of JVM, and this part of memory is reclaimed and cleaned up just like normal objects. All Buffer objects of type Heap contain an array property corresponding to the basic data type (for example: final * * [] hb), and the array is the underlying buffer of the Heap type Buffer.

However, Buffer of type Heap cannot be called directly as a buffer parameter, mainly for two reasons.

JVM may move the buffer (copy-defragment) during GC, and the address of the buffer is not fixed.

When a system call is made, the buffer needs to be contiguous, but the array may not be contiguous (the implementation of JVM does not require contiguity).

So when using Buffer of type Heap for IO, JVM needs to generate a temporary Buffer of type Direct, then copy the data, and then use the Buffer of temporary Direct as a parameter to make operating system calls. This results in low efficiency mainly for two reasons:

You need to copy the data from the Buffer of type Heap into the Buffer of the temporarily created Direct.

It is possible to produce a large number of Buffer objects, thus increasing the frequency of GC. So in the IO operation, it can be optimized by reusing Buffer.

Direct

Buffer of type Direct does not exist on the heap, but is a continuous piece of memory allocated directly by JVM through malloc, which becomes direct memory, and JVM uses direct memory as a buffer when making IO system calls.

-XX:MaxDirectMemorySize, which allows you to set the maximum amount of direct memory allowed to be allocated (memory allocated by MappedByteBuffer is not affected by this configuration).

The recycling of direct memory is different from that of heap memory. If direct memory is not used properly, it is easy to cause OutOfMemoryError. JAVA does not provide a display method to actively free direct memory, and the sun.misc.Unloaded class can perform direct underlying memory operations, through which direct memory can be actively freed and managed. By the same token, direct memory should be reused to improve efficiency.

The relationship between MappedByteBuffer and DirectByteBuffer

This is a little bit backwards: By rights MappedByteBuffer should be a subclass of DirectByteBuffer, but to keep the spec clear and simple, and for optimization purposes, it's easier to do it the other way around.This works because DirectByteBuffer is a package-private class. (this paragraph is extracted from the source code of MappedByteBuffer)

In fact, MappedByteBuffer belongs to the mapping buffer (see for yourself virtual memory), but DirectByteBuffer simply states that this part of the memory is a contiguous buffer allocated by JVM in the direct memory area, not mapped. That is to say, MappedByteBuffer should be a subclass of DirectByteBuffer, but for convenience and optimization, MappedByteBuffer is taken as the parent class of DirectByteBuffer. In addition, although MappedByteBuffer should be logically a subclass of DirectByteBuffer, and the memory GC of MappedByteBuffer is similar to the GC of direct memory (unlike heap GC), the size of the allocated MappedByteBuffer is not affected by the-XX:MaxDirectMemorySize parameter.

MappedByteBuffer encapsulates memory-mapped file operations, that is, file IO operations can only be performed. MappedByteBuffer is a mapping buffer generated according to mmap, which is mapped to the corresponding file page and belongs to direct memory user mode. The mapping buffer can be operated directly through MappedByteBuffer, and this part of the buffer is mapped to the file page. The operating system completes the writing and writing of the file through the call in and out of the corresponding memory page.

MappedByteBuffer

MappedByteBuffer is obtained by FileChannel.map (MapMode mode,long position, long size). The generation process of MappedByteBuffer is explained below combined with the source code.

Source code of FileChannel.map:

Public MappedByteBuffer map (MapMode mode, long position, long size) throws IOException {ensureOpen (); if (position

< 0L) throw new IllegalArgumentException("Negative position"); if (size < 0L) throw new IllegalArgumentException("Negative size"); if (position + size < 0) throw new IllegalArgumentException("Position + size overflow"); //最大2G if (size >

Integer.MAX_VALUE) throw new IllegalArgumentException ("Size exceeds Integer.MAX_VALUE"); int imode =-1; if (mode = = MapMode.READ_ONLY) imode = MAP_RO; else if (mode = = MapMode.READ_WRITE) imode = MAP_RW; else if (mode = = MapMode.PRIVATE) imode = MAP_PV; assert (imode > = 0) If ((mode! = MapMode.READ_ONLY) & &! writable) throw new NonWritableChannelException (); if (! readable) throw new NonReadableChannelException (); long addr =-1; int ti =-1; try {begin (); ti = threads.add (); if (! isOpen ()) return null / / size () returns the actual file size / / if the actual file size does not match, the file size is increased, the file size is changed, and the size of the file is set to 0 by default. If (size ()

< position + size) { // Extend file size if (!writable) { throw new IOException("Channel not open for writing " + "- cannot extend file to required size"); } int rv; do { //增大文件的大小 rv = nd.truncate(fd, position + size); } while ((rv == IOStatus.INTERRUPTED) && isOpen()); } //如果要求映射的文件大小为0，则不调用操作系统的mmap调用，只是生成一个空间容量为0的DirectByteBuffer //并返回 if (size == 0) { addr = 0; // a valid file descriptor is not required FileDescriptor dummy = new FileDescriptor(); if ((!writable) || (imode == MAP_RO)) return Util.newMappedByteBufferR(0, 0, dummy, null); else return Util.newMappedByteBuffer(0, 0, dummy, null); } //allocationGranularity的大小在我的系统上是4K //页对齐，pagePosition为第多少页 int pagePosition = (int)(position % allocationGranularity); //从页的最开始映射 long mapPosition = position - pagePosition; //因为从页的最开始映射，增大映射空间 long mapSize = size + pagePosition; try { // If no exception was thrown from map0, the address is valid //native方法，源代码在openjdk/jdk/src/solaris/native/sun/nio/ch/FileChannelImpl.c, //参见下面的说明 addr = map0(imode, mapPosition, mapSize); } catch (OutOfMemoryError x) { // An OutOfMemoryError may indicate that we've exhausted memory // so force gc and re-attempt map System.gc(); try { Thread.sleep(100); } catch (InterruptedException y) { Thread.currentThread().interrupt(); } try { addr = map0(imode, mapPosition, mapSize); } catch (OutOfMemoryError y) { // After a second OOME, fail throw new IOException("Map failed", y); } } // On Windows, and potentially other platforms, we need an open // file descriptor for some mapping operations. FileDescriptor mfd; try { mfd = nd.duplicateForMapping(fd); } catch (IOException ioe) { unmap0(addr, mapSize); throw ioe; } assert (IOStatus.checkAll(addr)); assert (addr % allocationGranularity == 0); int isize = (int)size; Unmapper um = new Unmapper(addr, mapSize, isize, mfd); if ((!writable) || (imode == MAP_RO)) { return Util.newMappedByteBufferR(isize, addr + pagePosition, mfd, um); } else { return Util.newMappedByteBuffer(isize, addr + pagePosition, mfd, um); } } finally { threads.remove(ti); end(IOStatus.checkAll(addr)); } } map0的源码实现： JNIEXPORT jlong JNICALLJava_sun_nio_ch_FileChannelImpl_map0(JNIEnv *env, jobject this, jint prot, jlong off, jlong len){ void *mapAddress = 0; jobject fdo = (*env)->

GetObjectField (env, this, chan_fd); / / linux system call refers to the file through the integer file id, where you get the file id jint fd = fdval (env, fdo); int protections = 0; int flags = 0; if (prot = = sun_nio_ch_FileChannelImpl_MAP_RO) {protections = PROT_READ; flags = MAP_SHARED } else if (prot = = sun_nio_ch_FileChannelImpl_MAP_RW) {protections = PROT_WRITE | PROT_READ; flags = MAP_SHARED;} else if (prot = = sun_nio_ch_FileChannelImpl_MAP_PV) {protections = PROT_WRITE | PROT_READ; flags = MAP_PRIVATE } / / here is the operating system call, and mmap64 is the macro definition The actual last call is mmap mapAddress = mmap64 (0, / * Let OS decide location * / len, / * Number of bytes to map * / protections, / * File permissions * / flags, / * Changes are shared * / fd, / * File descriptor of mapped file * / off) / * Offset into file * / if (mapAddress = = MAP_FAILED) {if (errno = = ENOMEM) {/ / if the mapping is not successful, throw OutOfMemoryError JNU_ThrowOutOfMemoryError (env, "Map failed") directly; return IOS_THROWN;} return handle (env,-1, "Map failed") Return ((jlong) (unsigned long) mapAddress);}

Although the zise parameter of FileChannel.map () is long, the maximum size of size is Integer.MAX_VALUE, that is, the maximum space that can only map the maximum 2G size. In fact, the MMAP provided by the operating system can allocate more space, but JAVA is limited to 2G Buffer and other ByteBuffers can only allocate a maximum buffer of 2G.

MappedByteBuffer is a buffer generated by mmap, which is created and managed by the operating system directly. Finally, JVM lets the operating system release this part of memory directly through unmmap.

Haep****Buffer

Let's take ByteBuffer as an example to illustrate the details of the Heap type Buffer.

This type of Buffer can be generated in the following ways:

ByteBuffer.allocate (int capacity)

ByteBuffer.wrap (byte [] array)

Using the incoming array as the underlying buffer, changing the array affects the buffer, and changing the buffer also affects the array.

ByteBuffer.wrap (byte [] array,int offset, int length)

Using part of the incoming array as the underlying buffer, changing the corresponding part of the array affects the buffer, and changing the buffer also affects the array.

DirectByteBuffer

DirectByteBuffer can only be generated through ByteBuffer.allocateDirect (int capacity).

The ByteBuffer.allocateDirect () source code is as follows:

Public static ByteBuffer allocateDirect (int capacity) {return new DirectByteBuffer (capacity);}

The DirectByteBuffer () source code is as follows:

DirectByteBuffer (int cap) {/ / package-private super (- 1, 0, cap, cap); / / whether the direct memory needs to be page aligned or not, I am not testing boolean pa = VM.isDirectMemoryPageAligned (); / / the page size is 4K int ps = Bits.pageSize () / / if the page is aligned, the size of the size is ps+cap,ps is one page, and the cap starts from a new page, that is, the page is aligned long size = Math.max (1L, (long) cap + (pa? Ps: 0); / / JVM maintains the size of all direct memory. If the allocated direct memory plus the maximum amount of direct memory to be allocated this time exceeds the maximum allowed to be allocated, it will cause GC, otherwise allow allocation and add the total amount of allocated direct memory to the size of this allocation. If the maximum allowed value is exceeded after GC, / / then throw new OutOfMemoryError ("Direct buffer memory"); Bits.reserveMemory (size, cap); long base = 0; try {/ / right, unsafe can directly manipulate the underlying memory base = unsafe.allocateMemory (size) } catch (OutOfMemoryError x) {, / / did not allocate successfully, subtracting the amount of allocated direct memory that was just added. Bits.unreserveMemory (size, cap); throw x;} unsafe.setMemory (base, size, (byte) 0); if (pa & & (base% ps! = 0)) {/ / Round up to page boundary address = base + ps-(base & (ps-1));} else {address = base } cleaner = Cleaner.create (this, new Deallocator (base, size, cap); att = null;}

The source code for unsafe.allocateMemory () is in openjdk/src/openjdk/hotspot/src/share/vm/prims/unsafe.cpp. Specific source codes are as follows:

UNSAFE_ENTRY (jlong, Unsafe_AllocateMemory (JNIEnv * env, jobject unsafe, jlong size) UnsafeWrapper ("Unsafe_AllocateMemory"); size_t sz = (size_t) size; if (sz! = (julong) size | | size < 0) {THROW_0 (vmSymbols::java_lang_IllegalArgumentException ());} if (sz = = 0) {return 0;} sz = round_to (sz, HeapWordSize) / / the last call is upright chars * ptr = (upright chars *):: malloc (size + space_before + space_after), namely malloc. Void* x = os::malloc (sz, mtInternal); if (x = = NULL) {THROW_0 (vmSymbols::java_lang_OutOfMemoryError ());} / / Copy::fill_to_words ((HeapWord*) x, sz / HeapWordSize); return addr_to_java (x); UNSAFE_END

JVM allocates consecutive buffers through malloc, which can be called by the operating system directly as buffer parameters.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.