Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what is the necessary NIO knowledge for reading a distributed framework". In daily operation, I believe that many people have doubts about reading the necessary NIO knowledge of a distributed framework. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the questions of "reading the necessary NIO knowledge of a distributed framework". Next, please follow the editor to study!

I. opening

When reading a distributed open source project, the most important thing is to understand the communication framework of the project.

Because a distributed open source framework is usually deployed in a cluster, different nodes and nodes need to communicate with each other to complete complex functions, and when reading these source codes, if you do not understand its communication mechanism, you will be lost in the code, like walking into a primeval forest.

For example, HDFS uses a self-encapsulated Hadoop Rpc;Spark underlying communication framework that uses Netty;, while the recently read Kafka source code uses native Java NIO.

So this time, let's talk about the main knowledge points of Java NIO.

Second, multi-graph to understand the three core concepts of NIO

When it comes to NIO, there are three core concepts: Channel, buffer, and selector.

Come straight to the point, it may sound a little confused, and we need to start from the beginning.

1. Channel

In the past, when the concurrency requirement was not very high, CPU was in charge of handling input and output (interrupts), as shown below:

The user program initiates read and write requests to the server, and cpu processes these requests directly. This has a drawback, when there are a lot of IO requests, it will take up a lot of CPU, so that the processing capacity of the whole system will be reduced.

With the development of computers, a new way has emerged to use DMA to handle IO requests with full authority, as shown in the following figure:

DMA is Direct Memory Access, direct memory access control.

Why add this device? Because the CPU interrupt mode can not meet the requirements of data transmission speed, because in the interrupt mode, each interrupt needs to save the breakpoint and the site, and when the interrupt returns, the breakpoint and the site should be restored.

All these reasons make it difficult for the interrupt mode to meet the transmission speed requirements of high-speed peripherals.

Therefore, there is a device such as DMA, in the process of data transmission in DMA mode, when the DMA O device needs to transfer data, it makes a request for DMA transmission to CPU through the DMA controller. After the CPU response, it will give up the system bus, and the DMA controller will take over the bus for data transmission. At this time, CPU can do its own thing in addition to doing some initialization operations.

However, with DMA, it still can not meet the needs of the rapid development of business, because when there are too many requests, there will be bus conflicts.

So there is a channel (Channel), which is different from DMA in that it has its own instruction system and program and is a coprocessor, while DMA can only achieve fixed data transfer control.

The Channel in Java NIO is the implementation of the channel in the figure above.

2. Buffer

The concept of channels is understood, and buffers are easy to understand.

A channel represents a connection that is opened to an Iamp O device (for example, files, sockets), but the channel itself does not store data. What is really used as a data transmission carrier is the buffer.

When the application wants to write data, it needs to write the data to the buffer first, and then the channel is responsible for sending the buffer data to the destination (file, disk, network), and then fetching the data from the buffer.

If you need to use the NIO system, you need to get the channel used to connect the Imax O device and the buffer used to hold the data, and then manipulate the buffer to process the data.

3. Selector

A selector, also known as a multiplexer, is a non-blocking Icord O. Now that we talk about the non-blocking type, we must first talk about the blocking type. The blocking type is shown in the following figure:

When the client makes a read and write request to the server, the server thread keeps looking for data in the kernel address space.

When the client has no data to send, the server thread will wait and nothing can be done in the meantime.

Until the data is sent from the client, the data will be copied from the kernel address space to the user address space, and then the data will be read.

This leads to a sharp decline in performance if there are a large number of requests coming, and the subsequent requests have to wait for the previous requests to be completed, resulting in a large number of queues, unable to make full use of cpu resources.

And look at how the selector works.

Now the communication between client and server is channel + buffer, so all channels will be registered with selector. The selector monitors the Istroke O status of these channels, such as connection, read, and write.

When an event on a channel is fully ready, the selector assigns the task to one or more threads on the server.

When the client does not have an event ready, the server thread will not block, it can do its own thing, until the client event is ready to deal with.

Compared with the blocking type, this non-blocking type can make further use of cpu resources.

Third, understand the concept, and then learn API

1. API of the buffer

To fully understand the buffer, you must know the four properties of the buffer, mark,position,limit,capacity, which you only need to run through the code.

(1) allocate a buffer of a certain size

/ / 1. Assign a buffer of the specified size ByteBuffer buffer = ByteBuffer.allocate (10); System.out.println ("- alocate"); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()); System.out.println ("capacity:" + buffer.capacity ())

Running result:

-alocate- position:0 limit:10 capacity:10

Here we allocate a 10-byte buffer, which opens up 10 bytes of space on the final byte [] hb; attribute of ByteBuffer.

So the capacity capacity is 10, the maximum location of limit readable and writable data is 10, and the position is 0 where the data can be manipulated.

(2) write data to the buffer

/ / 2. Write data to buffer String str = "abcde"; System.out.println ("- put-"); buffer.put (str.getBytes (StandardCharsets.UTF_8)); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()); System.out.println ("capacity:" + buffer.capacity ())

Running result:

-put- position:5 limit:10 capacity:10

Here we write 5 bytes of data to the buffer, so capacity and limit are still 10, but position is 5, because five have already been written.

(3) switch to the mode of reading data

/ / 3. Switch to read data mode buffer.flip (); System.out.println ("- flip-"); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()); System.out.println ("capacity:" + buffer.capacity ())

So now if we want to read some data from the buffer, we need to switch to flip mode, and flip will change the values of some properties.

Running result:

-flip- position:0 limit:5 capacity:10

Flip will change the value of position to 0 and limit to 5, which means I want to read from the beginning and can only read to the position of 5.

(4) read some data

/ / 4. Read data System.out.println ("- get-"); byte [] dest = new byte [buffer.limit ()]; buffer.get (dest); System.out.println (new String (dest,0,dest.length)); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()) System.out.println ("capacity:" + buffer.capacity ())

Running result:

-get- abcde position:5 limit:5 capacity:10

After reading the data, the position becomes 5, which means I have read to 5.

(5) repeat reading

/ / 5.rewind () buffer.rewind (); System.out.println ("- rewind-"); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()); System.out.println ("capacity:" + buffer.capacity ())

Running result:

-rewind- position:0 limit:5 capacity:10

Rewind means to read the data in buffer repeatedly.

(6) clear data

/ / 6.clear () buffer.clear (); System.out.println ("- clear-"); System.out.println ("position:" + buffer.position ()); System.out.println ("limit:" + buffer.limit ()); System.out.println ("capacity:" + buffer.capacity ())

Running result:

-clear- position:0 limit:10 capacity:10

After clear (), position goes back to 0, limit goes back to 10, and you can start writing data all over again, capable of writing 10 bytes.

However, it should be noted that the data in the buffer has not been emptied, and the data is still in it, in a "forgotten" state. These pointers are back to their original state.

(7) Mark

This is the fourth attribute: mark.

Mark can record the location of the position. You can return to the position of mark through the reset () method.

@ Test public void test2 () {/ / assign 10 bytes String str = "abcde"; ByteBuffer buffer = ByteBuffer.allocate (10); buffer.put (str.getBytes (StandardCharsets.UTF_8)); / / switch to read mode and read 2 bytes buffer.flip (); byte [] dest = new byte [buffer.limit ()] Buffer.get (dest, 0,2); System.out.println (new String (dest, 0,2)); System.out.println (buffer.position ()); / / mark record the current position buffer.mark (); / / read two more bytes of buffer.get (dest, 2,2); System.out.println (new String (dest, 2,2)) System.out.println (buffer.position ()); / / reset, return to the location of mark buffer.reset (); System.out.println (buffer.position ());} execution result: ```tex ab 2 cd 4 2

2. Use channels, buffers and selectors to complete a network program

(1) Server

Test public void testServer () throws IOException {ServerSocketChannel serverSocketChannel = ServerSocketChannel.open (); serverSocketChannel.configureBlocking (false); serverSocketChannel.bind (new InetSocketAddress (8989)); Selector selector = Selector.open (); serverSocketChannel.register (selector, SelectionKey.OP_ACCEPT); while (selector.select () > 0) {Iterator iterator = selector.selectedKeys () .iterator () While (iterator.hasNext ()) {SelectionKey key = iterator.next (); if (key.isAcceptable ()) {SocketChannel socketChannel = serverSocketChannel.accept (); socketChannel.configureBlocking (false); socketChannel.register (selector, SelectionKey.OP_READ) } else if (key.isReadable ()) {SocketChannel channel = (SocketChannel) key.channel (); ByteBuffer byteBuffer = ByteBuffer.allocate (1024); int len = 0; while ((len = channel.read (byteBuffer)) > 0) {byteBuffer.flip () System.out.println (new String (byteBuffer.array (), 0, len)); byteBuffer.clear ();} iterator.remove ();}}

1. First use ServerSocketChannel.open () to open a channel and set it to non-blocking mode

2. Bind to port 8989

3. Register the channel with the selector

4. While loop, whether there is an event on the selector, and if the event is a client connection event, open a SocketChannel, register as a non-blocking mode, and register an event to read data to the selector

5. When the client sends data, it can open a channel to read the data on the buffer.

6. And at this time, the server can accept requests from multiple clients at the same time.

(2) client

@ Test public void testClient () throws IOException {SocketChannel socketChannel = SocketChannel.open (new InetSocketAddress ("127.0.0.1", 8989)); socketChannel.configureBlocking (false); ByteBuffer byteBuffer = ByteBuffer.allocate (1024); byteBuffer.put (new Date (). ToString (). GetBytes (StandardCharsets.UTF_8)); byteBuffer.flip (); socketChannel.write (byteBuffer); byteBuffer.clear (); socketChannel.close ();}

1. The client opens a SocketChannel and configures it in non-blocking mode.

2. Use ByteBuffer to send data (note that before sending, flip)

3. Close the channel.

At this point, the study of "what is the necessary NIO knowledge for reading a distributed framework" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.