Shulou(Shulou.com)11/24 Report--

This article comes from the official account of Wechat: programming Technology Universe (ID:xuanyuancoding), author: Xuanyuan Wind O

Data Porter Hi, this is Ah Q from CPU Workshop 1. I haven't seen you for a while.

Still remember last time that we used DMA technology in our factory (too slow to bear! CPU is operating on hard drives and network cards again! After that, we are finally liberated and no longer have to move data between network cards, hard drives and memory.

Some time ago, I went to visit Huzi in No. 2 workshop and found that he was busy sweating.

"Brother, what task did you get? look, I'll give you a favor."

As soon as Huzi saw me coming over, he wiped the sweat off his head and said, "I'm carrying data. I've just finished a batch. I'm exhausted."

I have some doubts: "We now have DMA technology, looking for outsourced DMA controller handling ah, why do you do it yourself?"

"DMA is used to move data between I / O external devices and memory. My task now is to copy and copy between memory. This DMAC can't help me. I can't copy it myself."

I shrunk my mouth. "Yeah, I hope I don't get this kind of assignment."

"I won't talk to you first. I have more work to do." before the tiger's ass was hot, he got up and went to work, and I got up to get ready to go back.

"Damn it! why do you have to copy this data again!" I had only taken two steps when I heard Huzi's complaint.

I turned around and asked, "what's the matter?"

"I just copied this data from the kernel address space to the user-mode address space, and before I took a breath, I was asked to move from the user-mode to the kernel address space again, which is too troublesome for me."

I patted him on the shoulder and said: "Hey, there is no way, we are part-time workers, which round we pick and choose ah, come on!"

As I cheered him up, I prayed secretly that I would not be given this kind of work and whether I was tired or skilled.

There was an unexpected situation in the sky, and it was not long before I returned to the No. 1 workshop, I also had this kind of thing. As the old saying goes, there is nothing to be afraid of.

At first I was able to bear it, but after a long time, I couldn't restrain my dissatisfaction. I didn't know the pain until it fell on myself.

Four copies of the data the next day, I asked Huzi to talk to the memory management department of the operating system about it.

The memory management department unexpectedly kicks the ball, saying that this matter is not their responsibility, let's find the I / O department, there is no way, we came to the I / O department to respond to this matter.

After hearing our complaint, the people in the I / O department were also very helpless: "guys, we really didn't mean to tease you." there is really no way for you to move the data twice before, this is what the above application should write. They have to read out the files on the hard drive and then send them out through the network card. Don't you have to move this read and write twice? "

File.read (file, buf, len); Socket.send (socket, buf, len); "hard disk? network card? this, we have DMA technology, just to solve the problem of data transfer with them, why do we have to copy it back and forth between memory?" Oh, I asked.

The other party saw our doubts and drew a picture on the whiteboard next to us:

"you see, the data goes from the hard disk to the network card, and because of the participation of the application, they need to read their cache in user space before sending it out, so that there are a total of four data transfers. DMAC can help you carry these two links from hard disk to kernel space and from kernel space to network card. However, the remaining two copies of user space and kernel space are still up to you to carry. "

"so, alas, it seems that there is no way to avoid it. Let's go back first," Huzi said dejectedly after reading the picture.

However, I don't want to give up. I want to find out what can be optimized and improved in this picture.

"can we just copy the data once in the kernel space instead of going to the application, so we can move it one less time?" Well, I threw out a question.

"how is that possible? if he doesn't read it, how can he send it out later? no, no, no." people in the I / O department shook their heads again and again.

"you can still send it, ah. You see, in the end, you will send the data from the kernel space to the network card, just to avoid the waste of data wandering around the user space." I changed the picture he drew and refused to give up explanation.

The other party was awakened by my words, rolling his eyes from side to side, thinking over and over.

A moment later, he returned: "still no, if someone wants to modify or decrypt the data of the read file, he still has to read his user space cache."

I thought for a moment, this seems to be inevitable, said: "then we will recognize this situation, anyway, in my experience, you do not talk much about this situation." in most cases, the data is intact from the kernel to the user space, and back to the kernel from the user space. "

The people in the I / O department had nothing more to say, nodded and agreed, saying that they could not make a decision until they reported our opinions to the top of the Linux empire for discussion. Let's just go back and wait for the news.

Zero copy technology, but then the work was too busy and there was no news from the operating system. Slowly we forgot about it until a few days ago.

"Ah Q, have you heard that recently a new company in the Linux empire can write data from the network card to the hard drive by bypassing our CPU?" Huzi came to me in a hurry.

"it's impossible. According to our previous plan, we have to copy it at least once."

"not at all. They claim to be zero-copy technology."

We hurriedly put aside our work and went to find out what was going on.

It turns out that the Linux Empire recently launched a new API called sendfile.

Ssize_t sendfile (int out_fd, int in_fd, off_t * offset, size_t count); you only need to specify the descriptor of the open file and the network interface descriptor to be sent, and you can send the file directly through the network.

Once again, we came to the I / O department of the operating system, and the other side saw us and received us warmly.

"you're just in time, and I haven't had a chance to tell you. Last time, your idea was very good, and the top echelons of the Empire attached great importance to it. As soon as I responded to it, I immediately adopted your advice. As you probably know, a new API has been launched for applications, saving the cost of data going around the user space for nothing. It was very popular as soon as it was launched, and thank you very much."

"I see. I said that there has been a lot less work on how to move data recently. But how do you manage to make zero copies?"

The people in the I / O department glanced at us and smiled proudly. "when discussing your plan, the senior level of the Empire felt that it could be further optimized by directly giving the address and length of the data buffer read from the hard disk to the network socket descriptor, so you would not have to move the data again and completely liberate you, so it was called zero copy!"

We nodded and praised again and again.

"it's not over yet! our Linux empire has also extended this technology to file data replication, adding another API:splice, and file copying can lighten your burden in the future."

Ssize_t splice (int fd_in, loff_t * off_in, int fd_out, loff_t * off_out, size_t len, unsigned int flags); when we went back, we told the news to the whole factory, and everyone was overjoyed that all the workshops had suffered for a long time.

Egg 1 is on the distant Windows empire.

"Minister, I heard that the Linux Empire has launched a sendfile, known as zero copy."

"is there such a thing? we can't lag behind. Hurry up and study it."

A new company in the Egg 2Linux Empire specializes in network packet analysis.

"Boss, the packet has to be processed by the Linux Imperial protocol stack department every time to get it, which is too slow."

"can you bypass the protocol stack and grab the packet directly?"

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.