Kafka, why is that fast?

Kafka is a high-throughput messaging middleware that can be deployed on inexpensive hosts.

There are two reasons for speed:

1. From the writing data side: data written is written sequentially to disk;memory mapped files

The speed of disk reading and writing depends on how you use it. When writing, you can choose sequential writing and random writing. Kafka selects the sequential writing mechanism. The reading and writing speed of this mechanism is basically the same as the speed of memory.

Benefits of using sequential writing:

1. Disk sequential writes faster than random memory

2. When operating in memory, jvm has a large amount of data in the heap, and GC is inefficient. Using disks can avoid this problem.

3. Disk cache is still available after a sequential write system cold boot.

memory mapped files

Kafka does not write data to disk in real time, it uses modern operating system paging storage to improve I/O efficiency by using memory, first writing to memory, and flushing to disk at a certain time.

Kafka provides a parameter--producer.type to control whether it is active flush. If Kafka writes to mmap, it will immediately flush and then return to Producer, which is called synchronous (sync); after writing to mmap, it will immediately return Producer without calling flush, which is called asynchronous (async).

Memory mapped files(mmap) is a memory mapped file, which can generally represent 20G data files in 64-bit systems. Its working principle uses the page of the operating system to realize the direct mapping of files to physical memory. after mapping

Operations on physical memory can also be synchronized to the hard disk. This approach allows for a large I/O boost, eliminating the user space-to-kernel overhead (reads put data in kernel space first and then copy it into user space memory)

2. From reading data

Zero copy based on sendfile

The traditional read/write method for network file transmission, in which the file data goes through four copy processes

Hard disk---kernel buf--user buf---socket dependent buffer---protocol engine

The sendfile system call provides a way to reduce the number of copies and improve file transfer performance.

Hard disk---kernel buf---socket dependent cache---protocol engine

After kernel version 2.4, file descriptors changed as a result, and sendfile implemented a simpler approach, again eliminating one copy operation.