Shulou(Shulou.com)06/01 Report--

This article introduces what varint means, the content is very detailed, interested friends can use it for reference, I hope it can be helpful to you.

As readers familiar with Kafka, you must know that many length fields in Kafka messages are encoded in a variable length format, so what is this variable encoding format? yes, it is the varint encoding format that we are going to talk about today.

Byte order

In the process of information transmission, the computer uses a certain coding format to encode the data into binary, and when the data receiver receives the data, it will also decode the binary data into the response format. In a computer, a binary number (0 or 1) represents 1 Bit, also known as 1 byte (byte).

Byte order refers to the order in which multiple bytes are arranged in communication, and byte order is currently available in two formats:

Big end order: when the highest byte of an integer is stored in front of the lowest byte, it is called large end order. Popularly speaking, it is to convert the small end order in binary according to the writing order of numbers: the lowest byte of an integer in front of the highest byte is called small end order. Popularly speaking, it is converted into binary according to the reverse order of digital writing.

The figure below is the binary format of the intermittent and small end order of the number 123456:

First of all, the data of int type occupies 4 bytes. Taking the big end order as an example, we can see that the first byte of the high order of 123456 is useless. We can use three bytes to represent 123456. Because the value of the length field of Kafka will be much less than 123456, even 1 byte can indicate that it will waste a lot of space if we still use int to represent length. Kafka has a variable-length representation of its length fields in its v2 message format, which is encoded by varint.

Varint

Varint is not only used in kafka, but the famous Protocol Buffers also uses varint coding.

Varint is a way to serialize an integer with one or more bytes. It can encode a fixed-byte integer into a variable-length byte.

The highest bit of each byte in varint encoding does not store the true representation of the number, but indicates whether the current byte still belongs to the current data, with 1 for yes and 0 for no (that is, the byte is the last byte of the current data). The lower 7 bits of each byte are used to represent the binary complement with 7 bits as a set of stored numbers, with the least significant array first, which indicates that the varint codes are arranged in small end order.

In the figure, the number 123456 is encoded by varint. 123456 is represented as 1 11100010 01000000 in binary. Each time 7 bits are taken from low to high, plus the most significant bits, it becomes 110000001100010000000111.

Now we implement varint encoding and decoding through a piece of Java code, where only data of unsigned Interger type is implemented.

Public class VarInt {

Public static void writeUnsignedVarint (int value, DataOutput output) throws IOException {/ / value & whether the current byte of 0xffffff80 is the last byte, if not, execute while while ((value & 0xffffff80)! = 0) {/ / value & 0x7f to ensure that the lowest 7 bits of the integer can be fetched / / | the highest bit of 0x80 padding byte is 1. Because the current byte is not the last byte of the data byte b = (byte) ((value & 0x7f) | 0x80) Output.writeByte (b); System.out.println (b); value > = 7;} / / write the last byte System.out.println (value); output.writeByte (value);}

Public static int readUnsignedVarint (ByteBuffer buffer) {int value = 0; int b; int i = 0; while ((b = buffer.get ()) & 0x80)! = 0) {value | = (b & 0x7f) = 28) {throw new IllegalArgumentException ("illegal varint");}} value | = b

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.