Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of Redis compression list, which has certain reference value, and friends who need it can refer to it. I hope you will have a lot of success after reading this article. Let's take a look at it together.

This article focuses on one of the ways Redis stores data, compressed lists.

This article will introduce

1. Use scenarios of ziplist

2. How to save memory?

3. Storage format for compressed lists

4. Problem of Chain Renewal

5. conf file configuration.

In practice, the operation is mainly to configure the conf configuration file, there is no exact value, more is the experience value. There are also projects that use the original default values directly. This article will be helpful in understanding the underlying storage logic of a database. To cultivate energy storage, we should not only be rich, but also deep. I hope this article is useful for those who are also learning Redis. (Recommended tutorial: Redis tutorial)

I. Use scenarios of ziplist:

To optimize data storage and save memory, Redis implements the optimization of using compressed lists at the bottom of lists, dictionaries (hash keys) and ordered sets.

For example, if a hash key stores a short string, Redis stores it as a compressed list, i.e., as a byte array. A hash key that stores a smaller integer value internally will also be stored as a node in the compressed list. Similarly, list keys store small bits of data similarly to hash keys.

In this way, compressed lists are not a storage data structure in Redis that developers can call directly, but rather an effort at the bottom to optimize data storage in Redis. It's important to understand this.

2. How to achieve the effect of saving memory?

A compressed list is a serialized data structure whose function is to store a series of data and its encoded information in a contiguous memory area that is physically contiguous. But logically divided into multiple components, namely nodes. The goal is to reduce unnecessary memory overhead as much as possible under certain controllable time complexity conditions, so as to achieve the effect of saving memory. Need to understand is how to achieve the effect of saving memory, but also need to go to the decompression list storage format.

III. Storage format of compressed list:

Ziplist is one of the underlying implementations of Redis list keys, hash keys, and ordered set keys, essentially a serialized data storage structure. Unlike ordinary cases, Redis uses a double-ended list to represent lists, a hash table to represent hash keys, and a hash table + jump table to represent ordered sets. When a list or hash dictionary/ordered set contains only a small number of items, and each list or hash item/ordered set item is a small integer, or a relatively short string. Then Redis will use compressed lists for the underlying implementation.

A compressed list consists of a series of contiguous memory blocks encoded by Redis, each memory block being called an entry, and a compressed list can contain many nodes. Each node stores data in either byte array format (Chinese strings, etc., are converted to byte arrays) or integer values.

The length of the byte array can be one of the following:

1. Length less than or equal to 63 bytes (2 to the sixth power)

2. Length less than or equal to 16383 bytes (2 to the 14th power)

3. Length less than or equal to 4294967295 bytes (2 to the 32nd power)

The integer value may be one of six:

1. 4-bit unsigned integer between 0 and 12

2. signed integer 1 byte long

3. Signed integer 3 bytes long

4. int16_t Type integer

5. int32_Type integer

6. int64_t Type integer

Differences between normal and compressed list storage formats:

A list storage structure is typically a double-ended list, where each value is represented by a node, and each node has pointers to the previous node and the next node, as well as pointers to the string values contained in the node. The string value is stored in three parts: the first part stores the length of the string, the second part stores the number of bytes remaining in the string value, and the third part stores the string data itself. So a node usually needs to store three pointers, two integers that record string information, the string itself, and one extra byte. Overall the overhead is large (21 bytes).

Format of compressed list node:

Each node consists of three parts: previous_entry_length,encoding, and content. When traversing the compressed list, it is traversed from the back to the front.

1. previous_entry_length records the length of the previous node, just subtract this value from the current pointer to reach the starting address of the previous node.

2. encoding records the type and length of data stored by the node content attribute

3. content records the value of a node

Obviously compressing lists saves a lot of storage space. But it also raises the following questions.

IV. Problems of Chain Renewal:

In general, if the overall length of the previous node is less than 254 bytes, the previous_entry_length attribute requires only 1 byte of space to store this length value. When the current node is larger than 254 bytes, the previous_entry_length attribute uses 5 bytes to record the length value.

When a new node is inserted before a node with a length of about 254 bytes, it is necessary to add the previous_entry_length to record the offset from this node to the new node. At this point, the length of the node must be greater than 254 bytes. Therefore, the next node of this node cannot record the information of this node with only one byte of previous_entry_length, but needs 5 bytes to record it. If the length of several consecutive nodes is about 254 bytes, the insertion and deletion of nodes before/after one of them (the reasoning of deletion is opposite to insertion, and the previous node may be recorded with 5 bytes instead of 1 byte) may cause chain updates. Obviously, this is very unfavorable to the efficiency of system operation. However, in practice, this situation is still relatively rare.

Double-ended lists are much easier to update, add, and delete nodes. Because the information stored in each node is relatively independent.

Practical significance:

To estimate how many bytes of storage space a node will occupy, format the fields appropriately so that the stored field values do not fall around 254 bytes (excluding the encoding attribute and previous_entry_length attribute).

To view the length of strings and hash keys in Redis:

1. Query the length of the value corresponding to the string key

Command:

Strlen

For example:

127.0.0.1:6379> strlen m_name

(integer) 8

2. Query hash key length of a field

Command:

Hstrlen

For example:

127.0.0.1:6379> hstrlen good_list good_list1

(integer) 226

V. Conf file configuration:

By modifying the configuration file, you can control whether compressed lists are used to store the maximum number of elements and the size of the maximum elements of a key

Configuration in Conf file:

1.

[] -max-ziplist-entries: indicates the maximum number of elements for a key, that is, the number of nodes in a key under the specified value will be stored in a compressed list.

[] -max-ziplist-value: Indicates the maximum size of each node in the compressed list

In actual use, an element of a list key/hash key often stores a relatively large amount of information, which will be greater than 64 bytes, so it is likely to be larger than 64 when configured. At the same time, considering the actual storage capacity of data and the size of the previous_entry_length mentioned above, the [] -max-ziplist-value is reasonably configured.

Profile content:

# Hashes are encoded using a memory efficient data structure when they have a# small number of entries, and the biggest entry does not exceed a given# threshold. These thresholds can be configured using the following directives.hash-max-ziplist-entries 512hash-max-ziplist-value 64 ordered set keys # Similarly to hashes and lists, sorted sets are also specifically encoded in# order to save a lot of space. This encoding is only used when the length and# elements of a sorted set are below the following limits:zset-max-ziplist-entries 128zset-max-ziplist-value 64 List keys, special, directly use the specified size kb byte number representation (some conf file list keys and hash key expressions are not much different)# Lists are also encoded in a special way to save a lot of space.# The number of entries allowed per internal list node can be specified# as a fixed maximum size or a maximum number of elements.# For a fixed maximum size, use -5 through -1, meaning:# -5: max size: 64 Kb

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.