Shulou(Shulou.com)05/31 Report--

This article mainly introduces the Redis memory model and application knowledge points of what related knowledge, the content is detailed and easy to understand, the operation is simple and fast, has a certain reference value, I believe that you will have something to gain after reading this Redis memory model and application knowledge points, let's take a look at it.

Preface

Redis is one of the most popular in-memory databases, which greatly improves the speed of reading and writing by reading and writing data in memory. It can be said that Redis is an indispensable part of realizing high concurrency of websites. When we use Redis, we will come into contact with the five object types of Redis (string, hash, list, collection, ordered collection). The rich types are one of the advantages of Redis over Memcached and so on. On the basis of understanding the usage and characteristics of the five object types of Redis, further understanding the memory model of Redis is of great help to the use of Redis, such as:

1. Estimate Redis memory usage. So far, the cost of using memory is still relatively high, and there can be no scruples in using memory. According to the demand, a reasonable evaluation of the memory usage of Redis and the selection of appropriate machine configuration can save costs while meeting the demand.

2. Optimize memory usage. Understanding the Redis memory model allows you to choose more appropriate data types and encodings and make better use of Redis memory.

3. Analyze and solve the problem. When there are some problems such as blocking and memory occupation in Redis, the cause of the problem can be found as soon as possible, which is convenient to analyze and solve the problem.

This article mainly introduces the memory model of Redis (taking 3.0 as an example), including the memory occupied by Redis and how to query, the coding mode of different object types in memory, memory allocator (jemalloc), simple dynamic string (SDS), RedisObject, etc., and then introduces the application of several Redis memory models.

1. Redis memory statistics

If you want to do a good job, you must first sharpen its tools. Before explaining Redis memory, first explain how to count the memory usage of Redis.

After the client connects to the server through redis-cli (unless otherwise specified, the client will always use redis-cli), you can check the memory usage through the info command:

Info memory

Among them, the info command can display a lot of information about the redis server, including server basic information, CPU, memory, persistence, client connection information, and so on; memory is a parameter, indicating that only memory-related information is displayed.

Several important instructions in the returned results are as follows:

(1) the total amount of memory allocated by the used_memory:Redis allocator (in bytes), including the virtual memory used (that is, swap); the Redis allocator will describe later. Used_memory_human just appears to be more friendly.

(2) the used_memory_rss:Redis process occupies the memory of the operating system (in bytes), which is consistent with the value seen by the top and ps commands; in addition to the memory allocated by the allocator, used_memory_rss also includes the memory needed by the process itself, memory fragments, etc., but does not include virtual memory.

Therefore, used_memory and used_memory_rss, the former is the quantity obtained from the Redis point of view, the latter is the quantity obtained from the operating system point of view. The reason why the two are different, on the one hand, is that memory fragments and Redis processes need to occupy memory, so that the former may be smaller than the latter, on the other hand, the existence of virtual memory makes the former larger than the latter.

In practical applications, the amount of data of Redis will be relatively large, and the memory consumed by the process will be much smaller than the amount of Redis data and memory fragments. Therefore, the ratio of used_memory_rss to used_memory becomes a parameter to measure the fragmentation rate of Redis memory; this parameter is mem_fragmentation_ratio.

(3) mem_fragmentation_ratio: memory fragmentation ratio, which is the ratio of used_memory_rss / used_memory.

Mem_fragmentation_ratio is generally greater than 1, and the higher the value, the greater the percentage of memory fragments. Mem_fragmentation_ratio1), called a shared object. Redis to save memory, when some objects are repeated, the new program will not create new objects, but will still use the original objects. The object that is reused is the shared object. Currently, shared objects only support string objects with integer values.

The concrete realization of shared object

Redis's shared objects currently only support string objects with integer values. The reason for this is actually a balance between memory and CPU: sharing objects reduces memory consumption, but it takes extra time to determine whether two objects are equal.

For integer values, the operation complexity is judged to be O (1).

For ordinary strings, the judgment complexity is O (n).

For hashes, lists, sets and ordered sets, the complexity of judgment is O (n ^ 2).

Although shared objects can only be string objects with integer values, all five types can use shared objects (elements such as hashes, lists, and so on).

In the current implementation, when the Redis server initializes, it creates 10000 string objects with integer values of 0,9999, and these shared objects can be used directly when Redis needs to use string objects with the value of 0room9999. The number 10000 can be changed by adjusting the value of the parameter REDIS_SHARED_INTEGERS (OBJ_SHARED_INTEGERS in 4.0).

The number of references to a shared object can be viewed through the object refcount command, as shown in the following figure. The results page of the command execution confirms that only integers between 0,9999 will be shared objects.

(5) ptr

The ptr pointer points to specific data, as in the previous example, set hello world,ptr points to the SDS that contains the string world.

(6) Summary

To sum up, the structure of redisObject is related to object type, encoding, memory collection, and shared objects; the size of a redisObject object is 16 bytes:

4bit+4bit+24bit+4Byte+8Byte=16Byte .

4 、 SDS

Instead of directly using the C string (that is, an array of characters ending with the empty character'\ 0') as the default string representation, Redis uses SDS. SDS is an abbreviation for simple dynamic string (Simple Dynamic String).

(1) SDS structure

The structure of sds is as follows:

Struct sdshdr {int len; int free; char buf [];}

Where buf represents a byte array to store strings, len represents the length used by buf, and free represents the unused length of buf. Here are two examples.

Photo Source: "Redis Design and implementation"

From the structure of SDS, you can see that the length of the buf array = free+len+1 (where 1 represents the empty character at the end of the string); therefore, the space occupied by a SDS structure is: the length occupied by free + the length occupied by len + the length of the buf array = 4+4+free+len+1=free+len+9.

(2) comparison between SDS and C string

SDS adds free and len fields to the C string, which brings a lot of benefits:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Get string length: SDS is O (1), C string is O (n)

Buffer overflow: when using API of C string, if the string length increases (such as strcat operation) and forgets to reallocate memory, it is easy to cause buffer overflow; while SDS records the length, the corresponding API will automatically reallocate memory when it may cause buffer overflow to eliminate buffer overflow.

Memory reallocation when modifying strings: for C strings, if you want to modify strings, you must reallocate memory (release before applying), because if there is no reallocation, increasing string length will cause memory buffer overflow and decreasing string length will cause memory leak. For SDS, because len and free can be recorded, the association between string length and spatial array length is disassociated, which can be optimized on this basis: the space pre-allocation strategy (that is, allocating memory more than is actually needed) greatly reduces the probability of memory reallocation when the string length increases; the lazy space release strategy greatly reduces the probability of redistributing memory when the string length decreases hours.

Access to binary data: SDS can, C string can not. Because the C string ends with an empty character, and for some binary files (such as pictures, etc.), the content may include an empty string, so the C string cannot be accessed correctly, while SDS uses the string length len as the end of the string, so there is no problem.

In addition, because buf in SDS still uses C strings (that is, ending in'\ 0'), SDS can use some of the functions in the C string library; however, it is important to note that this can only be used when SDS is used to store text data, not when storing binary data ('\ 0' is not necessarily the end).

(3) the application of SDS and C string

Redis always uses SDS instead of C strings when storing objects. For example, the set hello world command, hello and world are stored in the form of SDS. The sadd myset member1 member2 member3 command, whether it is a key ("myset") or elements in the collection ("member1", "member2", and "member3"), is stored in the form of SDS. In addition to storing objects, SDS is also used to store various buffers.

The C string is used only if the string does not change, such as when printing a log.

IV. Object types and internal coding of Redis

As mentioned earlier, Redis supports five object types, and each structure has at least two encodings. The advantage of this is: on the one hand, the interface is separated from the implementation, and when you need to add or change the internal coding, the user's use will not be affected; on the other hand, you can switch the internal coding according to different application scenarios to improve efficiency.

The internal encodings supported by various object types of Redis are shown in the figure below (internal encoding is added in later versions of Redis3.0,Redis, not to mention; all of the internal encodings introduced in this chapter are based on 3.0):

Photo Source: "Redis Design and implementation"

The conversion of Redis internal coding is in accordance with the following rules: the coding conversion is completed when Redis writes data, and the conversion process is irreversible, so it can only be converted from small memory coding to large memory coding.

1. String

(1) Overview

Strings are the most basic type, because all keys are string types, and several complex types of elements other than strings are also strings.

The length of the string cannot exceed 512MB.

(2) Internal coding

There are three internal encodings of string types, and their application scenarios are as follows:

A long integer of int:8 bytes. When a string value is an integer, this value is represented by an long integer.

Embstr:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.