Shulou(Shulou.com)06/02 Report--

This article mainly explains the "methods of advanced features and performance tuning of Redis". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "Redis advanced features and performance tuning methods".

Overview

Redis is an open source, memory-based structured data storage medium that can be used as a database, caching service, or messaging service.

Redis supports a variety of data structures, including strings, hash tables, linked lists, collections, ordered collections, bitmaps, Hyperloglogs, etc.

Redis has the capabilities of LRU elimination, transaction implementation, and different levels of hard disk persistence, and supports replica sets and highly available solutions implemented through Redis Sentinel, as well as automatic data sharding through Redis Cluster.

The main functions of Redis are based on the single-threaded model, that is, Redis uses a single thread to serve all client requests, while Redis uses non-blocking IO and finely optimizes the algorithm time complexity of various commands. This information means:

Redis is thread-safe (because there is only one thread), all its operations are atomic, and there are no data exceptions due to concurrency.

The speed of Redis is very fast (because non-blocking IO is used, and the algorithm complexity of most commands is O (1))

Using time-consuming Redis commands is dangerous and can take up a lot of processing time from a single thread, causing all requests to be delayed. (for example, the KEYS command with a time complexity of O (N) is strictly prohibited in a production environment.)

Data structure of Redis and related common commands

This section describes the main data structures supported by Redis, as well as related common Redis commands. This section provides only a brief introduction to the Redis command and lists only the more commonly used commands. If you want to know the complete set of Redis commands, or how to use a command in detail, please refer to the official document: https://redis.io/commands

Common commands 1. Key

Redis adopts the basic data structure of Key- value, and any binary sequence can be used as the Key of Redis (such as an ordinary string or a JPEG picture).

Some considerations about Key:

Do not use overly long Key. For example, using a 1024-byte key is not a good idea, which not only consumes more memory, but also reduces the efficiency of lookups.

It is also bad that Key is so short that it lacks readability. For example, "u1000flw" saves less storage space than "user:1000:followers", but causes trouble in readability and maintainability.

It is best to use a unified specification to design Key, such as "object-type:id:attr". The Key designed with this specification may be "user:1000" or "comment:1234:reply-to".

The maximum Key length allowed by Redis is 512MB (the length limit for Value is also 512MB)

Common commands II. String

String is the basic data type of Redis. Redis does not have the concept of data types such as Int, Float, Boolean and so on. All basic types are embodied in String in Redis.

Common commands related to String:

SET: set value for a key. You can specify the validity period of the key with the EX/PX parameter, and use the NX/XX parameter to distinguish whether the key exists. The time complexity is O (1).

GET: get the value corresponding to a key, time complexity O (1)

GETSET: set value for a key and return the original value of the key with time complexity O (1)

MSET: set value for multiple key, time complexity O (N)

MSETNX: like MSET, if any of the specified key already exists, no operation is performed, and the time complexity is O (N).

MGET: get the value corresponding to multiple key, time complexity O (N)

As mentioned above, the basic data type of Redis is only String, but Redis can use String as an integer or floating point number, mainly reflected in the commands of INCR and DECR classes:

INCR: increments the value value corresponding to key by 1, and returns the value since increment. Only works on String data that can be converted to integers. Time complexity O (1)

INCRBY: increments the value value corresponding to key by the specified integer value, and returns the value since increment. Only works on String data that can be converted to integers. Time complexity O (1)

DECR/DECRBY: same as INCR/INCRBY, from increasing to decreasing.

The INCR/DECR series command requires that the value type of the operation is String and can be converted to 64-bit signed integer numbers, otherwise an error will be returned.

In other words, the value for INCR/DECR series commands must be in the range of [- 2 ^ 63 ~ 2 ^ 63-1].

As mentioned earlier, Redis uses a single-threaded model, which is naturally thread-safe, which makes it very convenient for INCR/DECR commands to achieve precise control in high concurrency scenarios.

Example 1: inventory control

Achieve accurate verification of inventory margin in high concurrency scenarios to ensure that it is not oversold.

Set the total inventory:

SET inv:remain "100"

Inventory deduction + margin check:

DECR inv:remain

When the value returned by the DECR command is greater than or equal to 0, the inventory margin check is passed, and if a value less than 0 is returned, the inventory is exhausted.

Assuming that there are 300 concurrent requests for inventory deduction at the same time, Redis can ensure that these 300 requests get a return value of 99 to-200, each request gets a unique return value, and there will never be a situation where two requests get the same return value.

Example 2: self-increasing sequence generation

Implement the Sequence function similar to RDBMS to generate a series of unique serial numbers

Set the sequence start value:

SET sequence "10000"

Get a sequence value:

INCR sequence

You can simply use the return value as a sequence.

Get a batch of (for example, 100) sequence values:

INCRBY sequence 100

Assuming that the return value is N, then the values of [N-99 ~ N] are all available sequence values.

When multiple clients apply for self-increasing sequences to Redis at the same time, Redis can ensure that the sequence values or sequence ranges obtained by each client are globally unique, and there is absolutely no case that different clients get duplicate sequence values.

Common commands 3. List

Redis's List is a chain-phenotypic data structure, and you can use commands such as LPUSH/RPUSH/LPOP/RPOP to perform insert and pop-up elements on both sides of the List. Although List also supports the ability to insert and read elements on a specific index, it has a high time complexity (O (N)) and should be used with care.

Common commands related to List:

LPUSH: inserts one or more elements to the left (that is, the header) of the specified List, returning the length of the inserted List. Time complexity O (N), where N is the number of inserted elements.

RPUSH: same as LPUSH, inserts 1 or more elements to the right (that is, tail) of the specified List

LPOP: removes an element from the left side of the specified List (that is, the header) and returns, time complexity O (1)

RPOP: same as LPOP, removes 1 element from the right side (that is, tail) of the specified List and returns

LPUSHX/RPUSHX: similar to LPUSH/RPUSH, except that if the key for the LPUSHX/RPUSHX operation does not exist, nothing will be done

LLEN: returns the length of the specified List, time complexity O (1)

LRANGE: returns the specified range of elements in the specified List (double-sided inclusion, that is, LRANGE key 010 returns 11 elements), time complexity O (N). You should try to control the number of elements to be fetched at once. Getting too many List elements at once can cause delays, and avoid using full traversal operations such as LRANGE key 0-1 for List whose length is unpredictable.

List-related commands that should be used with caution:

LINDEX: returns the element on the specified List, the specified index, or nil if the index is out of bounds. The index value is looped, that is,-1 represents the last position of List and-2 represents the penultimate position of List. Time complexity O (N)

LSET: set the element on the specified List specified index to value. If the index is out of bounds, an error is returned, and the time complexity is O (N). If you are operating on a header / trailer element, the time complexity is O (1).

LINSERT: inserts a new element before / after the specified element in the specified List and returns the List length after the operation. Returns-1 if the specified element does not exist. If the specified key does not exist, nothing will be done, time complexity O (N)

Because the List of Redis is linked list structure, the algorithm efficiency of the above three commands is low, and the List needs to be traversed. The time-consuming of the command can not be estimated, and the time-consuming will obviously increase when the length of the List is large, so it should be used with caution.

In other words, Redis's List is actually designed to implement queues, not lists like ArrayList. If you don't want to implement a double-ended queue, try not to use Redis's List data structure.

In order to better support the characteristics of the queue, Redis also provides a series of blocking operation commands, such as BLPOP/BRPOP, which can achieve the ability similar to BlockingQueue, that is, block the connection when the List is empty and return when there is an object in the List that can dequeue. Commands for blocking classes are not discussed in detail here, but please refer to the "Blocking operations on lists" section of the official documentation (https://redis.io/topics/data-types-intro).

Common commands 4. Hash

Hash is a hash table. Redis's Hash, like the traditional hash table, is a field- value data structure, which can be understood as moving HashMap into Redis.

Hash is very suitable for representing the data of object types, just use the field in Hash to correspond to the field of the object.

The advantages of Hash include:

Binary search can be implemented, such as "find the age of a user whose ID is 1000"

Compared with serializing the entire object as a method of String storage, Hash can effectively reduce the consumption of network transmission.

When maintaining a collection using Hash, random access commands are provided that are much more efficient than List

Common commands related to Hash:

HSET: set the field in the Hash corresponding to key to value. If the Hash does not exist, one is automatically created. Time complexity O (1)

HGET: returns the value of the field field in the specified Hash, time complexity O (1)

HMSET/HMGET: like HSET and HGET, you can batch operate multiple field under the same key. Time complexity: O (N). N is the number of field for one operation.

HSETNX: same as HSET, but if field already exists, HSETNX will not do any operation, time complexity O (1)

HEXISTS: judge whether field exists in the specified Hash, return 1 for existence, return 0 for non-existence, time complexity O (1)

HDEL: delete one or more field in the specified Hash. Time complexity: O (N), where N is the number of field for the operation.

HINCRBY: INCRBY a field in the specified Hash with the same INCRBY command, time complexity O (1)

Hash-related commands that should be used with caution:

HGETALL: returns all field-value pairs in the specified Hash. The returned result is an array in which field and value appear alternately. Time complexity O (N)

HKEYS/HVALS: returns all field/value in the specified Hash with time complexity O (N)

The above three commands all traverse the Hash completely, and the number of field in Hash is linearly related to the time consuming of the command. For Hash whose size is unpredictable, you should strictly avoid using the above three commands and instead use the HSCAN command for cursor traversal. For more information, please see https://redis.io/commands/scan

Common commands 5. Set

Redis Set is an unordered, non-repeatable collection of String.

Common commands related to Set:

SADD: adds one or more member to the specified Set. If the specified Set does not exist, one is automatically created. Time complexity O (N), N is the number of member added

SREM: remove one or more member from the specified Set, time complexity O (N), N is the number of member removed

SRANDMEMBER: randomly returns one or more member from the specified Set. The time complexity is O (N). N is the number of member returned.

SPOP: randomly removes and returns count member from the specified Set. Time complexity O (N). N is the number of member removed.

SCARD: returns the number of member in the specified Set, time complexity O (1)

SISMEMBER: determines whether the specified value exists in the specified Set, time complexity O (1)

SMOVE: moves the specified member from one Set to another Set

Set-related commands to use with caution:

SMEMBERS: returns all member in the specified Hash with time complexity O (N)

SUNION/SUNIONSTORE: calculates the union of multiple Set and returns / stores it to another Set. The time complexity is O (N), where N is the total number of member of all sets involved in the calculation.

SINTER/SINTERSTORE: calculates the intersection of multiple Set and returns / stores it to another Set with time complexity O (N), where N is the total number of member of all sets involved in the calculation

SDIFF/SDIFFSTORE: calculates the difference between one Set and one or more Set and returns / stores it to another Set with time complexity O (N), where N is the total number of member of all sets involved in the calculation.

The above commands involve a large amount of calculation and should be used with caution, especially when the size of the Set involved in the calculation is unknown, it should be strictly avoided. You can consider traversing through the SSCAN command to get all the member of the relevant Set (see https://redis.io/commands/scan for details). If you need to do union / intersection / difference calculation, you can do it on the client side or on the Slave that does not serve real-time query requests.

Common commands VI. Sorted Set

Redis Sorted Set is an ordered, non-repeatable collection of String. Each element in Sorted Set needs to be assigned a score, and Sorted Set sorts the elements in ascending order according to score. If more than one member has the same score, sort it in ascending dictionary order.

Sorted Set is ideal for achieving rankings.

The main commands of Sorted Set:

ZADD: add one or more member to the specified Sorted Set, time complexity O (Mlog (N)), M is the number of member added, N is the number of member in the Sorted Set

ZREM: delete one or more member from the specified Sorted Set, time complexity O (Mlog (N)), M is the number of member deleted, N is the number of member in the Sorted Set

ZCOUNT: returns the number of member in the specified Sorted Set within the specified score range. Time complexity: O (log (N))

ZCARD: returns the number of member in the specified Sorted Set, time complexity O (1)

ZSCORE: returns the score of the specified member in the specified Sorted Set with a time complexity of O (1)

ZRANK/ZREVRANK: returns the ranking of the specified member in Sorted Set, ZRANK returns the ranking in ascending order, and ZREVRANK returns the ranking in descending order. Time complexity O (log (N))

ZINCRBY: same as INCRBY, self-increment the score of the specified member in the specified Sorted Set, time complexity O (log (N))

Sorted Set-related commands to use with caution:

ZRANGE/ZREVRANGE: all member,ZRANGE in the specified Sorted Set within the specified ranking range are sorted by score ascending order, ZREVRANGE is sorted by score descending order, time complexity O (log (N) + M), and M is the number of member returned this time.

ZRANGEBYSCORE/ZREVRANGEBYSCORE: returns all member within the specified score in the specified Sorted Set. The returned results are sorted in ascending / descending order. Min and max can be specified as-inf and + inf, representing all member returned. Time complexity O (log (N) + M)

ZREMRANGEBYRANK/ZREMRANGEBYSCORE: removes all member within the specified ranking range / specified score range in the Sorted Set. Time complexity O (log (N) + M)

The above commands should try to avoid passing parameters such as [0-1] or [- inf + inf] to do an one-time complete traversal of Sorted Set, especially if the size of Sorted Set is unpredictable. Vernier traversal can be achieved by using the ZSCAN command (see https://redis.io/commands/scan for details), or by limiting the number of member returned through the LIMIT parameter (applicable to the ZRANGEBYSCORE and ZREVRANGEBYSCORE commands).

Common commands 7, Bitmap and HyperLogLog

These two data structures of Redis are not commonly used compared with the previous ones, so we will only give a brief introduction in this article. If you want to learn more about these two data structures and their related commands, please refer to the relevant sections in the official document https://redis.io/topics/data-types-intro.

Bitmap is not an actual data type in Redis, but a way to use String as a Bitmap. It can be understood as converting String to an array of bit. Using Bitmap to store simple data of type true/false is extremely space-saving.

HyperLogLogs is a data structure mainly used for quantitative statistics. Like Set, it maintains a non-repeatable String collection, but HyperLogLogs does not maintain specific member content, only the number of member. That is, HyperLogLogs can only be used to calculate the number of non-repeating elements in a collection, so it saves a lot of memory space than Set.

Other common commands

EXISTS: determines whether the specified key exists. Return 1 means it exists, 0 means it does not exist, and time complexity O (1)

DEL: delete the specified key and its corresponding value. Time complexity O (N). N is the number of key deleted.

EXPIRE/PEXPIRE: sets the validity period for a key (in seconds or milliseconds) with a time complexity of O (1)

TTL/PTTL: returns the remaining valid time of a key, in seconds or milliseconds, with time complexity O (1)

RENAME/RENAMENX: rename key to newkey. When using RENAME, if newkey already exists, its value is overwritten; when using RENAMENX, if newkey already exists, nothing will be done, and the time complexity is O (1).

TYPE: returns the type of the specified key, string, list, set, zset, hash. Time complexity O (1)

CONFIG GET: get the current value of a configuration item in Redis. You can use the * wildcard character, time complexity O (1).

CONFIG SET: set a new value for a configuration item of Redis, time complexity O (1)

CONFIG REWRITE: let Redis reload the configuration in redis.conf

Redis performance tuning

Although Redis is a very fast medium for in-memory data storage, it does not mean that Redis will not cause performance problems.

As mentioned earlier, Redis uses a single-threaded model, and all commands are executed sequentially by a single thread, so when a command takes a long time to execute, it slows down all subsequent commands, which makes Redis more sensitive to the efficiency of each task.

For the performance optimization of Redis, we mainly start from the following aspects:

First and foremost, make sure that Redis is not asked to execute time-consuming commands

Use pipelining to combine execution of continuously executed commands

The Transparent huge pages function of the operating system must be turned off:

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

| | 1 |

| |

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

| |

If you run Redis in a virtual machine, there may be inherent delays in the virtual machine environment. The inherent delay can be viewed with the. / redis-cli-intrinsic-latency 100 command. At the same time, if there are high requirements for the performance of Redis, Redis should be deployed directly on the physical machine as far as possible.

Check the data persistence policy

Consider introducing read-write separation mechanism

Long time-consuming command

The time complexity of most Redis read and write commands ranges from O (1) to O (N), and the time complexity of each command is described in the text and official documentation.

Generally speaking, the O (1) command is safe, and the O (N) command should be used with caution, and should be avoided if the order of magnitude of N is unpredictable. For example, executing HGETALL/HKEYS/HVALS commands on an unknown number of field Hash data, these commands are usually executed very quickly, but if there is a large number of field in this Hash, the time consuming will increase exponentially.

For example, when using SUNION to perform Union operations on two Set, or using SORT to perform sorting operations on List/Set, you should pay close attention.

There are several main ways to avoid problems when using these O (N) commands:

Don't use List as a list, just as a queue

Strictly control the size of Hash, Set and Sorted Set through the mechanism

If possible, put sorting, union, intersection and other operations on the client side to perform

The use of the KEYS command is absolutely prohibited

Instead of traversing all members of the collection type at once, use the commands of the SCAN class for batch, cursor traversal

Redis provides the SCAN command, which allows you to cursor through all the key stored in Redis, avoiding the performance problems caused by using the KEYS command. At the same time, there are commands such as SSCAN/HSCAN/ZSCAN, which are used for cursor traversal of elements in Set/Hash/Sorted Set, respectively. For the use of SCAN commands, please refer to the official document: https://redis.io/commands/scan

Redis provides a Slow Log function that automatically records time-consuming commands. There are two related configuration parameters:

Slowlog-log-slower-than xxxms # commands whose execution time is slower than xxx milliseconds are counted in the length of Slow Log slowlog-max-len xxx # Slow Log, that is, the maximum number of Slow Log records

Using the SLOWLOG GET [number] command, you can output the number bar command that recently entered Slow Log.

Using the SLOWLOG RESET command, you can reset Slow Log

Network-induced delay

Use persistent connections or connection pooling whenever possible to avoid frequent creation and destruction of connections

The batch data operations performed by the client should be done in one interaction using the Pipeline feature. For details, please refer to the Pipelining section of this article.

Latency caused by data persistence

Redis's data persistence work itself brings latency, and a reasonable persistence policy needs to be made according to the security level and performance requirements of the data:

Although the setting of AOF + fsync always can absolutely ensure data security, each operation will trigger a fsync, which will have an obvious impact on the performance of Redis.

AOF + fsync every second is a good compromise, fsync once per second

AOF + fsync never will provide the best performance under the AOF persistence scheme

Using RDB persistence usually provides higher performance than using AOF, but you need to pay attention to the policy configuration of RDB

Each RDB snapshot and AOF Rewrite requires the Redis main process to perform fork operations. The fork operation itself can be time-consuming, depending on the amount of memory consumed by CPU and Redis. Configure RDB snapshots and AOF Rewrite timing reasonably according to specific conditions to avoid delays caused by excessive fork

When Redis forks a child process, it needs to copy the memory paging table to the child process. Take the Redis instance that occupies 24GB memory as an example, you need to copy the data of 24GB / 4kB * 8 = 48MB. On physical machines that use a single Xeon 2.27Ghz, this fork operation takes 216ms.

You can view the time spent on the last fork operation (in microseconds) through the latest_fork_usec field returned by the INFO command

Delay caused by Swap

When Linux moves the memory paging used by Redis to swap space, it will block the Redis process and cause abnormal latency in Redis. Swap usually occurs when there is insufficient physical memory or when some processes are performing a large number of I-stroke O operations, both of which should be avoided as much as possible.

The swap record of the process is saved in the / proc//smaps file, and by looking at this file, you can determine whether the delay in Redis is caused by Swap. If a large Swap size is recorded in this file, the delay is most likely caused by Swap.

Delay caused by data elimination

Redis latency is also caused when a large number of key expires in the same second. The failure time of key should be staggered as far as possible when in use.

Introduction of read-write separation mechanism

The master-slave replication capability of Redis can implement an one-master-multi-slave multi-node architecture, in which the master node receives all write requests and synchronizes the data to multiple slave nodes.

On this basis, we can make the slave node provide read request service with low real-time requirements, so as to reduce the pressure on the master node.

Especially for some statistical tasks that use long-time commands, you can specify that they can be executed on one or more slave nodes to prevent these long-time commands from affecting the response of other requests.

For a specific description of the separation of read and write, see the following sections

Master-slave replication and cluster fragmentation master-slave replication

Redis supports a master-slave replication architecture with one master and multiple slaves. An Master instance handles all write requests, and Master synchronizes writes to all Slave.

With the master-slave replication of Redis, read-write separation and high availability can be achieved:

Real-time requirements are not very high read requests, which can be completed on Slave to improve efficiency. In particular, some periodic statistical tasks may need to execute some long-time-consuming Redis commands, and one or more Slave can be specially planned to serve these statistical tasks.

High availability can be achieved with the help of Redis Sentinel. After Master crash, Redis Sentinel can automatically promote a Slave to Master and continue to provide services.

Enabling master-slave replication is very simple, you only need to configure multiple Redis instances, in the Redis instance as Slave:

Slaveof 192.168.1.1 6379 # specify the IP and port of Master

When Slave starts, a cold start data synchronization is performed from Master, and Master triggers BGSAVE to generate RDB files and push them to Slave for import. After import, Master synchronizes the incremental data to Slave through Redis Protocol. After that, the data between master and slave has been synchronized with Redis Protocol.

Use Sentinel to do automatic failover

Redis's master-slave replication function itself only does data synchronization and does not provide monitoring and automatic failover capabilities. In order to achieve high availability of Redis through the master-slave replication function, we also need to introduce a component: Redis Sentinel

Redis Sentinel is a monitoring component officially developed by Redis, which can monitor the status of Redis instances, automatically discover Slave nodes through Master nodes, select a new Master when a Master node fails, and push new master-slave configurations to all Redis instances.

Redis Sentinel needs to deploy at least 3 instances to form an electoral relationship.

Key configurations:

Sentinel monitor mymaster 127.0.0.1 6379 2 # Master instance IP, port, and the number of votes required for the election sentinel down-after-milliseconds mymaster 60000 # how long no response is considered a Master failure sentinel failover-timeout mymaster 180000 # interval between two failover attempts sentinel parallel-syncs mymaster 1 # if there are multiple Slave, you can specify the number of Slave for data synchronization from the new Master simultaneously through this configuration Prevent all Slave from synchronizing data at the same time, causing the query service to be unavailable

In addition, it should be noted that the automatic failover implemented by Redis Sentinel is not completed on the same IP and port, that is, the IP and port of the new Master provided by the automatic failover are different from the previous Master, so to achieve HA, the client must support Sentinel and be able to interact with Sentinel to obtain the information of the new Master.

Cluster fragmentation

Why do you want to do cluster fragmentation:

The amount of data stored in Redis is so large that the physical memory of a host can no longer be accommodated.

Redis has a large number of write requests concurrently, and a Redis instance cannot be hosted.

When the above two problems arise, it is necessary to fragment the Redis.

There are many kinds of sharding schemes for Redis. For example, many Redis clients implement sharding function on their own, and there are also Redis sharding schemes implemented in a proxy way such as Twemproxy. However, the preferred solution should also be the Redis Cluster fragmentation scheme officially launched by Redis in version 3.0.

This article will not introduce the specific installation and deployment details of Redis Cluster, focusing on the benefits and disadvantages of Redis Cluster.

The ability of Redis Cluster

Ability to automatically distribute data across multiple nodes

When the accessed key is not on the current shard, the request can be automatically forwarded to the correct shard

Services can still be provided when some nodes in the cluster fail.

The third point is based on master-slave replication. Each data fragment of Redis Cluster adopts the structure of master-slave replication, and the principle is completely the same as the master-slave replication described above. The only difference is that the extra component of Redis Sentinel is omitted, and Redis Cluster is responsible for node monitoring and automatic failover within the fragment.

Redis Cluster slicing principle

A total of 16384 hash slot,Redis in Redis Cluster will calculate the CRC16 of each key, and compare the result with 16384 to determine which hash slot the key is stored in. At the same time, you need to specify the number of Slot for each data shard in the Redis Cluster. The allocation of Slot can be reassigned at any point in time.

When the client reads and writes to the key, it can connect to any shard in the Cluster. If the operating key is not within the Slot range for which the shard is responsible, the Redis Cluster will automatically redirect the request to the correct shard.

Hash tags

In the basic slicing principle, Redis also supports the hash tags function, and key in the format clearly required by hash tags will ensure that it enters the same Slot. For example: {uiv} user:1000 and {uiv} user:1001 have the same hash tag {uiv} and will be saved in the same Slot.

When using Redis Cluster, the key involved in the pipelining, transaction, and LUA Script functions must be on the same data shard, otherwise an error will be returned. If you want to use the above functionality in Redis Cluster, you must use hash tags to ensure that all key operated in a pipeline or transaction are in the same Slot.

Some clients (such as Redisson) implement clustered pipelining operations, which can automatically group commands in a pipeline according to the shards in which the key is located, and send them to different shards for execution. However, Redis does not support cross-shard transactions, transactions and LUA Script must still follow the rules of all key on one shard.

Master-slave replication vs cluster fragmentation

When designing software architecture, how to choose between master-slave replication and cluster fragmentation?

In all respects, Redis Cluster is superior to master-slave replication.

Redis Cluster can solve the problem of excessive data on a single node.

Redis Cluster can solve the problem of excessive access pressure on a single node.

Redis Cluster includes the ability of master-slave replication.

Does that mean that Redis Cluster is always a better choice than master-slave replication?

It's not.

Software architecture is never the more complex the better. Complex architecture not only brings significant benefits, but also brings corresponding disadvantages.

When using Redis Cluster, the number of Redis instances that need to be maintained is doubled, the number of hosts that need to be monitored increases accordingly, and the complexity of data backup / persistence increases. At the same time, in the addition and subtraction of fragments, the reshard operation is also needed, which is much more complex than adding a Slave in the master-slave mode.

When the client uses the connection pool, it needs to maintain a connection pool for each data fragment, and the number of connections that the client needs to maintain at the same time increases exponentially, which increases the consumption of the client itself and operating system resources.

You may need to view Slow Log and Swap logs on multiple shards to locate performance issues.

The cost of using transactions and LUA Script has increased. There are strict restrictions on the use of transaction and LUA Script features in Redis Cluster, and the key operated in the transaction and Script must be located on the same shard, which makes it necessary to make additional planning and specification requirements for the key involved in the corresponding scenario. If a large number of transactions and Script are involved in the application scenario, it will be difficult to divide the data equally among multiple data fragments on the premise of ensuring the normal operation of these two functions.

Therefore, when making a choice between master-slave replication and cluster fragmentation, we should comprehensively consider the functional characteristics of the application software, the level of data and access, and future development planning, and use Redis Cluster only when it is really necessary to introduce data fragmentation.

Here are some suggestions:

How large is the data that needs to be stored in Redis? How big is it likely to develop in the next 2 years? Does all this data need to be preserved for a long time? Can the LRU algorithm be used to eliminate non-hot data? Considering the above factors, the physical memory needed for Redis is evaluated.

How large is the host physical memory used to deploy Redis? Is it sufficient to compare the memory requirements assessment in (1)?

How much pressure will Redis face on concurrent writes? When not using pipelining, the write performance of Redis can exceed 100000 / s (for more benchmark, please refer to https://redis.io/topics/benchmarks)

Do you use pipelining and transaction functionality when using Redis? Do you use a lot of scenes?

Considering the above considerations, if the available physical memory of a single host is sufficient to support the capacity requirements of Redis, and the concurrent write pressure faced by Redis is still far from the Benchmark value, it is recommended to adopt master-slave replication architecture, which can save a lot of unnecessary trouble. At the same time, if the application uses a lot of pipelining and transactions, it is also recommended to choose the master-slave replication architecture as much as possible, which can reduce the complexity of design and development.

There are many Java clients for Redis, and three are officially recommended: Jedis, Redisson and lettuce.

Here is a comparative introduction of Jedis and Redisson

Jedis:

Support for connection pooling

Support for pipelining, transaction, LUA Scripting, Redis Sentinel, Redis Cluster

The document is poor (really bad, almost no. )

Redisson:

Based on Netty implementation, using non-blocking IO, high performance

Support for asynchronous requests

Support for connection pooling

Transactions are not supported. It is officially recommended to replace transactions with LUA Scripting.

Support for using pipelining under Redis Cluster architecture

It supports read-write separation and read load balancing. It can be used in both master-slave replication and Redis Cluster architecture.

Built-in Tomcat Session Manager to provide session sharing for Tomcat in 6-7-8

Can be integrated with Spring Session to achieve session sharing based on Redis

Rich in documents, with Chinese documents

For the choice of Jedis and Redisson, we should also follow the above-mentioned principle. Although Jedis has various shortcomings compared with Redisson, we should also choose Redisson when we need to use the advanced features of Redisson to avoid unnecessary increase in program complexity.

Jedis:

Github: https://github.com/xetorthio/jedis

Documentation: https://github.com/xetorthio/jedis/wiki

Redisson:

Github: https://github.com/redisson/redisson

Documentation: https://github.com/redisson/redisson/wiki

Thank you for your reading, the above is the content of "Redis Advanced Features and performance tuning methods". After the study of this article, I believe you have a deeper understanding of the problem of Redis advanced features and performance tuning methods, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.