Shulou(Shulou.com)05/31 Report--

Most people do not understand the knowledge points of this "Redis cluster case analysis" article, so the editor summarizes the following content, detailed content, clear steps, and has a certain reference value. I hope you can gain something after reading this article. Let's take a look at this "Redis cluster case analysis" article.

1. Why K8s

1. Resource isolation

The current Redis Cluster is deployed on physical machine clusters. In order to improve resource utilization and save costs, Redis clusters with multiple lines of business are mixed. Because there is no CPU resource isolation, the high CPU utilization of a Redis node often causes the nodes of other Redis clusters to compete for CPU resources, resulting in delay jitter. Because of the mixed distribution of different clusters, this kind of problem is difficult to locate quickly and affect the efficiency of operation and maintenance. CPU request and CPU limit can be specified for K8s containerized deployment, which can improve resource utilization while avoiding resource competition.

2. Automated deployment

Currently, the process of deploying Redis Cluster on physical machines is very tedious. You need to check the meta-information database to find the machines with spare resources, manually modify many configuration files and deploy nodes one by one, and finally use redis_trib tool to create clusters. Initialization of new clusters often takes an hour or two.

K8s deploys Redis clusters through StatefulSet and uses configmap to manage configuration files. It takes only a few minutes for new clusters to be deployed, which greatly improves the efficiency of operation and maintenance.

2. How K8s

The client accesses through the VIP of LVS and forwards the service request to the Redis Cluster cluster through Redis Proxy. Here we introduce Redis Proxy to forward the request.

1. Redis Cluster deployment method

Redis is deployed as StatefulSet. As a stateful service, it is most reasonable to choose StatefulSet, which can persist the RDB/AOF of nodes to distributed storage. When the node restarts and drifts to another machine, you can get the original RDB/AOF through the mounted PVC (PersistentVolumeClaim) to synchronize the data.

The persistent storage PV (PersistentVolume) we chose is Ceph Block Service. The read and write performance of Ceph is lower than that of the local disk, which will bring the read and write delay of 100~200ms. However, because the writes of Redis's RDB/AOF are asynchronous, the read and write latency caused by distributed storage has no impact on the service.

2. Proxy selection

There are many open source Redis Proxy. Common open source Redis Proxy is as follows:

We want to be able to continue to use Redis Cluster to manage Redis clusters, so Codis and Twemproxy are no longer considered. Redis-cluster-proxy is a Proxy that supports Redis Cluster protocol officially launched by Redis in version 6.0, but there is no stable version and it cannot be applied on a large scale for the time being.

The only alternatives are Cerberus and Predixy. We tested the performance of Cerberus and Predixy on K8s environment, and the results are as follows:

Test environment

Testing tool: redis-benchmark

Proxy CPU: 2 core

Client CPU: 2 core

Redis Cluster: 3 master nodes, 1 CPU per node

Test result

Under the same workload and configuration, the highest QPS of Predixy is better than Cerberus, and the delay is similar. Generally speaking, the performance of Predixy is 33% / 60% higher than that of Cerberus, and the larger the key/value of the data, the more obvious the advantage of Predixy, so we chose Predixy in the end.

In order to adapt to the business and K8s environment, we have made a lot of changes to Predixy and added many new features before launch, such as dynamically switching back-end Redis Cluster, blacklist and whitelist, abnormal operation audit and so on.

3. Proxy deployment method

As a deployment deployment, Proxy is stateless and lightweight, and provides services through LB, so it is easy to expand and scale up dynamically. At the same time, we have developed the function of dynamically switching back-end Redis Cluster for Proxy, which can add and switch Redis Cluster online.

4. Proxy automatic expansion and reduction mode

We use the native HPA (Horizontal Pod Autoscaler) of K8s to achieve the dynamic expansion and reduction of Proxy. When the aPCge CPU utilization of all pod in Proxy exceeds a certain threshold, capacity expansion will be triggered automatically, and HPA will increase the number of replica of Proxy by 1, and then LVS will detect the new Proxy pod and cut part of the traffic. If the CPU utilization still exceeds the specified threshold after the expansion, the expansion logic will continue to be triggered. However, within 5 minutes of successful expansion, no matter how low the CPU utilization is, the scaling down logic will not be triggered, thus avoiding the impact of frequent expansion on the stability of the cluster.

HPA can configure the minimum (MINPODS) and maximum (MAXPODS) number of pod of a cluster. No matter how low the cluster load is, it will not scale down to the number of pods below MINPODS. It is suggested that customers can decide the values of MINPODS and MAXPODS according to their actual business situation.

III. Why Proxy

1. Restart of Redis pod can cause changes in IP

For Redis clients using Redis Cluster, you need to configure part of the IP and Port of the cluster to find the entry of Redis Cluster when the client is restarted. For the Redis node deployed in the physical machine cluster, the IP and Port can remain unchanged even if the instance is restarted or the machine is restarted, and the client can still find the Redis Cluster topology. However, the restart of Redis Cluster,pod deployed on K8s does not guarantee that the IP will remain unchanged (even if it is restarted on the original K8s node), so that when the client is restarted, the entry to Redis Cluster may not be found.

By adding Proxy between the client and Redis Cluster, the information of Redis Cluster is shielded from the client. Proxy can dynamically perceive the topology changes of Redis Cluster. The client only needs to use the IP:Port of LVS as an entry and forward the request to Proxy, that is, you can use Redis Cluster cluster like a stand-alone Redis without the need for Redis smart client.

2. Redis processing has a high connection load.

Prior to version 6. 0, Redis handled most tasks in a single thread. When the connections of Redis nodes are high, Redis needs to consume a lot of CPU resources to process these connections, resulting in an increase in delay. With Proxy, a large number of connections are on Proxy, while there are only a few connections between Proxy and Redis instances, which reduces the burden of Redis and avoids the increase of Redis latency caused by the increase of connections.

3. Cluster migration and handover requires application restart

In the process of use, with the growth of business, the amount of data of Redis cluster will continue to increase. When the amount of data of each node is too high, the time of BGSAVE will be greatly extended and the availability of the cluster will be reduced. At the same time, the increase of QPS will also lead to an increase in the utilization of CPU in each node. All these need to be solved by increasing the capacity of clusters. At present, the scale-out ability of Redis Cluster is not very good, and the efficiency of the native slots migration scheme is very low. After adding a new node, some clients, such as Lettuce, will not recognize the new node because of the security mechanism. In addition, the migration time is completely unpredictable, and the problems encountered in the migration process cannot be reversed.

The current expansion scheme for physical machine clusters is:

Create a new cluster on demand

Use synchronization tools to synchronize data from the old cluster to the new cluster

After confirming that the data is correct, communicate with the business, restart the service and switch to the new cluster.

The whole process is tedious and risky, and the business restart service is needed.

With the Proxy layer, you can mask the back-end creation, synchronization, and switching clusters from the client. After the synchronization between the new cluster and the old cluster is completed, send a command to Proxy to change the connection to the new cluster, and the cluster can be scaled and scaled without being aware of the client.

4. Data security risk

Redis implements authentication operation through AUTH. The client is directly connected to the Redis, and the password still needs to be saved on the client. With Proxy, the client only needs to access Proxy through the password of Proxy and does not need to know the password of Redis. Proxy also restricts FLUSHDB, CONFIG SET and other operations, avoids customers' misoperation of emptying data or modifying Redis configuration, and greatly improves the security of the system.

At the same time, Redis does not provide audit functionality. We have added log preservation for high-risk operations on Proxy to provide audit capabilities without affecting overall performance.

IV. Problems brought by Proxy

1. The delay caused by one more hop

Proxy between the client and the Redis instance, the client accesses the Redis data first and then the Redis node. If there is one more hop, the delay will increase. It has been tested that one more hop increases the latency of the 0.2~0.3ms, but this is usually acceptable for the business.

2. Pod drift causes IP change.

Proxy is deployed through deployment on K8s, and there will also be the problem of IP changes caused by node restart. Our LB solution of K8s can sense the IP changes of Proxy and dynamically cut the traffic of LVS to the rebooted Proxy.

3. Delay caused by LVS

LVS also brings latency, such as the test in the following table. For different data length get/set operations, the delay introduced by LVS is less than 0.1ms.

Fifth, the benefits of K8s

1. Easy deployment

Through the operation and maintenance platform to call K8s API to deploy the cluster, the efficiency of operation and maintenance is greatly improved.

2. Solve port management problems

Currently, Xiaomi's deployment of Redis instances on physical machines is distinguished by ports, and offline ports cannot be reused, which means that each Redis instance in the company has a unique port number. At present, more than 40000 ports have been used. At the current rate of business development, port resources will be exhausted within two years. Through K8s deployment, the K8s pod corresponding to each Redis instance has its own IP, and there is no port exhaustion and complex management problems.

3. Lower the threshold for customers to use

For applications, only a stand-alone version of the non-smart client is needed to connect to VIP, which lowers the threshold and avoids tedious and complicated parameter settings. At the same time, because the VIP and ports are fixed, the application no longer needs to manage the topology of the Redis Cluster itself.

4. Improve client performance

Using a non-smart client can also reduce the load on the client, because the smart client needs to hash the key on the client to determine which Redis node to send the request to, which consumes the CPU resources of the client machine when the QPS is high. Of course, in order to reduce the difficulty of client application migration, we let Proxy also support the smart client protocol.

5. Dynamic upgrade and expansion

Proxy supports the dynamic addition of switching Redis Cluster, so that the cluster upgrade and expansion switching process of Redis Cluster can be completely unaware of the business end. For example, the business side uses a 30-node Redis Cluster cluster, due to the increase in business volume, the amount of data and QPS are growing rapidly, so it is necessary to double the size of the cluster. If you expand the capacity on the original physical machine, the following process is required:

Coordinate resources to deploy a new cluster of 60 nodes

Manually configure the migration tool to migrate the data from the current cluster to the new cluster

After verifying the data, notify the business side to modify the Redis Cluster connection pool topology and restart the service.

Although Redis Cluster supports online capacity expansion, slots migration during the expansion process will affect online business, and the migration time is uncontrollable, so this method is rarely used at this stage, and will only be used occasionally when resources are seriously insufficient.

Under the new K8s architecture, the migration process is as follows:

Create a new cluster of 60 nodes with one click through the API interface

Also create a cluster synchronization tool with one click through the API interface to migrate data to the new cluster

After verifying that the data is correct, send a command to Proxy to add the new cluster information and complete the switch.

The whole process is completely unaware of the business side.

Cluster upgrade is also convenient: if the business can accept a certain delay burr, it can be achieved through StatefulSet rolling upgrade at low peak; if the business requires delay, it can be achieved by creating a new cluster to migrate data.

6. Improve service stability and resource utilization

Through the resource isolation capability of K8s, it can be mixed with other different types of applications, which can not only improve the resource utilization, but also ensure the service stability.

VI. Problems encountered

1. Data loss due to Pod restart

When the pod of K8s restarts when it encounters a problem, because the restart speed is too fast, the pod will be restarted before the Redis Cluster cluster discovers and cuts the master. If the Redis on pod is slave, it won't make any difference. However, if Redis is master and there is no AOF, the original memory data will be emptied after restart, and Redis will reload the previously stored RDB file, but the RDB file is not real-time data. After that, slave will also synchronize its own data into the data mirror in the previous RDB file, resulting in some data loss.

StatefulSet is a stateful service, and the deployed pod name is in a fixed format (StatefulSet name + number). When we initialize the Redis Cluster, we set the adjacent numbered pod as the master-slave relationship. When the pod is restarted, its slave is determined by the pod name, and a cluster failover command is sent to the slave node before restarting the pod, forcing the living slave node to cut off the master. In this way, after restarting, the node will automatically join the cluster as a slave node.

LVS mapping delay

Proxy's pod is load balanced through LVS. There is a certain delay for LVS to map the backend IP:Port to take effect, and the sudden offline of the Proxy node will result in partial connection loss. To reduce the impact of Proxy OPS on business, we have added the following options to the deployment template of Proxy:

Lifecycle: preStop: exec: command:-sleep-"171"

For normal Proxy pod offline, such as cluster downsizing, rolling update of Proxy version, and other pod offline controlled by K8s, a message will be sent to LVS and wait 171s before pod is offline, which is enough time for LVS to gradually cut the traffic of this pod to other pod without business perception.

2. K8s StatefulSet cannot meet the requirements of Redis Cluster deployment.

The native StatefulSet of K8s does not fully meet the requirements of Redis Cluster deployment:

1) Redis Cluster does not allow nodes with the same master / slave relationship to be deployed on the same machine. It is easy to understand that if the machine goes down, it will cause the data shard to be unavailable.

2) Redis Cluster does not allow more than half of the master nodes in the cluster to fail, because if more than half of the master nodes fail, there will not be enough nodes to vote to meet the requirements of the gossip protocol. Because the master / slave of Redis Cluster can be switched at any time, we can't avoid the situation that all nodes on the same machine are master nodes, so we cannot allow nodes in the cluster with more than 1 server 4 to be deployed on the same machine.

To meet the above requirements, native StatefulSet can use the anti-affinity feature to ensure that the same cluster deploys only one node on the same machine, but the machine utilization is very low.

Therefore, we have developed a StatefulSet-based CRD:RedisStatefulSet that uses a variety of strategies to deploy Redis nodes. At the same time, some Redis management functions have been added to RedisStatefulSet. We will continue to discuss these in detail in other articles.

VII. Summary

At present, dozens of Redis clusters with multiple businesses within the group have been deployed to K8s and run for more than half a year. Thanks to the rapid deployment and fault migration capability of K8s, the operation and maintenance workload of these clusters is much lower than that of the Redis cluster on the physical machine, and the stability has been fully verified.

We have also encountered a lot of problems in the process of operation and maintenance, and many of the functions mentioned in the article are extracted according to the actual needs. At present, there are still many problems to be solved step by step in the follow-up, in order to further improve the resource utilization and service quality.

1. Mixed Vs. Independent deployment

Redis instances of physical machines are deployed independently, and all Redis instances are deployed on a single physical machine, which is convenient for management, but the resource utilization is not high. The Redis instance uses CPU, memory, and network IO, but the storage space is basically wasted. When an Redis instance is deployed on K8s, other services of any type may also be deployed on the machine. Although this can improve the utilization of the machine, it is unacceptable to be expelled because of insufficient machine memory for services such as Redis, which require high availability and latency. This requires the operation and maintenance staff to monitor the machine memory of all deployed Redis instances, and cut off the master and migration nodes once the memory is insufficient, but this increases the workload of the operation and maintenance staff.

At the same time, if the other services of the mixed part are applications with high network throughput, it may also affect Redis services. Although the anti-affinity function of K8s can selectively deploy Redis instances to machines that do not have such applications, this situation can not be avoided when machine resources are tight.

2. Redis Cluster management

Redis Cluster is a P2P cluster architecture without central nodes, which relies on gossip protocol to propagate and automatically repair the state of the cluster. Node downline and network problems may lead to problems in the state of some nodes in Redis Cluster, such as nodes with failed or handshake status in the cluster topology, or even brain cleavage. For this abnormal state, we can add more functions on Redis CRD to solve it step by step and further improve the efficiency of operation and maintenance.

3. Audit and security

Redis itself only provides Auth password authentication protection function, there is no rights management, the security is poor. Through Proxy, we can distinguish the type of client by password. Administrators and ordinary users log in with different passwords, and the operation permissions that can be performed are also different, so that we can achieve functions such as rights management and operation audit.

4. Support multiple Redis Cluster

Due to the limitation of gossip protocol, the scale-out ability of a single Redis Cluster is limited. When the size of the cluster is 300nodes, the efficiency of this kind of topology change is obviously reduced. At the same time, because the capacity of a single Redis instance should not be too high, it is difficult for a single Redis Cluster to support data scale above TB. Through Proxy, we can logically fragment the key, so that a single Proxy can be connected to multiple Redis Cluster. From the client's point of view, it is equivalent to connecting a Redis cluster that can support the scale of big data.

The above is the content of this article on "Redis Cluster instance Analysis". I believe we all have a certain understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.