Shulou(Shulou.com)06/01 Report--

This article shows you how to develop a high-performance redis cluster proxy, the content is concise and easy to understand, can definitely brighten your eyes, through the detailed introduction of this article, I hope you can get something.

Background

Introduction to redis cluster

Redis cluster is an official cluster scheme provided by redis, which is designed with a decentralized architecture. Nodes exchange states with each other through gossip protocol, and redis cluster uses data slicing to build a cluster. The cluster has 16384 hash slots built in, and each key belongs to one of the 16384 hash slots. The hash value is calculated by the crc16 algorithm, and then the hash slot to which each key belongs can be obtained. Redis cluster supports dynamic addition of new nodes, dynamic migration of slot, automatic failover, and so on.

The architecture of Redis cluster requires that the client needs to establish a connection with each node in the redis cluster directly, and when there are events such as new node joining, node downtime failover, slot migration and so on, the client needs to be able to update the local slot mapping table through redis cluster protocol and can handle ASK/MOVE semantics. Therefore, we generally call the client that implements the redis cluster protocol smart redis client.

Redis cluster can build a cluster with a maximum of more than 100 master nodes (after which the gossip protocol costs too much and may cause cluster instability). According to the 10G capacity of a single node (excessive memory of a single instance may lead to performance degradation), a single cluster can support a capacity of about 1T at most.

Ask questions

Redis cluster has many advantages (for example, it can build large-capacity clusters, good performance, flexible expansion and expansion), but when some project projects expect to migrate from redis to redis cluster, the client faces a lot of transformation work, which brings a lot of testing work and new risks, which is undoubtedly a huge cost for some stable online projects.

Need

In order to migrate the business to redis cluster more conveniently, it is most expected that the API of the client SDK is fully compatible with the RedisTemplate provided by redis/redis-cluster,spring. However, for projects that do not use SpringRedisTemplate, many client-side implementations of redis and redis-cluster access to API are inconsistent (such as the popular Jedis in Java), which virtually increases the workload and complexity of the migration work. Redis cluster proxy is a good choice at this time, with proxy. You can operate redis cluster like a single instance redis, and the client program does not need to make any changes.

Of course, adding a layer of proxy will inevitably lead to a decline in performance to a certain extent, but proxy, as a stateless service, can theoretically scale horizontally, and due to the existence of the proxy layer, it can reduce the number of connections to the back-end redis server, and even improve the overall throughput of the redis cluster in some limit scenarios. In addition, based on proxy, we can do a lot of extra things:

For example, sharding logic can be done in the proxy layer, so that when the redis cluster of a single cluster does not meet the requirements (memory / QPS), multiple redis cluster clusters can be accessed transparently through the proxy layer.

For example, you can do double-write logic in the proxy layer, so that when migrating or splitting cache-type redis, you do not need to use tools such as redis-migrate-tool for full migration, but only need to double-write on demand to complete the migration.

In addition, because proxy implements the redis protocol, it can use other storage media to implement redis-related commands in the proxy layer, which can be simulated as redis external services. A typical scenario is hot and cold separate storage.

Work and energy

For the above reasons and requirements, we have developed a middleware such as camellia-redis-proxy based on netty, which supports the following features:

Support setting password

Support proxy to normal redis, also support proxy to redis cluster

Support for configuring custom sharding logic (can be proxied to multiple redis/redis-cluster clusters)

Support the configuration of custom dual write logic (the server will recognize the read and write attributes of the command, and after configuring double write, the write command will be sent to multiple backends at the same time)

Support external plug-ins so that the protocol parsing module can be reused (currently includes the camellia-redis-proxy-hbase plug-in, which enables hot and cold storage of zset commands)

Support for online configuration changes (need to introduce camellia-dashboard)

Support multiple business logic to share a set of proxy clusters, such as: a service configuration forwarding rule 1 Magi B service configuration forwarding rule 2 (you need to set the service type through the client command when establishing a redis connection)

A spring-boot-starter,3 line code is provided to quickly build a proxy cluster.

Of course, there is another scenario in which you can register the proxy node with a registry such as zk/Eureka/Consul, and the client pulls and listens to the list of proxy, and then accesses each proxy in the same way as a single node redis. Take Jedis as an example. Just replace JedisPool with RedisProxyJedisPool that encapsulates registration discovery logic, and you can use proxy just like accessing normal redis. In this case, the deployment architecture diagram is as follows:

Application scenario

You need to migrate from redis to redis-cluster, but the client code is not easy to modify

The client is directly connected to redis-cluster, resulting in too many connections to the cluster server, resulting in server performance degradation

The capacity of a single redis/redis-cluster cluster / QPS does not meet the business requirements, so use the slicing function of camellia-redis-proxy

Cache class redis/redis-cluster cluster split migration, using the double write function of camellia-redis-proxy

Using double write function for disaster recovery of redis/redis-cluster

Some business scenarios with mixed use of sharding and double-write functions

Develop custom plug-ins based on the plug-in function of camellia-redis-proxy

Redis cluster is an officially recommended cluster solution. It is against this background that more and more projects have been or are being migrated to redis cluster,camellia-redis-proxy. In particular, if you are a Java developer, camellia also provides a solution such as CamelliaRedisTemplate. CamelliaRedisTemplate has the same API as ordinary Jedis, provides features such as mget/mset/pipeline that are not supported by native JedisCluster, and provides sharding / one double write and other features consistent with camellia-redis-proxy.

The above content is how to develop a high-performance redis cluster proxy, have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.