Shulou(Shulou.com)06/01 Report--

In Redis Enterprise version 5. 0, his team introduced support for the open source (OSS) cluster API, which allows Redis Enterprise clusters to scale linearly by adding shards and nodes. In this paper, the first linear expansion benchmark is given and this infinite linear scalability is demonstrated.

What is a linear extension?

According to Wikipedia, an extensible database is a database that can be upgraded to handle more transactions by adding new processors and storage, and can be easily upgraded without affecting its use. The database can scale out (by adding nodes to the cluster, rebalancing and allocating the database) or upwards (by adding shards to the database without adding nodes to the cluster). Redis Enterprise is optimized to be extended simply by binding it to disk.

Linear expansion means that the database is extended sequentially by adding resources related to throughput (in Redis terms, "resources" refer to nodes and shards). True linear scalability means that the amount of resources increases in the same proportion as database throughput and in a deterministic manner. For example, increasing cluster resources by 50% will result in a 50% increase in throughput.

If the database can be scaled linearly, you can minimize operational overhead and scale the business without worrying about size restrictions or performance bottlenecks in the database. However, there is usually expansion-related overhead, which means that when the capacity increases N, the database throughput usually increases by a number less (or much less) than N.

The database provides:

Sub-linear expansion-when the relative capacity is less than the number of resources.

Linear expansion-when the relative capacity is equal to the relative number of resources added.

Superlinear expansion-when the relative capacity increases with the increase of resources.

A brief introduction to Redis Enterprise

A cluster in Redis Enterprise terminology is a group of cloud instances, virtual machine / container nodes or bare metal servers that allow the creation of any number of Redis databases (databases in Redis Enterprise terminology are entities that manage the entire dataset across multiple Redis shards / instances, do not confuse this with databases in each Redis instance, you can use the Redis SELECT command to segment in the key space) in the entire collection shared memory pool. The cluster has a symmetrical no-sharing architecture, and the data path is completely separated from the control and management path, which includes the following main components:

Redis Shards- has a master or slave Redis instance

Zero-latency Proxy- is built on a multithreaded, stateless architecture that hides the complexity of the cluster, enhances security (SSL, authentication, DDoS protection) and improves performance (no TCP connection management)

Cluster Manager (Control and Management path)-built by a group of distributed processes on the cluster node, responsible for cluster configuration, response to requests, resource management, etc., and acts as a resource monitor and completely enables Redis shards to manage the health of other shards in the cluster or perform failover.

You can create a database in a Redis Enterprise cluster in any of the following configurations:

Build a test environment

Considering all factors, his team decided to build a test environment on AWS to verify whether Redis Enterprise could scale infinitely in a linear manner. The final test uses EC2 m4.16xlarge instances (64 cores, 256GB RAM) for cluster nodes and c4.8xlarge instances (36 cores, 60GB RAM) for running memtier_benchmark, an open source multithreaded load generation tool.

It is necessary to use multiple memtier_benchmark instances, because in many cases, a single Redis Enterprise node can handle more traffic than a single memtier_benchmark instance, which avoids the bandwidth of a single NIC network and data transmission limits per second, and can gradually increase the traffic load (instance by instance).

This is the final setting for his team:

6x m4.16xlarge instance of Redis Enterprise cluster node:

An 8x c4.8xlarge instance running memtier_benchmark:

Over the past few months, his team has conducted several tests, including k-node Redis Enterprise clusters and other benchmark n-shard databases, as shown below:

Table 1:Redis Enterprise linear extension while providing sub-millisecond performance.

Figure 1: cluster throughput (@ 1 Ms delay)

This indicates that as throughput increases, the linear scalability of its nodes increases, and Redis Enterprise can consistently provide sub-millisecond latency across all data sizes and workloads.

Create and adjust cluster database

His team used Redis Enterprise API to create a 192-shard cluster Redis database with the following parameters:

We adjust the proxy on each node to cope with the expected load by setting the number of proxy threads to 24:

Of course, other database vendors have also released a lot of benchmark data about their scalability. In fact, the results show that Redis Enterprise outperforms its NoSQL counterparts. The following chart shows benchmark results from other NoSQL vendors, comparing Apache Cassandra,HBase,MongoDB with Couchbase.

As can be seen from the chart, all vendors provide sub-linear scalability. For example, if you analyze the throughput of Cassandra by node, 1 node of Cassandra can handle about 18700 ops / s, and so on, 32 nodes should be able to handle about 600000 ops / s. But in reality, it can only handle about 330000ops per second and has only 55% of the ability to scale the database linearly.

With its latest benchmark, Redis Enterprise has proved that it can handle millions of operations per second, even in the most basic configuration. As shown in the following figure, Redis Enterprise outperforms other databases and provides superlinear scaling without compromising performance!

Table 3:Redis Enterprise-Best and actual Throughput of Node

This new benchmark demonstrates that Redis Enterprises can achieve true linear scalability while providing strong performance through efficient resource utilization, but the benchmark does not mention whether the other participating database versions are enterprise or open source, so the data remains to be considered.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.