Shulou(Shulou.com)05/31 Report--

This article introduces you how to migrate Food Feed business from Redis to Cassandra, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Here's how Zomato migrates its Food Feed business from Redis to Cassandra.

Food Feed is an integral part of Zomato's social scene because it allows our users to participate and keep up with friends' restaurant reviews and pictures, and even get discounts and discounts from restaurants. At first, we chose Redis as the storage engine for the message Feed stream, because it was the best choice in the user scenario at that time. But with the development of the business, we need higher availability and load support, and Redis can not meet this demand very well. Although we can avoid system disruption by losing some data, this is not what we want to do. In order to ensure the high availability of our system, we had to abandon Redis and choose Cassandra as its replacement.

Cassandra is a good fit for this use case because it is distributed, high availability, and so on. And Cassandra can also be used to store time series data-this is actually our Feed stream. We do have some experience with Cassandra before making this big change, but it's certainly not enough for something as important as Feed. We have to figure out how to make a smooth transition from Redis to Cassandra and run Feed as efficiently as on Redis without downtime.

We began to spend time on Cassandra, exploring its configuration in depth in the first two weeks and adjusting it to meet our requirements. Next, before finalizing the architecture of Feed, we identify two situations:

Feed stream information is generally only used for reading and basically does not modify. When using Redis, we can read hundreds of keys at the same time without worrying about read latency, but for Cassandra, connection latency can be a fairly important part of the read request process.

Schema needs to be flexible enough to allow new types of data in Feed in the future. Given that we are constantly iterating and committed to enriching the product experience, it is almost inevitable to add elements and functionality to Feed.

We spent a few days collecting our project's data patterns and various user cases, and then started using two data centers, each with three nodes. We migrated about 60 million records from Redis to Cassandra to test its performance. Since it is a test phase, we only cut part of the traffic into Cassandra and prepare two versions of the code, which are written to Cassandra and Redis respectively. The architecture diagram is as follows

We monitor the system for latency and other problems, and surprisingly, we encounter bottlenecks in write throughput. We know that Cassandra is very capable of writing, but we can't achieve the write throughput we read in a variety of blog posts and articles. We know what's wrong, but we don't know what it is. The best result we got from the three nodes was 1500 writes per second, which was not enough to meet the online needs and we had to roll back and reevaluate within a few hours.

After several days of investigation, we realized that the problem was not Cassandra, but Elastic Block Store (EBS). EBS is a network drive installed on each EC2 instance with 10 Gigabits of shared bandwidth and network traffic. This bandwidth becomes our bottleneck when shared among all users on a single EC2 instance. To meet this requirement, we moved data from network-based EBS storage to disk storage in the same EC2 instance. Then we deploy the new Food Feed supported by Cassandra on each server one by one, so that we can control the throughput. Happily, it was a success this time.

Then we started migrating data from our production Redis server (it took us 14 hours to migrate everything) without any failure or extra load during the migration. This is the power of Redis and Cassandra. Today, our Food Feed runs entirely on Cassandra, and we did this without downtime. The new architecture is as follows:

All in all, through the above project, we have learned the following points:

Avoid reading data during writes. the read throughput remains roughly the same, while the write scale is proportional to the number of nodes

Avoid data deletion. Delete means compaction, and when it runs, the node's resources are occupied

Delay is a problem. Compared with Redis, Cassandra has a high connection latency, which is about the 10x-15x of Redis.

On how to migrate Food Feed business from Redis to Cassandra to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.