Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the challenges and solutions in the process of Redis migration synchronization. Many people may not know much about it. In order to let you know more, Xiaobian summarizes the following contents for you. I hope you can gain something according to this article.

With the rapid development of cloud computing for more than ten years, as the latest stage visible at present, multi-cloud is making great strides forward. The cloud migration caused by the cloudy trend is also gradually normalized. Therefore, caching Redis has become standard for improving data access speed in high-concurrency scenarios.

Not only is the data migration between clouds, but at present, large systems rely heavily on cache, which causes most enterprises to face slow access speed when reading and writing a large number of concurrent data, high database pressure, and cache breakdown and avalanche risks caused by cache data failure. Among them, Redis plays a role in reducing database pressure and improving data access speed.

The following figure shows the process of solving the problem of slow access speed of a website business and introducing cache Redis to improve access speed:

However, there are many challenges in the Redis migration synchronization process:

Rdb versions do not consistently cause source redis dump files not to load in target redis

Redis master-slave replication scheme failure caused by inconsistent data nodes

The need for quick cuts should be met

idempotent requirements for idempotent commands

This is why it came into being._ RedisSyncer is Jingdong Cloud's product for data synchronization between redis, supporting data synchronization across versions and heterogeneous clusters._ It simulates the replication protocol of redis, performs command conversion when rdb versions are inconsistent, to achieve cross-rdb version migration, and completes the idempotent conversion of INCR, INCRBY, DECR, DECRBY and other commands by caching value. Finally, it can adapt to various scenarios such as cloud and cloud, native and hosting, and can quickly and flexibly meet the needs of users for synchronization, migration and expansion.

RedisSyncer Case Study

In order to ensure financial data security and comply with financial data standards, a financial institution needs to migrate the original "redis cluster" to the upredis cluster developed by the financial institution itself, and ensure smooth transition of business systems.

Project Challenge

version discrepancy

downmigration

The number of cluster nodes is inconsistent

Fast backoff demand

Final customer revenue

256GB+ data migration in 15 minutes

20 minutes to complete system cutover

The migration process is "silky"

RedisSyncer Operation

necessary environment

docker

docker-compose

Docker-compose deployment service

Plain Text

git clone https://github.com/TraceNature/redissyncer.gitcd redissyncerdocker-compose up -d

Download and configure cli client

wget https://github.com/TraceNature/redissyncercli/releases/download/v0.1.0/redissyncer-cli-0.1.0-linux-amd64.tar.gz.config.yamlsyncserver: http://10.0.1.20:8080token: 379F5E2BD55A4608B6A7557F0583CFC5

data generation

./ rsst -c ../ config.yml generatedata -i 1 -a 10.0.1.101:6479 -predistest0102

Writing the task to perform json redissyncer-cli requires adding taskexamples

"sourcePassword": "redistest0102","sourceRedisAddress": "10.0.1.101:6379","targetRedisAddress": "10.0.1.102:6379","targetPassword": "redistest0102","taskName": "testtask","targetRedisVersion": 4.0,"autostart": true,"afresh": true,"batchSize": 100}

start task

redissyncer-cli -iredissyncer-cli > task create source ./ task.json

data check

wgethttps://github.com/TraceNature/rediscompare/releases/download/v1.0.0/rediscompare-1.0.0-linux-amd64.tar.gzrediscompare compare single2single --saddr "10.0.1.101:6479" --spassword "redistest0102" --taddr "10.0.1.102:6479" --tpassword"redistest0102" --compartimes 3 After reading the above, do you have any further understanding of the challenges and solutions in the Redis migration synchronization process? If you still want to know more knowledge or related content, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.