Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to understand Redis-based distributed locks in distributed systems". In daily operation, I believe many people have doubts about how to understand Redis-based distributed locks in distributed systems. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to understand Redis-based distributed locks in distributed systems". Next, please follow the editor to study!

For new projects, there will be occasional uneven accounts. The explanation given by the previous technical boss when he left was: I checked, but I didn't find the reason, and then I was too busy to solve it, which may be the reason for the framework.

Now that the project is delivered, such a problem must be solved. Combed all the accounting processing logic, and finally found the reason: database concurrent operations caused by hot accounts. On this issue, let's talk about Redis-based distributed locks in distributed systems. By the way, also break down the cause of the problem and the solution.

Cause analysis

The system concurrency is not high, there are hot accounts, but it is not so serious. The root of the problem lies in the system architecture design, which artificially creates concurrency. The scenario is like this: merchants import a batch of data in batches, the system will carry out pre-processing, and increase or decrease the account balance.

At this point, another scheduled task will also scan and update the account. And the operation of the same account is distributed to each system, the hot account also appears.

In view of the solution to this problem, from the architectural level, we can consider to extract the accounting system and deal with it in one system, and all the database transactions and execution order will be handled by the accounting system as a whole. From a technical point of view, hot accounts can be locked through the locking mechanism.

This article gives a detailed explanation on the implementation of distributed locks for hot accounts.

Lock analysis

In the multithreaded environment of Java, there are usually several types of locks that can be used:

JVM memory model-level locks, such as synchronized, Lock, etc.

Database locks, such as optimistic locks, pessimistic locks, etc.

Distributed lock

JVM memory-level locks can ensure the safety of threads under single service, such as multiple threads accessing / modifying a global variable. However, when the system is deployed in a cluster, local locks at the JVM level are powerless.

Pessimistic lock and optimistic lock

As in the above case, hot spot accounts belong to shared resources in distributed systems, and we usually use database locks or distributed locks to solve them.

Database lock is divided into optimistic lock and pessimistic lock.

Pessimistic locks are based on the exclusive locks provided by the database (Mysql's InnoDB). When performing a transaction operation, through select... For update statement, MySQL adds an exclusive lock to each row of data in the query result set, and other threads block update and delete operations on the record. In order to achieve the sequential execution of shared resources (modification)

Optimistic lock is relative to pessimistic lock, optimistic lock assumes that the data generally will not cause conflict, so when the data is submitted for update, it will formally detect whether the data conflict or not. If there is a conflict, the abnormal information is returned to the user and the user is allowed to decide what to do. Optimistic locks are suitable for scenarios with more reads and less writes, which can improve the throughput of the program. Optimistic lock implementation is usually based on recording status or adding a version version.

Pessimistic lock failure scenario

Pessimistic locks are used in the project, but pessimistic locks fail. This is also a common misunderstanding when using pessimistic locks. Let's analyze it below.

The normal process of using pessimistic locks:

Through select... For update lock record

Calculate the new balance, modify the amount and save

Execute complete release lock

A process that often makes mistakes:

Query account balance and calculate new balance

Through select... For update lock record

Modify the amount and save it

Execute complete release lock

In the wrong process, for example, the balance queried by An and B services is 100 minus A minus 50 minus B minus 40, and then A locks the record and updates the database to 50. After A releases the lock, B locks the record and updates the database to 60. Obviously, the latter overwrites the updates of the former. The solution is to expand the scope of the lock and advance the lock before the new balance is calculated.

Usually pessimistic locks put great pressure on the database, and in practice, optimistic locks or distributed locks are usually used according to the scenario.

Let's get down to business and talk about the implementation of distributed locks based on Redis.

Redis distributed lock actual combat exercise

Here, Spring Boot, Redis, Lua scripts are used as examples to demonstrate the implementation of distributed locks. To simplify processing, Redis in the example takes on both the distributed lock function and the database function.

Scene construction

In a cluster environment, the basic steps are to operate the amount of the same account:

Read the user amount from the database

Program modification amount

Then store the latest amount in the database

The following from the initial unlocked, non-synchronous processing, gradually deduce the final distributed lock.

Basic Integration and Class Construction

Prepare a basic business environment for unlocked processing.

First, introduce related dependencies into the Spring Boot project:

Org.springframework.boot spring-boot-starter-data-redis org.springframework.boot spring-boot-starter-web

The entity class UserAccount corresponding to the account:

Public class UserAccount {/ / user ID private String userId; / / account amount private int amount; / / add account amount public void addAmount (int amount) {this.amount = this.amount + amount;} / / omit construction method and getter/setter}

Create a thread implementation class AccountOperationThread:

Public class AccountOperationThread implements Runnable {private final static Logger logger = LoggerFactory.getLogger (AccountOperationThread.class); private static final Long RELEASE_SUCCESS = 1L; private String userId; private RedisTemplate redisTemplate; public AccountOperationThread (String userId, RedisTemplate redisTemplate) {this.userId = userId; this.redisTemplate = redisTemplate;} @ Override public void run () {noLock ();} / * * unlocked * / private void noLock () {try {Random random = new Random () / / simulate thread for business processing TimeUnit.MILLISECONDS.sleep (random.nextInt (100) + 1);} catch (InterruptedException e) {e.printStackTrace ();} / / get user account UserAccount userAccount = (UserAccount) redisTemplate.opsForValue () .get (userId) from the simulation database; / / amount + 1 userAccount.addAmount (1) Logger.info (Thread.currentThread (). GetName () + ": user id:" + userId + "amount:" + userAccount.getAmount ()); / / simulate saving back to the database redisTemplate.opsForValue () .set (userId, userAccount);}}

The instantiation of RedisTemplate is given to Spring Boot:

@ Configurationpublic class RedisConfig {@ Bean public RedisTemplate redisTemplate (RedisConnectionFactory redisConnectionFactory) {RedisTemplate redisTemplate = new RedisTemplate (); redisTemplate.setConnectionFactory (redisConnectionFactory); Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer (Object.class); ObjectMapper objectMapper = new ObjectMapper (); objectMapper.setVisibility (PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY); objectMapper.enableDefaultTyping (ObjectMapper.DefaultTyping.NON_FINAL); jackson2JsonRedisSerializer.setObjectMapper (objectMapper) / / set value serialization rules and key serialization rules redisTemplate.setValueSerializer (jackson2JsonRedisSerializer); redisTemplate.setKeySerializer (new StringRedisSerializer ()); redisTemplate.afterPropertiesSet (); return redisTemplate;}}

Finally, prepare a TestController to trigger the multithreading:

@ RestControllerpublic class TestController {private final static Logger logger = LoggerFactory.getLogger (TestController.class); private static ExecutorService executorService = Executors.newFixedThreadPool (10); @ Autowired private RedisTemplate redisTemplate; @ GetMapping ("/ test") public String test () throws InterruptedException {/ / initialize user user_001 to Redis with an account amount of 0 redisTemplate.opsForValue () .set ("user_001", new UserAccount ("user_001", 0)) / / start 10 threads for synchronous testing, and each thread adds 1 yuan of for to the account (int I = 0; I

< 10; i++) { logger.info("创建线程i=" + i); executorService.execute(new AccountOperationThread("user_001", redisTemplate)); } // 主线程休眠1秒等待线程跑完 TimeUnit.MILLISECONDS.sleep(1000); // 查询Redis中的user_001账户 UserAccount userAccount = (UserAccount) redisTemplate.opsForValue().get("user_001"); logger.info("user id : " + userAccount.getUserId() + " amount : " + userAccount.getAmount()); return "success"; }} 执行上述程序，正常来说10个线程，每个线程加1，结果应该是10。但多执行几次，会发现，结果变化很大，基本上都要比10小。 [pool-1-thread-5] c.s.redis.thread.AccountOperationThread : pool-1-thread-5 : user id : user_001 amount : 1[pool-1-thread-4] c.s.redis.thread.AccountOperationThread : pool-1-thread-4 : user id : user_001 amount : 1[pool-1-thread-3] c.s.redis.thread.AccountOperationThread : pool-1-thread-3 : user id : user_001 amount : 1[pool-1-thread-1] c.s.redis.thread.AccountOperationThread : pool-1-thread-1 : user id : user_001 amount : 1[pool-1-thread-1] c.s.redis.thread.AccountOperationThread : pool-1-thread-1 : user id : user_001 amount : 2[pool-1-thread-2] c.s.redis.thread.AccountOperationThread : pool-1-thread-2 : user id : user_001 amount : 2[pool-1-thread-5] c.s.redis.thread.AccountOperationThread : pool-1-thread-5 : user id : user_001 amount : 2[pool-1-thread-4] c.s.redis.thread.AccountOperationThread : pool-1-thread-4 : user id : user_001 amount : 3[pool-1-thread-1] c.s.redis.thread.AccountOperationThread : pool-1-thread-1 : user id : user_001 amount : 4[pool-1-thread-3] c.s.redis.thread.AccountOperationThread : pool-1-thread-3 : user id : user_001 amount : 5[nio-8080-exec-1] c.s.redis.controller.TestController : user id : user_001 amount : 5 以上述日志为例，前四个线程都将值改为1，也就是后面三个线程都将前面的修改进行了覆盖，导致最终结果不是10，只有5。这显然是有问题的。 Redis同步锁实现 针对上面的情况，在同一个JVM当中，我们可以通过线程加锁来完成。但在分布式环境下，JVM级别的锁是没办法实现的，这里可以采用Redis同步锁实现。 基本思路：第一个线程进入时，在Redis中进记录，当后续线程过来请求时，判断Redis是否存在该记录，如果存在则说明处于锁定状态，进行等待或返回。如果不存在，则进行后续业务处理。 /** * 1.抢占资源时判断是否被锁。 * 2.如未锁则抢占成功且加锁，否则等待锁释放。 * 3.业务完成后释放锁,让给其它线程。 * * 该方案并未解决同步问题，原因：线程获得锁和加锁的过程，并非原子性操作，可能会导致线程A获得锁，还未加锁时，线程B也获得了锁。 */ private void redisLock() { Random random = new Random(); try { TimeUnit.MILLISECONDS.sleep(random.nextInt(1000) + 1); } catch (InterruptedException e) { e.printStackTrace(); } while (true) { Object lock = redisTemplate.opsForValue().get(userId + ":syn"); if (lock == null) { // 获得锁 ->

Lock-> jump out of the loop logger.info (Thread.currentThread (). GetName () + ": acquire the lock"); redisTemplate.opsForValue (). Set (userId + ": syn", "lock"); break;} try {/ / wait 500ms to retry to acquire the lock TimeUnit.MILLISECONDS.sleep;} catch (InterruptedException e) {e.printStackTrace () }} try {/ / get user account UserAccount userAccount = (UserAccount) redisTemplate.opsForValue () .get (userId); if (userAccount! = null) {/ / set amount userAccount.addAmount (1); logger.info (Thread.currentThread (). GetName () + ": user id:" + userId + "amount:" + userAccount.getAmount ()) / / simulate saving to the database redisTemplate.opsForValue (). Set (userId, userAccount);}} finally {/ / release lock redisTemplate.delete (userId + ": syn"); logger.info (Thread.currentThread (). GetName () + ": release lock");}}

In the while code block, first determine whether the corresponding user ID exists in the Redis, if not, then lock the set, and if so, jump out of the loop and continue to wait.

The above code seems to implement the locking function, but when the program is executed, it will be found that there is still a concurrency problem as without locking. The reason is that the operation of acquiring and adding locks is not atomic. For example, two threads find that the lock is null and both are locked, and the concurrency problem still exists.

Redis atomic synchronization lock

In order to solve the above problems, the process of acquiring lock and locking can be atomized. Based on the atomized API provided by spring-boot-data-redis, you can achieve:

/ / this method uses the instruction of redis: SETNX key value// 1.key does not exist. If value,setIfAbsent is set successfully, true;// 2.key is returned. If setting fails, null,setIfAbsent returns false;// 3. Atomic operation; Boolean setIfAbsent (K var1, V var2)

The atomization operation of the above method encapsulates the setnx command of Redis. The use of setnx in Redis is as follows:

Redis > SETNX mykey "Hello" (integer) 1redis > SETNX mykey "World" (integer) 0redis > GET mykey "Hello"

The first time, when setting mykey, it does not exist, 1 is returned, which means the setting is successful; the second time, when mykey is set, it already exists, 0 is returned, indicating that the setting failed. If you query the corresponding value of mykey again, you will find that it is still the value set for the first time. In other words, redis's setnx ensures that only one key can be set successfully by one service.

After understanding the above API and underlying principles, let's take a look at the implementation method code in the thread as follows:

/ * 1. Atomic operation lock * 2. Contending thread loop retries to acquire lock * 3. Service completes releasing lock * / private void atomicityRedisLock () {/ / Spring data redis supported atomic operation while (! redisTemplate.opsForValue (). SetIfAbsent (userId + ": syn", "lock")) {try {/ / wait for 100ms to retry to acquire lock TimeUnit.MILLISECONDS.sleep (100);} catch (InterruptedException e) {e.printStackTrace () }} logger.info (Thread.currentThread (). GetName () + ": get the lock"); try {/ / get the user account in the simulation database UserAccount userAccount = (UserAccount) redisTemplate.opsForValue (). Get (userId); if (userAccount! = null) {/ / set amount userAccount.addAmount (1) Logger.info (Thread.currentThread (). GetName () + ": user id:" + userId + "amount:" + userAccount.getAmount ()); / / simulate saving back to the database redisTemplate.opsForValue (). Set (userId, userAccount);}} finally {/ / release lock redisTemplate.delete (userId + ": syn"); logger.info (Thread.currentThread (). GetName () + ": release lock") }}

If you execute the code again, you will find that the result is correct, that is, you can successfully lock the distributed thread.

Deadlock of Redis distributed Lock

Although the above code execution results are fine, if the application is unusually down and does not have time to execute the method of releasing the lock in finally, then other threads will never be able to acquire the lock.

At this point, you can use the overloading method of setIfAbsent:

Boolean setIfAbsent (K var1, V var2, long var3, TimeUnit var5)

Based on this method, the expiration time of the lock can be set. In this way, even if the thread that acquired the lock goes down, other threads can acquire the lock normally after the data in the Redis expires.

The sample code is as follows:

Private void atomicityAndExRedisLock () {atomic operation supported by try {/ / Spring data redis, and set the 5-second expiration time while (! redisTemplate.opsForValue (). SetIfAbsent (userId + ": syn", System.currentTimeMillis () + 5000, 5000, TimeUnit.MILLISECONDS)) {/ / wait 100ms to retry to acquire the lock logger.info (Thread.currentThread (). GetName () + ": try to loop to acquire the lock") TimeUnit.MILLISECONDS.sleep (1000);} logger.info (Thread.currentThread (). GetName () + ": acquire lock -"); / / the application crashes here, the process exits, unable to execute finally; Thread.currentThread (). Interrupt (); / / business logic.} catch (InterruptedException e) {e.printStackTrace () } finally {/ / release lock if (! Thread.currentThread (). IsInterrupted ()) {redisTemplate.delete (userId + ": syn"); logger.info (Thread.currentThread (). GetName () + ": release lock");}} service timeout and daemon thread

The above adds the timeout of Redis, which seems to solve the problem, but introduces a new problem.

For example, under normal circumstances, thread A can normally process the business in 5 seconds, but occasionally it will occur for more than 5 seconds. If the timeout is set to 5 seconds, thread An acquires the lock, but business logic processing takes 6 seconds. At this point, thread An is still in normal business logic, and thread B has acquired the lock. When thread A finishes processing, it is possible to release thread B's lock.

There are two problem points in the above scenario:

First, threads An and B may be executing at the same time, and there is a concurrency problem.

Second, thread A may release thread B's lock, causing a series of vicious cycles.

Of course, you can determine whether the lock belongs to thread An or thread B by setting the value value in Redis. But a closer analysis reveals that the essence of the problem is that thread A takes more time to execute business logic than the lock timeout.

Then there are two solutions:

First, set the timeout long enough to ensure that the business code can be executed before the lock is released

Second, add a daemon thread to the lock to increase the time for locks that are about to expire but not released

The first method requires the time-consuming of the business logic in most cases of the whole bank, and sets the timeout.

Second, you can dynamically increase the lock timeout by using the following daemon threads.

Public class DaemonThread implements Runnable {private final static Logger logger = LoggerFactory.getLogger (DaemonThread.class); / / end the daemon thread private volatile boolean daemon = true; / / daemon lock private String lockKey; private RedisTemplate redisTemplate; public DaemonThread (String lockKey, RedisTemplate redisTemplate) {this.lockKey = lockKey; this.redisTemplate = redisTemplate } @ Override public void run () {try {while (daemon) {long time = redisTemplate.getExpire (lockKey, TimeUnit.MILLISECONDS); / / renew if if the remaining validity period is less than 1 second (time < 1000) {logger.info ("daemon:" + Thread.currentThread (). GetName () + "extend the lock time by 5000 milliseconds") RedisTemplate.expire (lockKey, 5000, TimeUnit.MILLISECONDS);} TimeUnit.MILLISECONDS.sleep (300);} logger.info ("daemon:" + Thread.currentThread (). GetName () + "close");} catch (InterruptedException e) {e.printStackTrace ();}} / / main thread initiatively calls end public void stop () {daemon = false;}}

The above thread acquires the timeout of the lock in the Redis every 300ms, and if it is less than 1 second, it will be extended by 5 seconds. When the main thread call closes, the daemon thread shuts down.

The related code implementation in the main thread:

Private void deamonRedisLock () {/ / daemon thread DaemonThread daemonThread = atomic operations supported by null; / / Spring data redis, and set the 5-second expiration time String uuid = UUID.randomUUID () .toString (); String value = Thread.currentThread () .getId () + ":" + uuid Try {while (! redisTemplate.opsForValue (). SetIfAbsent (userId + ": syn", value, 5000, TimeUnit.MILLISECONDS)) {/ / wait 100ms to retry to acquire the lock logger.info (Thread.currentThread (). GetName () + ": try to loop to acquire the lock"); TimeUnit.MILLISECONDS.sleep (1000);} logger.info (Thread.currentThread (). GetName () + ": acquire the lock -") / / start the daemon thread daemonThread = new DaemonThread (userId + ": syn", redisTemplate); Thread thread = new Thread (daemonThread); thread.start (); / / execute the business logic for 10 seconds. TimeUnit.MILLISECONDS.sleep (10000);} catch (InterruptedException e) {e.printStackTrace ();} finally {/ / release lock here also requires atomic operation. In the future, we will talk about String result = (String) redisTemplate.opsForValue (). Get (userId + ": syn") through Redis + Lua; if (value.equals (result)) {redisTemplate.delete (userId + ": syn") Logger.info (Thread.currentThread (). GetName () + ": release lock -");} / / close the daemon thread if (daemonThread! = null) {daemonThread.stop ();}

After the lock is acquired, the daemon thread is opened and closed in finally.

Implementation based on Lua script

In the above logic, we guarantee the atomization of lock judgment and execution based on the atomization operation provided by spring-boot-data-redis. In non-Spring Boot projects, it can be implemented based on Lua scripts.

First, define the locked and unlocked Lua scripts and the corresponding DefaultRedisScript objects, and add the following instantiation code to the RedisConfig configuration class:

Configurationpublic class RedisConfig {/ / lock script private static final String LOCK_SCRIPT = "if redis.call ('setnx',KEYS [1], ARGV [1]) = 1" + "then redis.call (' expire',KEYS [1], ARGV [2])" + "return 1" + "else return 0 end" Private static final String UNLOCK_SCRIPT = "if redis.call ('get', KEYS [1]) = = ARGV [1] then return redis.call" + "(' del', KEYS [1]) else return 0 end"; / /. Omit some code @ Bean public DefaultRedisScript lockRedisScript () {DefaultRedisScript defaultRedisScript = new DefaultRedisScript (); defaultRedisScript.setResultType (Boolean.class); defaultRedisScript.setScriptText (LOCK_SCRIPT); return defaultRedisScript;} @ Bean public DefaultRedisScript unlockRedisScript () {DefaultRedisScript defaultRedisScript = new DefaultRedisScript (); defaultRedisScript.setResultType (Long.class); defaultRedisScript.setScriptText (UNLOCK_SCRIPT); return defaultRedisScript;}}

Then pass the above two objects into the class by creating a new constructor in the AccountOperationThread class (omitting this part of the demonstration). Then, it can be called based on RedisTemplate, and the modified code is implemented as follows:

Private void deamonRedisLockWithLua () {/ / daemon thread DaemonThread daemonThread = atomic operations supported by null; / / Spring data redis, and set the 5-second expiration time String uuid = UUID.randomUUID () .toString (); String value = Thread.currentThread () .getId () + ":" + uuid Try {while (! redisTemplate.execute (lockRedisScript, Collections.singletonList (userId + ": syn"), value, 5) {/ / wait for 1000 milliseconds to retry to acquire the lock logger.info (Thread.currentThread (). GetName () + ": try to loop to acquire the lock"); TimeUnit.MILLISECONDS.sleep (1000);} logger.info (Thread.currentThread (). GetName () + ": acquire the lock -") / / start the daemon thread daemonThread = new DaemonThread (userId + ": syn", redisTemplate); Thread thread = new Thread (daemonThread); thread.start (); / / execute the business logic for 10 seconds. TimeUnit.MILLISECONDS.sleep (10000);} catch (InterruptedException e) {logger.error ("exception", e);} finally {/ / use Lua script: first determine whether the lock is set by yourself, then delete / / key exists, delete key;key exists when current value = expected value, return 0 when current value = expected value Long result = redisTemplate.execute (unlockRedisScript, Collections.singletonList (userId + ": syn"), value); logger.info ("redis unlock: {}", RELEASE_SUCCESS.equals (result)); if (RELEASE_SUCCESS.equals (result)) {if (daemonThread! = null) {/ / close daemon thread daemonThread.stop () Logger.info (Thread.currentThread (). GetName () + ": release lock--);}

Locking in the while loop and releasing locks in finally are implemented based on Lua scripts.

Other factors of Redis lock

In addition to the above examples, the following situations and scenarios can be considered when using Redis distributed locks.

Non-reentrant of Redis lock

When a thread requests a lock again while holding a lock, if a lock supports multiple locks by a thread, then the lock is reentrant. If a non-reentrant lock is locked again, it will fail because the lock is already held. Redis can count locks by reentering, adding 1 when locking, minus 1 when unlocking, and releasing locks when the count returns to 0.

Reentrant locks are efficient but increase the complexity of the code, which is not illustrated here.

Wait for the lock to be released

In some business scenarios, if you find that you are locked, you will return directly. However, in some scenarios, the client needs to wait for the lock to be released and then grab the lock. The above example belongs to the latter. There are also two options for waiting for the lock to be released:

Client rotation training: when the lock is not obtained, wait a period of time and then reacquire it until it is successful. The above example is based on this approach. The disadvantage of this method is also obvious, it consumes server resources, and when the concurrency is large, it will affect the efficiency of the server.

Use the subscription publishing function of Redis: when the lock acquisition fails, the subscription lock release message is released, and when the lock is successfully acquired, the release message is sent.

Active / standby switching and brain fissure in Cluster

In the cluster deployment where Redis includes master-slave synchronization, if the master node dies, the slave node is promoted to the master node. If client A successfully locks the master node and the instruction is not synchronized to the slave node, the master node dies and the slave node is promoted to the master node, and there is no locked data in the new master node. In this case, client B may lock successfully, resulting in a concurrency scenario.

When the brain fissure occurs in the cluster, the Redis master node is in a different network zone from the slave node and the sentinel cluster. The sentinel cluster is not aware of the existence of master and promotes the slave node to a master node, at which point there are two different master nodes. This will also lead to concurrency problems. Redis Cluster clusters are deployed in the same way.

At this point, the study on "how to understand Redis-based distributed locks in distributed systems" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.