Shulou(Shulou.com)06/01 Report--

The knowledge of this article "how to achieve task scheduling based on Redis distributed locks" is not quite understood by most people, so the editor summarizes the following contents, detailed content, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "how to achieve task scheduling based on Redis distributed locks" article.

In the process of distributed mass data acquisition, source management is particularly important. In order to ensure that the same task can only be processed by one collector at the same time, the uniqueness of task scheduling must be ensured. Usually, when we carry out distributed data collection, there will be a scheduling module, whose main responsibility is to be responsible for the distribution of the collection task, while ensuring the uniqueness of the task.

Because it is distributed, involving multiple servers (multiple machines), each server involves multiple collectors (multi-processes), and each collector may involve multithreading, so the locking mechanism in the task scheduling module is particularly important. In general, locks may be implemented in the following types according to the implementation architecture of the application:

If the handler is single-process and multithreaded, under python, the Lock object of the threading module can be used to restrict synchronous access to shared variables to achieve thread safety.

In the case of multi-process on a single machine, under python, you can use the Lock object of multiprocessing to handle.

In the case of multi-machine and multi-process deployment, we have to rely on a third-party component (storage lock object) to implement a distributed synchronous lock.

Because the scheduling module is a processing mechanism of multi-machine, multi-process and multi-thread, it accords with the third way.

Distributed lock implementation

At present, the mainstream implementation methods of distributed locks are as follows:

Based on database implementation, such as mysql

Implementation based on cache, such as redis

Based on zookeeper to realize

Each implementation method has its own advantages, taking into account, Redis is the most appropriate choice. The main reasons are:

Redis operates based on memory, and its access speed is faster than that of the database. Under high concurrency, the performance after locking will not degrade too much.

Redis can set the time to live of key values (TTL)

The use of redis is simple, and the overall implementation cost is low.

However, distributed locks implemented using redis also require the following conditions:

Only one thread can hold the lock at a time, and other threads must wait until the lock is released

The operation of the lock must satisfy atomicity

Deadlocks do not occur, for example, the thread that has acquired the lock abruptly exits before releasing the lock, causing other threads to loop and wait for the lock to be released

The addition and release of locks must be set by the same thread

We use redis to implement a distributed synchronization lock to ensure data consistency, which needs to meet the following characteristics:

For mutual exclusion, only one thread can acquire the lock at a time.

Using the ttl of redis to ensure that there is no deadlock, but at the same time, it also brings the problem that multiple threads occupy the lock at the same time, which requires us to set the expiration time of the lock reasonably to avoid it.

Make use of the uniqueness of the lock to ensure that the lock will not be deleted by mistake

In the actual operation, I detach the scheduling module from the whole collection system, based on Java client Jredis (JRedis is a high-performance Java client, which is used to connect to Redis distributed hash key-value database. Provides synchronous and asynchronous) + SpringBoot, which implements a separate service. So that other collectors can request the collection task to be processed by HTTP. The processing process is roughly as follows:

The collector sends task requests to the dispatch center through HTTP.

The dispatch center determines whether the lock exists, and if so, it directly returns the empty set.

If there is no lock, lock the request and obtain the corresponding collection task according to the source rule

Return to the acquired task (or empty if there is no outstanding task), and then delete the lock.

The code implementation of the scheduling module is roughly as follows:

Public static List fetchTask (String lockKeyValue, RedisHashUtils redisHashUtils, HttpServletRequest request

HashServiceInterface hif, ZSetServiceInterface zScoreSet, String dicName) {

List result = new ArrayList ()

Try {

String dicNameLock = "Dispatcher_Task_Lock"; / / Task scheduling lock

If (! redisHashUtils.keyIsExit (dicNameLock, lockKeyValue)) {/ / determine whether the lock exists

/ / add a lock (write the unique identity of the task to the record)

RedisHashUtils.addOneData (dicNameLock, lockKeyValue

DateUtil.getYMDHMS ()

/ / processing task logic

.

/ / remove the lock (task uniqueness identification)

Hsdi.remove (redisHashUtils, dicNameLock, lockKeyValue)

} else {

/ / the lock already exists

System.out.println ("task in progress, temporarily returning an empty collection.")

}

} catch (

Exception e) {e.printStackTrace ()

}

Return result

}

In the actual operation, when adding the lock, the lock must be added with the expiration time, otherwise some unknown exceptions may cause the lock can not be released, and the collector has been unable to obtain the acquisition task.

The above is the content of this article on "how to achieve task scheduling based on Redis distributed locks". I believe we all have some understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.