Shulou(Shulou.com)06/02 Report--

This article introduces the knowledge of "how to solve the problem of python-redis-lock distributed locks". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Problem background

In order to improve efficiency when using celery to perform our asynchronous tasks, celery can start multiple processes to start the corresponding worker.

But there will be such a situation: after obtaining the data source, the database will be scanned, and according to UUID to determine whether to insert or update, two worker at the same time (a difference of 0.001s) got the UUID, but when one of them did not insert, the other also scanned the database, then both worker will think that the UUID they got does not exist in the database, so they will call the INSERT method to insert.

Several solutions

In order to solve this problem, there are generally the following solutions.

Distributed lock family:

Database:

Exclusive lock (pessimistic lock)

Optimistic lock

Redis

Implement the Redis SET SETNX operation by yourself, and ensure the atomic operation with the Lua script

The algorithm of distributed lock implementation in RedLock Redis is controversial and should be used cautiously.

The techniques that python-redis-lock will introduce in this article. The distributed locks provided by this library are very flexible. Do you need to time out? Do you need to refresh automatically? Do you want to block it? It's all optional. There is no best algorithm, only the most appropriate algorithm, developers should carefully choose which technology to implement according to the actual demand scenario.

Zookeeper

This should be the most powerful, professional and stable. I haven't used it yet. I'll write an article to sum it up when I get it.

Expansion of ideas

In the celery scenario, celery_once can also be used for task de-reoperation, and the underlying celery_once is also implemented using redis.

You can refer to this article

Talk is cheap, show me your code!

A simple demo

Import randomimport timeimport threadingimport redis_lockimport redisHOST = 'YOUR IP LOCATE'PORT =' 6379'PASSWORD = 'password'def get_redis (): pool= redis.ConnectionPool (host=HOST, port=PORT, password=PASSWORD, decode_responses=True, db=2) r = redis.Redis (connection_pool=pool) return rdef ask_lock (uuid): lock = redis_lock.Lock (get_redis () Uuid) if lock.acquire (blocking=False): print ("% s Got the lock."% uuid) time.sleep (5) lock.release () print ("% s Release the lock."% uuid) else: print ("% s Someone else has the lock."% uuid) def simulate (): for i in range (10): id = random.randint (0) 5) t = threading.Thread (target=ask_lock, args= (str (id) t.start () simulate ()

Output:

4 Got the lock.

5 Got the lock.

3 Got the lock.

5 Someone else has the lock.

5 Someone else has the lock.

2 Got the lock.

5 Someone else has the lock.

4 Someone else has the lock.

3 Someone else has the lock.

3 Someone else has the lock.

2 Release the lock.

5 Release the lock.

4 Release the lock.

3 Release the lock.

This is the end of the content of "how to solve the problem of python-redis-lock distributed locks". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.