Shulou(Shulou.com)06/02 Report--

This article is about how to use Celery Once to prevent Celery from performing the same task repeatedly. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

When using Celery, I found that sometimes Celery will execute the same task twice. What I have encountered is that the same task is executed separately in different worker, and the time difference is only a few milliseconds. I always thought there was something wrong with my logic, but later I found that other people had similar problems, and then basically those who went wrong used Redis as Broker. On the one hand, I didn't want to replace Redis, so I had to add distributed locks when task was executed.

However, a search in Celery's issue shows that some people use Redis to implement distributed locks, and then others use Celery Once. After a rough look at Celery Once, I found that it was very consistent with the current situation, so I used it.

Celery Once is also implemented by Redis locking. Celery Once implements the QueueOnce class on the basis of the Task class, which provides the function of deduplicating tasks, so when using it, the method we implement ourselves needs to set QueueOnce to base.

@ task (base=QueueOnce, once= {'graceful': True})

The following once parameter indicates how to handle a duplicate method. The default graceful is False, so Celery will throw an AlreadyQueued exception and manually set it to True, then it will be handled silently.

In addition, if you want to set the key of the task manually, you can specify the keys parameter

Celery.task (base=QueueOnce, once= {'keys': [' a']}) def slow_add (a, b): sleep (30) return a + b generally speaking, it is divided into several steps

Step one, install

Pip install-U celery_once

The second step is to add configuration

From celery import Celeryfrom celery_once import QueueOncefrom time import sleepcelery = Celery ('tasks', broker='amqp://guest@localhost//') celery.conf.ONCE = {' backend': 'celery_once.backends.Redis',' settings': {'url':' redis://localhost:6379/0', 'default_timeout': 60 * 60}}

Third, modify the delay method

Example.delay (10) # modified to result = example.apply_async (args= (10))

Step 4: modify the task parameter

@ celery.task (base=QueueOnce, once= {'graceful': True, keys': [' a']}) def slow_add (a, b): sleep (30) return a + b Thank you for reading! This is the end of this article on "how to use Celery Once to prevent Celery from repeatedly performing the same task". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.