Shulou(Shulou.com)06/02 Report--

This article mainly introduces how to avoid the repeated execution of Celery timing tasks through celery_one, which has a certain reference value, and interested friends can refer to it. I hope you can learn a lot after reading this article.

When using Celery to calculate the number of daily visits, it is found that a task will be executed twice at the same time, and it is found that two tasks were sent at the same time (within 1 second), that is, two worker were generated at the same time, resulting in statistics twice, but the reason can not be found.

Reference: https://blog.csdn.net/qq_41333582/article/details/83899884

Some people use Redis to implement distributed locks, and then others use Celery Once.

Celery Once is also implemented by Redis locking. Celery Once implements the QueueOnce class on the basis of the Task class, which provides the function of deduplicating tasks, so when using it, the method we implement ourselves needs to set QueueOnce to base.

@ task (base=QueueOnce, once= {'graceful': True})

The following once parameter indicates how to handle a duplicate method. The default graceful is False, so Celery will throw an AlreadyQueued exception and manually set it to True, then it will be handled silently.

In addition, if you want to set the key of the task manually, you can specify the keys parameter

@ celery.task (base=QueueOnce, once= {'keys': [' a']}) def slow_add (a, b): sleep (30) return a + b solution step

Celery One allows you to queue up Celery tasks to prevent multiple execution

Installation

Pip install-U celery_once

Required, need Celery4.0, the old version may run, but not officially supported.

Using celery_once,tasks requires inheriting an abstract base tasks named QueueOnce

After the Once installation is complete, you need to configure some options about ONCE in the Celery configuration

From celery import Celeryfrom celery_once import QueueOncefrom time import sleepcelery = Celery ('tasks', broker='amqp://guest@localhost//') # this is not generally available in previous configurations Need to add celery.conf.ONCE = {'backend':' celery_once.backends.Redis', 'settings': {' url': 'redis://localhost:6379/0',' default_timeout': 60 * 60}} # add base@celery.task (base=QueueOnce) def slow_task (): sleep (30) return "Done!" to the original parameter.

To determine the configuration, depending on which backend is used for locking, check the Backends

On the backend, this will override apply_async and delay. It does not affect the direct call to the task.

When running the task, celery_once checks to see if it is not locked (for the Redis key). Otherwise, the task will run normally. Once the task is completed (or ends with an exception), the lock is cleared. If you try to run the task again before the task is completed, an AlreadyQueued exception will be thrown.

Example.delay (10)

Example.delay (10)

Traceback (most recent call last):

..

AlreadyQueued ()

Result = example.apply_async (args= (10))

Result = example.apply_async (args= (10))

Traceback (most recent call last):

..

AlreadyQueued ()

Graceful: if once= {'graceful': True} is set in the task's options, or apply_async is set at run time, the task can return None instead of throwing an AlreadyQueued exception.

From celery_once import AlreadyQueued# Either catch the exception,try: example.delay (10) except AlreadyQueued: pass# Or, handle it gracefully at run time.result = example.apply (args= (10), once= {'graceful': True}) # or by default.@celery.task (base=QueueOnce, once= {' graceful': True}) def slow_task (): sleep (30) return "Done!" Thank you for reading this article carefully. I hope the article "how to avoid the repeated execution of Celery scheduled tasks through celery_one" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.