Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to configure Celery in Redis. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

Let's put aside the Web framework and introduce how to configure Celery tasks based on Redis.

Pip install celery [redis]

Project structure

$tree your_project your_project ├── _ _ init__.py ├── main.py ├── celery.py └── tasks.py 0 directories, 4 files

Where main.py is the business code that triggers Task. Of course, the file name can be changed at will. Celery.py is the location of the app definition of Celery, tasks.py is the location of the definition of Task, and the file name is not recommended.

Configure Celery

Write the following code in celery.py:

From celery import Celery from .settings import REDIS_URL APP = Celery (main=__package__, broker=REDIS_URL, backend=REDIS_URL, include= [f'{_ _ package__} .tasks'],) APP.conf.update (task_track_started=True)

Where REDIS_URL is introduced from the same configuration settings.py, probably in the form of redis://localhost:6379/0. Here Redis is used as both broker and backend. That is, a database that serves as both a message queue and a result feedback (only saved for 1 day by default).

In include=, you need to fill in a list of package names for downstream worker. The tasks.py file for the same package is selected here.

The additional setting of task_track_started is the command Worker to feedback the STARTED status. By default, there is no way to know when the task starts.

Write a task and call

In the tasks.py file, add the implementation of the asynchronous task.

From .celery import APP @ APP.task def do_sth (): pass

Asynchronous calls can be triggered with .apply _ async where tasks need to be initiated. That is, the actual message is only sent to the message queue, and the real operation is performed remotely.

From celery.result import AsyncResult from .tasks imprt do_sth result = do_sth.apply_async () assert isinstance (result, AsyncResult)

Run Worker:

Celery-A your_project worker

Operation principle

The sequence diagram of a Task from trigger to completion is as follows:

Where main represents the main process of the business code. It could be a Web service such as Django or Flask, or it could be another type of process. Worker is the Worker of Celery.

After main sends a message, you get an AsyncResult that contains task_id. Only through task_id, you can also construct an AsyncResult to query relevant information. Among them, the running process is mainly represented by state.

Worker keeps an eye on Redis (or other message queues, such as RabbitMQ), querying for new messages. If you get a new message, consume it and start running do_sth. After running, the results corresponding to the returned values, as well as some running information, will be written back to Redis (or other backend, such as Django database, etc.). Anywhere in the system, the results can be queried through the corresponding AsyncResult (task_id).

Status of Celery Task

The following is the state diagram:

Among them, in addition to SUCCESS, there are two end states of failure (FAILURE) and cancellation (REVOKED). On the other hand, RETRY is a temporary waiting state after setting the retry mechanism.

In addition, if the result information saved in Redis is cleaned (only 1 day by default), the task status will become PENDING again. This is a huge problem in design, and fault tolerance should be done when using it.

Common control operation

Result = AsyncResult (task_id) # blocking waiting to return result.wait () # cancel task result.revoke () # Delete task record result.forget ()

Sometimes you need to use wait when you need to wait for the result of running asynchronously in the business main process. If you want to cancel a queued or executed task, you can use revoke. Revoke can be used even if the task has been executed, but there will be no change. If you need to delete task records in advance, you can use forget.

After reading the above, do you have any further understanding of how to configure Celery in Redis? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.