Shulou(Shulou.com)06/01 Report--

The development environment for this series of articles:

Window 7 + python2.7 + pycharm5 + celery3.1.25 + django1.9.4

In our daily development work, we often encounter these situations:

1231. In web applications, the user triggers an action and executes a background handler that takes a long time to return results. How can you improve the user experience without blocking http requests and keeping users waiting? 2. Timed task scripts: some timed task scripts are often run in the production environment. If you have thousands of servers and thousands of tasks, it is difficult to manage scheduled tasks. How to manage job effectively? 3. Asynchronous requirements: such as sending SMS / email, pushing messages, cleaning / setting cache?

If you have the above needs, then Celery may be very useful to you.

Celery-distributed task queue system

Celery is a distributed task system that can handle a large number of messages. It is widely used because of its simplicity, flexibility and reliability. Celery focuses on real-time processing tasks and also supports scheduled task scheduling.

1. Characteristics:

Check the execution of scheduled tasks, such as whether the execution was successful, current status, time spent executing the task, and so on.

It is easy to integrate with other frameworks, such as using django to manage background adding, updating and deleting tasks.

It is easy to associate tasks with configuration management.

Multi-process, Eventlet and Gevent modes are optional for concurrent execution.

Provides an error handling mechanism.

2. Architecture diagram

You can see from the above figure that Celery contains the following components:

Producer: it is the task producer that calls Celery API, function, or decorator to generate a task and hand it over to the task queue for processing.

Task scheduling component: the Beat process reads the contents of the configuration file and periodically sends tasks that need to be executed due in the configuration to the task queue.

Celery Worker: the thread responsible for executing tasks, which can be run on multiple servers to improve execution efficiency.

Broker: message middleware that accepts tasks from task producers and forwards work for execution. Celery currently supports RabbitMQ, Redis, MongoDB, Beanstalk, SQLAlchemy, Zookeeper as message middleware, and RabbitMQ is officially recommended.

Result Backend: save status information and results for query after the task is processed. Celery supports Redis, RabbitMQ, MongoDB, Django ORM, SQLAlchemy and other methods by default.

Start using Celery

Here both Broker and Result Backend choose RabbitMQ.

1. Installation

1) install RabbitMQ

2) install Celery 3.1.25

Why did you choose this lower version? Please see the list of questions at the bottom.

1pip install celery==3.1.252, a simple example

Create a tasks.py file in a directory with the following contents:

Celery = Celery (, backend=,broker=

2.2) start celery worker

Go to the directory at the level of the tasks.py file and execute the following command:

Celery-A tasks worker-- loglevel=info

The startup output information is as follows:

View Code

2.3) Test results

From another terminal, go to the directory at the tasks.py level and enter the python command line:

At this point, celery worker will have a message:

At this point, the introduction to celery ends here, and the next section describes how to use celery in django.

List of problems encountered:

1. Error starting celery worker Times

ERROR:

] Unrecoverable error: TypeError ('must be integer, not _ subprocess_handle',)

Traceback (most recent call last):

File "c:\ development\ env\ zillow2\ lib\ site-packages\ celery\ worker\ worker.py", line 203, in start

Self.blueprint.start (self)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ celery\ bootsteps.py", line 119, in start

Step.start (parent)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ celery\ bootsteps.py", line 370, in start

Return self.obj.start ()

File "c:\ development\ env\ zillow2\ lib\ site-packages\ celery\ concurrency\ base.py", line 131, in start

Self.on_start ()

File "c:\ development\ env\ zillow2\ lib\ site-packages\ celery\ concurrency\ prefork.py", line 112, in on_start

* * self.options)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ pool.py", line 1008, in init

Self._create_worker_process (I)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ pool.py", line 1117, in _ create_worker_process

W.start ()

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ process.py", line 122, in start

Self._popen = self._Popen (self)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ context.py", line 383, in _ Popen

Return Popen (process_obj)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ popen_spawn_win32.py", line 64, in init

_ winapi.CloseHandle (ht)

TypeError: must be integer, not _ subprocess_handle

(zillow2) C:\ DEVELOPMENT\ zillow2\ project > Traceback (most recent call last):

File "", line 1, in

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ spawn.py", line 159, in spawn_main

New_handle = steal_handle (parent_pid, pipe_handle)

File "c:\ development\ env\ zillow2\ lib\ site-packages\ billiard\ reduction.py", line 121, in steal_handle

_ winapi.PROCESS_DUP_HANDLE, False, source_pid)

WindowsError: [Error 87] The parameter is incorrect

Solution: this is because the windows system does not support versions above celery4. Please downgrade to 3.xx to solve this problem.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.