Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to dispatch Go worker by Python Celery". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "how to dispatch Go worker by Python Celery".

We can use Go to write a Worker for computing-intensive tasks, and then use Python's Celery beat to schedule the Worker. Here's a demonstration:

1. Write Go Worker

It is best to transform compute-intensive tasks into Go language versions so that the benefits can be maximized.

Here, for example, I use the old stem that was reduced from 100 million to 1 last time.

PS, don't be frightened by the following code, in fact, most of the configuration items can be removed, and the core code is only a few lines.

Enter the command:

Go run main.go

You can run the worker

two。 Write Python client

Schedule 100 million to 1 every 5 seconds, but don't run Python worker. Because Go Worker is running, the minus here will be consumed by Go Worker.

Also note that the minus function here is actually written to be recognized, and its content is meaningless, so it's okay to write a pass (because Go Worker is actually consuming it).

When you are finished, start beat for the go_tasks module:

Celery-A go_tasks beat

At this point, the scheduler dispatches the Go Worker to execute the task:

As you can see, we successfully dispatched the Worker written by Go with Python's Celery Beat! You are to be congratulated.

Next, you can see how time-consuming it is to do such a calculation with Python's Worker alone:

Start worker:

Celery worker-A python_tasks-l info-- pool=eventlet

Start beat Scheduler:

Celery-A python_tasks beat

The results are as follows:

As you can see, it takes an average of about 5.2 seconds to reduce Python from 100 million to 1, which is about 100 times different from the Go version.

If we increase the frequency of the scheduler to calculate once per second, the Python version of Worker's task queue will certainly be blocked because Worker consumption is not strong enough. By contrast, the Go version of Worker is very powerful.

So, if you have such a compute-intensive task in your project, try extracting it into a Go version. It might be a surprise.

At this point, I believe that everyone has a deeper understanding of "Python Celery how to dispatch Go worker", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.