Celery and long-term tasks
I'm just watching a youtube video where the host mentioned that you need to design your celery to keep it short. Tasks that take a few minutes to complete are bad.
It is right? What I see is that I have some kind of long task that lasts 10 minutes. When this task is scheduled frequently, the queue fills up and no other tasks are scheduled. This is the reason?
If so, what should you use for long-term tasks?
source to share
The problem with long running tasks is that you have to wait for them when you push a new version of the software on your server. If you don't wait, your task might be triggered by possibly incompatible code, especially if you've marinated some complex object as a parameter (which is highly discouraged).
source to share
As @ user2097159 said it is good practice to keep long running tasks in a given queue. You have to do it by routing using "settings.CELERY_ROUTES" more details here
If you could estimate how long a task can run, I recommend using soft_time_limit per task, you should be able to handle it.
There is a point in the conversation I gave here
source to share
Amend the base definition Task
to treat the task instance as a generator as needed, and check for TERM or soft timeout on each iteration through the generator. Usually inject "state" dict kwarg into tasks that support it. If this is the first run of the task, allocate a new one in the results cache, otherwise find the existing one from the results cache.
Find a good place in your task to get the result in a short time frame. Update the parameter if necessary state
.
When the control returns to the main task class, check for TERM or soft timeout and if there is, save the object state
and respond to the signal.
source to share