How to ensure a canceled Celery task is never started after the whole workflow is down and back
Use case is used here.
Day 1: Call the celery task with a countdown after 7 days Day 2: Cancel this task Day 3: The update happens, so all workflows drop down and then come back again after a while
I tested a similar scenario, I realized that there is a revocation list for processes that are canceled in all workflows. But the message (corresponding to the task) remains in the workflow to which the task is delegated. Therefore, when all worker processes are omitted, the revocation list information is also lost.
I want to understand, if this happens, then after all the workers return, then will this process start without canceling or canceling? I say this because the revocation list information is (from what I feel) only in workflows, not brokers.
Can anyone confirm this behavior?
source to share
You're right. Celery workers keep the list of recalled tasks in memory, and if all workers restart, the list disappears. Quoting the celery user guide on workers:
Canceling tasks work by sending a broadcast message to all workers, then the workers keep a list of the canceled tasks in memory. When a worker starts, it syncs the revoked tasks with other workers in the cluster.
The revoked task list is in memory, so if all workers restart the revoked ID list, it will also disappear. If you want to keep this list between restarts, you need to specify the file to store the data using the -statedb argument to the celery worker:
For more information, see the Permanent Recovery section in the Celery User Guide.
source to share