How can I split a large django-celery tasks.py module into smaller chunks?
I have very long task functions that I would like to break out of my module tasks.py
, but can still be called like from myapp.tasks import my_task
. I've tried creating submodules for each task, but then I need to insert some tricks in tasks/__init__.py
, which seem pretty hacky (and require the tasks to be the same names as the modules):
import pkgutil
for module_loader, name, ispkg in pkgutil.iter_modules(__path__):
__import__('myapp.tasks.%s' % name)
locals().update({name: getattr(locals()[name], name)})
source to share
I think the most straight forward solution is to write the tasks as Django control commands first, and then call them from tasks.py. This way you can split your tasks into one task per file and have the added benefit of being able to arbitrarily call your tasks from the command line.
So in tasks.py
:
from django.core import management
@task()
def my_task():
management.call_command('my_task')
source to share
I'm not familiar with Django-Celery, but I am very familiar with Celery, so this is more of a general post that might answer your question ...
Celery tasks are identified by name when executed. By default, tasks are given the name of their Python location, i.e. '%s.%s' % (my_task.__module__, my_task.__name__)
, or 'myapp.tasks.function_name'. However, you can override the task name by giving the @task decorator the "name" kwarg. So if you want to override the task name:
# We're in myapp.tasks but the task will not be named myapp.tasks.my_task as expected
@task(name='some.overridden.name')
def my_task():
print 'Something'
source to share
The following also works:
- create package
tasks
- distribute your tasks in python modules inside a package
tasks
- import modules into
tasks/__init__.py
Example:
django_app
| tasks
| __init__.py
| filesystem_tasks.py
| admin.py
| url.py
-
tasks/filesystem_tasks.py
may contain the following:from __future__ import absolute_import, unicode_literals from celery import shared_task @shared_task def add(x, y): return x + y
-
tasks/__init__.py
may contain the following:from .filesystem_tasks import *
-
Celery launch indicates tasks as they would be declared in the same
tasks.py
at the same level as ex.url.py
...
source to share