Celery Queue related events

I have two Django projects, each with a Celery app:

- fooproj.celery_app
- barproj.celery_app

      

Each application has its own celery worker:

celery worker -A fooproj.celery_app -l info -E -Q foo_queue
celery worker -A barproj.celery_app -l info -E -Q bar_queue

      

This is how I set up my Celery apps:

import os
from celery import Celery
from django.conf import settings


# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings.local')


app = Celery('celery_app', broker=settings.BROKER_URL)
app.conf.update(
    CELERY_ACCEPT_CONTENT=['json'],
    CELERY_TASK_SERIALIZER='json',
    CELERY_RESULT_SERIALIZER='json',
    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
    CELERY_SEND_EVENTS=True,
    CELERY_DEFAULT_QUEUE=settings.CELERY_DEFAULT_QUEUE,
    CELERY_DEFAULT_EXCHANGE=settings.CELERY_DEFAULT_EXCHANGE,
    CELERY_DEFAULT_ROUTING_KEY=settings.CELERY_DEFAULT_ROUTING_KEY,
    CELERY_DEFAULT_EXCHANGE_TYPE='direct',
    CELERY_ROUTES = ('proj.celeryrouters.MainRouter', ),
    CELERY_IMPORTS=(
        'apps.qux.tasks',
        'apps.lorem.tasks',
        'apps.ipsum.tasks',
        'apps.sit.tasks'
    ),
)

      

My router class:

from django.conf import settings


class MainRouter(object):
    """
    Routes Celery tasks to a proper exchange and queue
    """
    def route_for_task(self, task, args=None, kwargs=None):
        return {
            'exchange': settings.CELERY_DEFAULT_EXCHANGE,
            'exchange_type': 'direct',
            'queue': settings.CELERY_DEFAULT_QUEUE,
            'routing_key': settings.CELERY_DEFAULT_ROUTING_KEY,
        }

      

fooproj has settings:

BROKER_URL = redis://localhost:6379/0
CELERY_DEFAULT_EXCHANGE = 'foo_exchange'
CELERY_DEFAULT_QUEUE = 'foo_queue'
CELERY_DEFAULT_ROUTING_KEY = 'foo_routing_key'

      

barproj has settings:

BROKER_URL = redis://localhost:6379/1
CELERY_DEFAULT_EXCHANGE = 'foo_exchange'
CELERY_DEFAULT_QUEUE = 'foo_queue'
CELERY_DEFAULT_ROUTING_KEY = 'foo_routing_key'

      

As you can see, both projects use their own Redis database as the broker, their own MySQL database as the database, their own exchange, queue and routing key.

I am trying to start two Celery event processes, one for each application:

celery events -A fooproj.celery_app -l info -c djcelery.snapshot.Camera
celery events -A barproj.celery_app -l info -c djcelery.snapshot.Camera

      

The problem is that both celery manufacturing processes raise challenges from all of my celery workers! So in fooproj database I can see task results from barproj database.

Any idea how to fix this problem?

+3


source to share


1 answer


From http://celery.readthedocs.org/en/latest/getting-started/brokers/redis.html :

Monitoring events (used by flower and other tools) are global and independent of the virtual host settings.

This is due to a limitation in Redis. Redis PUB / SUB channels are global and independent of the database number.



This is similar to one of Redis' caveats :(

+3


source







All Articles