Using eventlet.corolocal with celery
Does anyone have any experience with eventlet.corolocal, especially with celery (with eventlets for workers)?
If so, could you please shed some light on what is wrong with the example code below?
...
from eventlet.corolocal import local
...
ev_local = local()
@app.task
def dummy_task(self, a):
if hasattr(ev_local, a):
ev_local.a += a
else:
ev_local.a = a
print ev_local.a
if __name__ == '__main__':
app.start()
If I am now starting a celery worker with concurrency, say 5,
celery multi start 1 -A process_mss -l info -P eventlet -c 5 --verbose
and call the task (dummy_task) 20 times -
...
for i in xrange(20):
dummy_task.delay(1)
I see that 5 eventlets are handling 4 tasks each (using the return id of eventlet.corolocal.get_ident ()). But, eventlets always find ev_local without knowing the / attr 'a' variable. So the print statement in dummy_task (...) always prints 1.
Any pointers on what might go wrong?
source to share
Eventlet does not guarantee that you will get the same greenthread. In fact, every task can run in a new greenthread.
The greenthread id (eventlet.corolocal.get_ident ()) is the memory address of the Python object (greenthread I think). So when a new greenthread is used to perform a celery task, the local thread store will "disappear". If you're lucky and the celery task reuses greenthread, then the information in the local thread store will reappear.
You can get around this by using the streaming plug model.
Today I still have to solve this problem.
source to share