Threading in Django doesn't work in production
I have a function in my Django views.py that looks like this.
def process(request):
form = ProcessForm(request.POST, request.FILES)
if form.is_valid():
instance = form.save(commit=False)
instance.requested_by = request.user
instance.save()
t = threading.Thread(target=utils.background_match, args=(instance,), kwargs={})
t.setDaemon(True)
t.start()
return HttpResponseRedirect(reverse('mart:processing'))
Here I am trying to call the 'background_match' function on a separate thread when submitting a ProcessForm. Since this flow takes some time, I redirect the user to another page named "mart: processing".
The problem I'm running into is that it all works fine on my local machine, but doesn't work on the production server, which is an AWS EC2 instance. The thread doesn't start at all. There's a for loop inside the background_match function that doesn't move forward.
However, if I refresh (CTRL + R) the "mart: processing" page, it moves 1 or 2 iterations. So, for a full loop of 1000 iterations to run, I need to refresh the page 1000 times. If after, say, 100 iterations, I don't refresh the page, it gets stuck at that point and doesn't go to the 101st iteration. Please, help!
source to share
Wrong architecture. Django and other web applications should spawn such themes. The correct way is to create an asynchronous task using the task queue. The most popular task for django is Celery .
The page then has mart:processing
to check the async result to determine if the task was completed. A rough sketch looks like this.
from celery.result import AsynResult
from myapp.tasks import my_task
...
if form.is_valid():
...
task_id = my_task()
request.session['task_id']=task_id
return HttpResponseRedirect(reverse('mart:processing'))
...
On the next page
task_id = request.session.get('task_id')
if task_id:
task = AsyncResult(task_id)
source to share