How do I handle dynamic reloads when using Django?

EDIT: Added simplified example code, explained the current requests / completion better, and added the gist from CrazyPython.

At first glance, you have 4 problems to solve. I'll take them in turn, and then give some sample code that should help clarify:

SIGTERM processing

It's simple. You just need to set up a signal handler to mark what you need to close. PMOTW has a good set of examples of how to catch a signal. You can use variations of this code to catch the SIGTERM and set a global flag that says you are terminating.

Rejecting new requests

Django middleware provides a convenient way to connect any HTTP request to your application. You can create a simple hook process_request()

that returns an error page if the global flag is set (top).

Completing existing requests

When terminating any new requests, you will now need to terminate your current requests. While you may not believe this right now, it means that you just do nothing and let the program just continue to run as usual after SIGTERM. Let me expand on this ...

The contract with heroku is that you must fill the SIGTERM within 10 seconds, otherwise it will send a SIGKILL. This means that there is nothing you can do (like a well managed application) to ensure that all requests are always completed. Consider two cases:

• Your application processes all existing requests within 10 seconds. In this case, you will leave your program without the ability to complete requests. No special code is required to run requests - all threads / processes are already doing what you need!
• For some requests, your application takes more than 10 seconds. In this case, there is nothing you can do - it will be terminated with maximum power to the hero until the completion of the long request. If you think you can ignore SIGKILL, think about it ... It's not allowed - see the documentation.

In both cases, the solution is to let your program continue to run so that as many current requests are executed before completion.

Shutting down the application

The simplest task might be to wait for SIGKILL to arrive with the hero after 10 seconds. It's not elegant, but it should be fine because you are rejecting any new requests.

If that's not good enough, you need to track your outstanding requests and use that to decide when you can close the application. The exact way to close the application will depend on what is hosting it, so I cannot give you an exact guide. Hopefully the example code gives you enough pointer.

Sample code

Starting with the example signal handler in PMOTW, I augmented the code to add some thread handling requests and a completion dispatcher to catch the signal and let the application terminate gracefully. You have to run this in Python2.7 and then try to kill the process.

Based on this example, CrazyPython created this gist to give a concrete implementation in django.

import signal
import os
import time
import threading
import random


class TerminationManager(object):

    def __init__(self):
        self._running = True
        self._requests = 0
        self._lock = threading.Lock()
        signal.signal(signal.SIGTERM, self._start_shutdown)

    def _start_shutdown(self, signum, stack):
        print 'Received:', signum
        self._running = False

    def start_request(self):
        with self._lock:
            self._requests += 1

    def stop_request(self):
        with self._lock:
            self._requests -= 1

    def is_running(self):
        return self._running or self._requests > 0

    def running_requests(self):
        return self._requests


class DummyWorker(threading.Thread):

    def __init__(self, app_manager):
        super(DummyWorker, self).__init__()
        self._manager = app_manager

    def run(self):
        while self._manager.is_running():
            # Emulate random work and delay between requests.
            if random.random() > 0.9:
                self._manager.start_request()
                time.sleep(random.randint(1, 3))
                self._manager.stop_request()
            else:
                time.sleep(1)
        print "Stopping worker"


manager = TerminationManager()
print 'My PID is:', os.getpid()

for _ in xrange(10):
    t = DummyWorker(manager)
    t.start()

while manager.is_running():
    print 'Waiting with {} running requests'.format(manager.running_requests())
    time.sleep(5)

print 'All done!'