Try-except-finally code doesn't work as expected in a streamed application

Execution abruptly stops if thread / process is killed, makes sense

Why won't it execute the cleanup code when I normally exit the main program by clicking [X] in the terminal window?

I am still looking into all sorts of multi-threaded applications, and my guess is that my problems are due to a misunderstanding of how Python handles death threads.


  • Why finally:

    won't my block run all the time?
  • When will the block not be executed yet finally:

  • What happens to code execution inside a thread when the thread is destroyed?
  • What happens to daemon / non-daemon threads when exiting the main process?


I am trying to write a multi-threaded program using ZMQ sockets that (among other things) write stuff to a log file. I want the logging thread to do some messages and cleanup unconditionally right before it dies, but it won't in most cases.

The function below starts an infinite loop on a background thread and returns a socket zmq.PAIR

for communication. The loop it starts is listening on a socket, and whatever is written to that socket is written to a file. The loop also (should) send back diagnostic messages like "I'm starting to register now!", "Sorry, an error occurred!" "I'm leaving now." therefore the main program can keep track of it.

The program main

generates multiple threads using this pattern to control / manipulate various bits and parts. It checks multiple ZMQ sockets (connected to STDIN and serial) for messages and forwards some of them to a socket connected to a file.

But now I am stuck. The routing and program control logic is main

working fine. get_logfile_sock

File handling works fine and normal exception handling works as expected. But the "I'm quitting now" code doesn't execute when the thread is killed from the main program, or when I stop the main program altogether.


def get_logfile_sock(context, file_name):
    Returns a ZMQ socket. Anything written to the socket gets appended to the a specified file. The socket will send diagnostic messages about file opening/closing and any exceptions encountered. 


    def log_file_loop(socket):
        Read characters from `socket` and write them to a file. Send back diagnostic and exception information.
            socket.send("Starting Log File {}".format(file_name))
            with open(file_name, "a+") as fh:
                # File must start with a timestamp of when it was opened
                # Write all strings/bytes to the file
                while True:
                    message = socket.recv()


                    # Un-comment this line to demonstrate that the except: and finally: blocks both get executed when there an error in the loop
                    # raise SystemExit

        except Exception as e:
            # This works fine when/if there an exception in the loop
            socket.send("::".join(['FATALERROR', e.__class__.__name__, e.message]))
            # This works fine if there an exception raised in the loop
            # Why doesn't this get executed when my program exits? Isn't that just the main program raising SystemExit? 

            # Additional cleanup code goes here
            socket.send("Closing socket to log file {}".format(file_name))

    # Make a socket pair for communication with the loop thread
    basename = os.path.basename(file_name).replace(":", "").replace(" ", "_").replace(".", "")
    SOCKNAME = 'inproc://logfile-{}'.format(basename)
    writer = context.socket(zmq.PAIR)
    reader = context.socket(zmq.PAIR)

    # Start the loop function in a separate thread
    thread = threading.Thread(target=log_file_loop, args=[writer])
    thread.daemon = True  # is this the right thing to do?

    # Return a socket endpoint to the thread
    return reader



source to share

2 answers

not executed when the thread is killed

Don't kill streams. Ask them to come out nicely and then join

at them. Consider going to Condition

for verification.

Long answer: execution kill

will cause the thread to exit without guaranteeing that it will complete any particular block, and you shouldn't expect your system to behave well after that. It's probably a little safer if used multiprocessing




How to include try: / finally: work as needed

The best practice is to create your own signal layer (which allows a lot, including sending / receiving a soft SigKILL signal).

This keeps your inter-process messaging clean and under your control.

After receiving a soft SigKILL, your thread code can handle all the necessary steps, including. by creating your own subtype of the exception (s) that makes sense according to your intended structure related to exceptions:

   # ... primary flow of a <code-block>-execution
   if ( SigINPUT == "SigKILL" ):
      raise SigKILL_EXCEPTION
except KeyboardInterrupt:
   # ... handle KeyboardInterrupt

except MemoryError:
   # ... handle MemoryError

except NotImplemented:
   # ... handle NotImplemented

   # ... handle SigKILL_EXCEPTION
   # situation-specific <code-block> shall rather be here, than in "finally:"

   # /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
   # ... handle *EXC
   # +++ ALWAYS DO THIS |||||||||||||||||||||||||||||||||||||||||||||||||||||
   # ... a common <code-block> is ALWAYS executed, under all circumstances
   # ->  put an attempt to RETURN into SigKILL_EXCEPTION section a test this 
   # +++ ALWAYS DO THIS |||||||||||||||||||||||||||||||||||||||||||||||||||||


FINALLY Demonstrator: Suggestion

def testTryFinally():
        print "TRY:"                   # show
        raise KeyboardInterrupt        # used to simulate SigKILL
    except    KeyboardInterrupt:       # EXC. to handle   SigKILL ( emulated by KBDI )
        print  "EXC.KBDI/SigKILL"                # show
        print  "EXC.KBDI:Going to RET(SigKILL)"  # remind the next instr. RET!!
        return "EXC.KBDI:RET(SigKILL)"           # execute RET <value1>
    except:                                      # EXC. collects all unhandled EXC-s
        print  "EXC.*"                           # show
    finally:                                     # FINALLY: clause
        print  "FINALLY: entered"                # show
    return     "RET(End)"                        # execute RET <value2>

>>> testTryFinally()
FINALLY: entered


How to clean up code after clicking [x] -window-frame-icon

To handle clicking the [X] -window-frame-right icon, at the top of the window, there is a good solution available in Tkinter. There it is possible to assign this event to be handled by specialized code (anEventHANDLER) that can still survive such a kiss of murder and that does all the dirty things responsibly (including taking care to gracefully release all resources) before the process dies after how it will be removed from the outside by the OS.

win.protocol( 'WM_DELETE_WINDOW', lambda:None ) # blocks this way to terminate
win.protocol( 'WM_DELETE_WINDOW', aSendSigKILL_eventHANDLER )


By creating soft signaling between processes, you can manage and send Soft-SIGs to allow / enforce execution of all distributed threads to receive the SIG message and handle their own execution accordingly.



All Articles