How to clean up after an incorrectly closed file object in the Python standard library (after an exception)

TL; DR: The standard library cannot close the file when an exception is thrown. I am looking for the best way to handle this situation. Feel free to read from the paragraph starting with "Closer Look at the CPython Source Code". Also scroll down to the bottom of the question to get a stand-alone script that reproduces this issue on Windows.

I am writing a Python package in which I use STL ConfigParser

(2.x) or ConfigParser

(3.x) to parse a custom config file (I will refer to both ConfigParser

since the problem is mostly with the 2.x implementation). From now on, my respective lines of code on GitHub will be linked whenever possible.

(used in my code here ) throws an exception ConfigParser.Error

when the config file is malformed. I have some code in my test suite targeting this situation using unittest.TestCase.assertRaises(ConfigParser.Error)

. Invalid config file generated correctly with tempfile.mkstemp

(returned fd closed with os.close

before anything) and I am trying to dodelete the temporary file with os.remove



the problem begins. My tests fail on Windows (while running both OS X and Ubuntu) with Python 2.7 (see this AppVeyor build ):

Traceback (most recent call last):
  File "C:\projects\storyboard\tests\", line 235, in test_option_reader
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\storyboard-test-3clysg.conf'


Note that, as I said above, is malformed_conf_file

generated with tempfile.mkstemp

and immediately closed with os.close

, so the only time it opens is when I call[malformed_conf_file])

here inside the context unittest.TestCase.assertRaises(ConfigParser.Error)

. So the culprit seems to be the STL and not my own code.

Upon closer inspection of the CPython source code, I found that I really didn't close the file properly when an exception was thrown . The method from 2.7 ( here on CPython Mercurial ) has the following lines:


for filename in filenames:
        fp = open(filename)
    except IOError:
    self._read(fp, filename)


An exception (if any) is thrown self._read(fp, filename)

, but as you can see, if it self._read

does, it fp

will not be closed, as it fp.close()

is only thrown after returning self._read


Meanwhile, the method read

from 3.4 ( here ) does not suffer from the same problem, as this time they correctly built file handling into context:

for filename in filenames:
        with open(filename, encoding=encoding) as fp:
            self._read(fp, filename)
    except OSError:


So, I think it's pretty clear that the problem is a defect in the 2.7 STL. And what is the best way to handle this situation? In particular:

  • Is there anything I can do on my end to close this file?
  • Should I report bugs to

For now, I'm guessing I'll just add try .. except OSError ..

to this os.remove

(any suggestions?).

Update: A stand-alone script that can be used to reproduce this issue on Windows:

#!/usr/bin/env python2.7
import ConfigParser
import os
import tempfile

def main():
    fd, path = tempfile.mkstemp()
    with open(path, 'w') as f:
    config = ConfigParser.ConfigParser()
    except ConfigParser.Error:

if __name__ == '__main__':


When I run it using Python 2.7 interpreter:

Traceback (most recent call last):
  File ".\", line 19, in <module>
  File ".\", line 16, in main
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\redacted\\appdata\\local\\temp\\tmp07yq2v'



source to share

1 answer

This is an interesting problem. As Lukas Graf pointed out in a comment, the problem is that the exception traceback object contains a reference to the call frame in which the exception was thrown. This call frame includes local variables that existed at the time, one of which is a link to an open file. So the object file still has a reference to it and is not closed properly.

For your stand-alone example, just remove try/except ConfigParser.Error

"works": Exception from file with invalid formats doesn't show up and stops the program. However, it catches an assertRaises

exception in your actual application to check that it is being tested. I'm not 100% sure why the trace persists even after a block with assertRaises

, but it seems to do it.

For your example, another promising solution is to change pass

in your proposal except

to sys.exc_clear()


except ConfigParser.Error:


This will save you the dull trace object and allow you to close the file.

However, it is not clear how to do this in your real application, because the offending sentence except

is inside unittest

. I think the easiest is not to use assertRaises

directly. Instead, write a helper function that runs your test, check for the exception you want, clean up the trick sys.exc_clear()

, and then add another custom exception. Then, complete the call to this helper method in assertRaises

. This way you gain control over the problematic exception thrown by ConfigParser and can properly clean it up (which unittest

it doesn't).

Here's a sketch of what I mean:

# in your test method
assertRaises(CleanedUpConfigError, helperMethod, conf_file, malformed_conf_file)

# helper method that you add to your test case class
def helperMethod(self, conf_file, malformed_conf_file):
     gotRightError = False
          or6 = OptionReader(
               config_files=[conf_file, malformed_conf_file],
     except ConfigParser.Error:
          gotRightError = True
     if gotRightError:
          raise CleanedUpConfigError("Config error was raised and cleaned up!")


I haven't tested this of course, because I don't have the whole setup tuned to your code. You may need to tweak it a bit. (Think about it, you might not even need exc_clear()

it if you do this, because since the exception handler is now in a separate function, the traceback should be properly cleaned up when it helperMethod

exits.) However, I think this idea might lead you somewhere. Basically you need to make sure that the sentence except

that catches this particular ConfigParser.Error

one is written by you so that it can be cleaned up before attempting to delete the test file.

Addendum: It seems that if the context manager handles the exception, the traceback actually persists until the function containing the block with

ends, as shown in this example:

class CM(object):
    def __init__(self):

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, tb):
        return True

def foo():
    with CM():
        raise ValueError


Even though the block has with

completed by the time print

it was thrown, so exception handling must be completed, sys.exc_info

it still returns information about the exception as if there was an active exception. This is what happens in your code too: the block with assertRaises

causes the trace to persist until the end of that function, interfering with yours os.remove

. This seems like buggy behavior and I notice that it no longer works that way in Python 3 ( print

prints (None, None None)

), so I think it was a wart fixed with Python 3.

Based on this, I suspect it might be enough to just insert sys.exc_clear()

right before yours os.remove

(after the block with assertRaises




All Articles