How to clean up after an incorrectly closed file object in the Python standard library (after an exception)
TL; DR: The standard library cannot close the file when an exception is thrown. I am looking for the best way to handle this situation. Feel free to read from the paragraph starting with "Closer Look at the CPython Source Code". Also scroll down to the bottom of the question to get a stand-alone script that reproduces this issue on Windows.
I am writing a Python package in which I use STL
(3.x) to parse a custom config file (I will refer to both
since the problem is mostly with the 2.x implementation). From now on, my respective lines of code on GitHub will be linked whenever possible.
(used in my code here ) throws an exception
when the config file is malformed. I have some code in my test suite targeting this situation using
. Invalid config file generated correctly with
(returned fd closed with
before anything) and I am trying to dodelete the temporary file with
the problem begins. My tests fail on Windows (while running both OS X and Ubuntu) with Python 2.7 (see this AppVeyor build ):
Traceback (most recent call last): File "C:\projects\storyboard\tests\test_util.py", line 235, in test_option_reader os.remove(malformed_conf_file) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\storyboard-test-3clysg.conf'
Note that, as I said above, is
and immediately closed with
, so the only time it opens is when I call
here inside the context
. So the culprit seems to be the STL and not my own code.
Upon closer inspection of the CPython source code, I found that I really didn't close the file properly when an exception was thrown . The method from 2.7 ( here on CPython Mercurial ) has the following lines:
for filename in filenames: try: fp = open(filename) except IOError: continue self._read(fp, filename) fp.close() read_ok.append(filename)
An exception (if any) is thrown
, but as you can see, if it
will not be closed, as it
is only thrown after returning
Meanwhile, the method
from 3.4 ( here ) does not suffer from the same problem, as this time they correctly built file handling into context:
for filename in filenames: try: with open(filename, encoding=encoding) as fp: self._read(fp, filename) except OSError: continue read_ok.append(filename)
So, I think it's pretty clear that the problem is a defect in the 2.7 STL. And what is the best way to handle this situation? In particular:
- Is there anything I can do on my end to close this file?
- Should I report bugs to bugs.python.org?
For now, I'm guessing I'll just add
try .. except OSError ..
Update: A stand-alone script that can be used to reproduce this issue on Windows:
#!/usr/bin/env python2.7 import ConfigParser import os import tempfile def main(): fd, path = tempfile.mkstemp() os.close(fd) with open(path, 'w') as f: f.write("malformed\n") config = ConfigParser.ConfigParser() try: config.read(path) except ConfigParser.Error: pass os.remove(path) if __name__ == '__main__': main()
When I run it using Python 2.7 interpreter:
Traceback (most recent call last): File ".\example.py", line 19, in <module> main() File ".\example.py", line 16, in main os.remove(path) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\redacted\\appdata\\local\\temp\\tmp07yq2v'
source to share
This is an interesting problem. As Lukas Graf pointed out in a comment, the problem is that the exception traceback object contains a reference to the call frame in which the exception was thrown. This call frame includes local variables that existed at the time, one of which is a link to an open file. So the object file still has a reference to it and is not closed properly.
For your stand-alone example, just remove
"works": Exception from file with invalid formats doesn't show up and stops the program. However, it catches an
exception in your actual application to check that it is being tested. I'm not 100% sure why the trace persists even after a block
, but it seems to do it.
For your example, another promising solution is to change
in your proposal
try: config.read(path) except ConfigParser.Error: sys.exc_clear()
This will save you the dull trace object and allow you to close the file.
However, it is not clear how to do this in your real application, because the offending sentence
. I think the easiest is not to use
directly. Instead, write a helper function that runs your test, check for the exception you want, clean up the trick
, and then add another custom exception. Then, complete the call to this helper method in
. This way you gain control over the problematic exception thrown by ConfigParser and can properly clean it up (which
Here's a sketch of what I mean:
# in your test method assertRaises(CleanedUpConfigError, helperMethod, conf_file, malformed_conf_file) # helper method that you add to your test case class def helperMethod(self, conf_file, malformed_conf_file): gotRightError = False try: or6 = OptionReader( config_files=[conf_file, malformed_conf_file], section='sec', ) except ConfigParser.Error: gotRightError = True sys.exc_clear() if gotRightError: raise CleanedUpConfigError("Config error was raised and cleaned up!")
I haven't tested this of course, because I don't have the whole setup tuned to your code. You may need to tweak it a bit. (Think about it, you might not even need
it if you do this, because since the exception handler is now in a separate function, the traceback should be properly cleaned up when it
exits.) However, I think this idea might lead you somewhere. Basically you need to make sure that the sentence
that catches this particular
one is written by you so that it can be cleaned up before attempting to delete the test file.
Addendum: It seems that if the context manager handles the exception, the traceback actually persists until the function containing the block
ends, as shown in this example:
class CM(object): def __init__(self): pass def __enter__(self): return self def __exit__(self, exc_type, exc_value, tb): return True def foo(): with CM(): raise ValueError print(sys.exc_info())
Even though the block has
completed by the time
it was thrown, so exception handling must be completed,
it still returns information about the exception as if there was an active exception. This is what happens in your code too: the block
causes the trace to persist until the end of that function, interfering with yours
. This seems like buggy behavior and I notice that it no longer works that way in Python 3 (
(None, None None)
), so I think it was a wart fixed with Python 3.
Based on this, I suspect it might be enough to just insert
right before yours
(after the block
source to share