Exception not caught mixing contextmanager with decorator
I have been struggling with this problem for a long time. In some code that I am writing, I need to write a bunch of files and create a directory tree as needed. My idea is this: catch exception IOError, and if its first argument is ENOENT, create a directory structure and try to write the file again.
I wrote a relatively small repeat function, but I would like to generalize it to "any" code that might throw an exception. It all worked until I came across something like this:
def retry(f):
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except:
print "Gotcha here!"
return wrapper
def update(file, value):
@contextmanager
@retry
def safeopen(file, mode):
with open(file, mode) as f:
yield f
try:
with safeopen(file, 'w') as f:
f.write(value)
except:
print "Gotcha there!"
update( 'tests/nonexisting/dummy.txt', 'Dummy line')
I've hardened the code to the bare minimum to show what fails when open()
throwing an exception. In this code, the exception is only caught from the except block in update()
and not in wrapper()
, so I always get Gotcha there!
, although I expected it Gotcha here
instead. I tried to replace the @decorator and @contextmanager lines, no way. I checked and made sure the wrapper is called: it is. Just that he doesn't exclude from f()
.
What am I doing wrong?
The problem is you are mixing decorators @contextmanager
with normal functions. The decorator @retry
is a normal function, but you use it to decorate the generator @contextmanager
- it won't behave the way you expect, because when you call a function @contextmanager
, its function body isn In fact, it gets executed. An object is returned instead GeneratorContextManager
. The function body is not executed until the method is called __enter__
GeneratorContextManager
, either directly or using a statement with
.
Consider the following example:
from contextlib import contextmanager
def retry(f):
def wrapper(*args, **kwargs):
try:
print("in wrapper")
return f(*args, **kwargs)
except:
print "Gotcha here!"
finally:
print "done"
return wrapper
@contextmanager
@retry
def safeopen(file, mode):
print("in safe open")
with open(file, mode) as f:
yield f
def update(file, value):
try:
print("CALLING SAFE OPEN")
with safeopen(file, 'w') as f:
f.write(value)
except:
print "Gotcha there!"
update( 'tests/nonexisting/dummy.txt', 'Dummy line')
It outputs:
CALLING SAFE OPEN
in wrapper
done
in safe open
Gotcha there!
As you can see, we exit the shell retry
before we ever enter the body safeopen
, because it safeopen
is the context manager. This only happens until the object GeneratorContextManager
is actually returned and evaluated as part of an instruction with
executed by the body, but by then it is too late; retry
released.
To fix this, you need to make retry
a @contextmanager
and use it to decorate the context manager safeopen
:
from contextlib import contextmanager
def retry(f):
@contextmanager
def wrapper(*args, **kwargs):
try:
print("in wrapper")
with f(*args, **kwargs) as out:
yield out
except:
print "Gotcha here!"
finally:
print "done"
return wrapper
@retry
@contextmanager
def safeopen(file, mode):
print("in safe open")
with open(file, mode) as f:
yield f
def update(file, value):
print("CALLING SAFE OPEN")
with safeopen(file, 'w') as f:
f.write(value)
update( 'tests/nonexisting/dummy.txt', 'Dummy line')
Output:
CALLING SAFE OPEN
in wrapper
in safe open
Gotcha here!
done
Edit:
If you reverse the ordering of decorators to retry
decorate safeopen
directly, you can make the implementation a retry
little simpler, since you are now decorating the generator function, not the context manager
def retry(f):
def wrapper(*args, **kwargs):
try:
print("in wrapper")
return next(f(*args, **kwargs)) # Call next on the generator object
except:
print "Gotcha here!"
finally:
print "done"
return wrapper
@contextmanager
@retry
def safeopen(file, mode):
print("in safe open")
with open(file, mode) as f:
yield f
source to share