Ignore generator exception
Using os.walk()
to navigate the folder as follows:
for subdir, dirs, files in os.walk(path):
do something...
An exception will be thrown : UnicodeDecodeError
, I want to ignore the exception and continue and I tried this:
try:
for subdir, dirs, files in os.walk(path):
do something...
except Exception, e:
logging.exception(e)
continue # this continue is illegal
as noted in the comment, the snippet continue
in the exception is a syntax error. Is there a way to ignore the exception and keep navigating?
The exception is thrown from os.walk()
, so it is impossible to put try/except
inside for
. os.walk()
will return a python generator, how to catch an exception in it?
source to share
Update:
I originally thought the error was caused by code do something...
. Since it actually goes up os.walk
, you need to do something a little different:
walker = os.walk(path)
while True:
try:
subdir, dirs, files = next(walker)
except UnicodeDecodeError as e:
logging.exception(e)
continue
except StopIteration:
break
do something...
Basically, it takes advantage of the fact that the os.walk
generating object is returning. This allows us to name next
and thereby control the iteration at each step.
The line is subdir, dirs, files = next(walker)
trying to advance the iteration. If a is UnicodeDecodeError
up, it is registered and we move on to the next step. If an exception is thrown StopIteration
, it means that we have finished traversing the directory tree. So we break the loop.
Since it continue
must be inside the loop, you will need to move the block as well try/except
:
for subdir, dirs, files in os.walk(path):
try:
do something...
except Exception, e:
logging.exception(e)
continue # this continue is *not* illegal
Also by doing:
except Exception, e:
deprecated. ,
You should use the keyword instead as
:
except Exception as e:
While you're at it, you should replace the general Exception
with the specific UnicodeDecodeError
:
except UnicodeDecodeError as e:
You should always try to catch the most specific exception you can. Otherwise, you run the risk of accidentally catching an exception that you didn't want to handle.
source to share
I had a similar situation, iterating over links using Beautiful Soup. This is the code I wrote to do this:
class suppressed_iterator:
def __init__(self, wrapped_iter, skipped_exc = Exception):
self.wrapped_iter = wrapped_iter
self.skipped_exc = skipped_exc
def __next__(self):
while True:
try:
return next(self.wrapped_iter)
except StopIteration:
raise
except self.skipped_exc:
pass
class suppressed_generator:
def __init__(self, wrapped_obj, skipped_exc = Exception):
self.wrapped_obj = wrapped_obj
self.skipped_exc = skipped_exc
def __iter__(self):
return suppressed_iterator(iter(self.wrapped_obj), self.skipped_exc)
Example:
class IHateThirteen:
''' Throws exception while iterating on value 13 '''
def __init__(self, iterable):
self.it = iter(iterable)
def __iter__(self):
return self
def __next__(self):
v = next(self.it)
if v == 13:
raise ValueError('I hate 13!')
return v
# Outputs [10, 11, 12, 14, 15]
exception_at_thirteen = IHateThirteen([10, 11, 12, 13, 14, 15])
print(list(suppressed_generator(exception_at_thirteen)))
# Raises ValueError
exception_at_thirteen = IHateThirteen([10, 11, 12, 13, 14, 15])
print(list(exception_at_thirteen))
You can fix your code using the above code:
for subdir, dirs, files in suppressed_generator(os.walk(path)):
do something...
The higher code could be extended more to have callbacks for the omitted exception types if needed, but in this case it might be more pythonic to use iCodez .
source to share