Problem with python function returning generator or normal object
I have defined the function f
as
def f(flag):
n = 10
if flag:
for i in range(n):
yield i
else:
return range(n)
But the f
generator returns no matter what flag
:
>>> f(True)
<generator object f at 0x0000000003C5EEA0>
>>> f(False)
<generator object f at 0x0000000007AC4828>
And if I iterate over the returned object:
# prints normally
for i in f(True):
print(i)
# doesn't print
for i in f(False):
print(i)
It looks like it is f(False)
returning a generator that has been iterated over. What reason? Thank.
source to share
A function containing an yield
always operator returns a generator object.
Only when you iterate over this generator object will the code in the function be executed. Until that time, the code in the function is not executed, and Python cannot know that you will just return.
Note that use return
in a generator function has different semantics than in a regular function; return
in this case it is simply regarded as "generator output here"; the return value is discarded because the generator can only generate values using expressions yield
.
It sounds like you want to use instead yield from
:
def f(flag):
n = 10
if flag:
for i in range(n):
yield i
else:
yield from range(n)
yield from
requires Python 3.3 or higher.
See the documentation yield
:
Using an expression
yield
in a function body calls this function as a generator.When a generator function is called, it returns an iterator known as a generator. This generator then controls the execution of the generator function. Execution starts when one of the generator methods is called. At this time, execution moves to the first expression
yield
, where it pauses again, returning the value of expression_list to the calling generator.
Iterating over the generator calls the method generator.__next__()
that starts the execution.
If you want to return a generator for a while, then don't use it yield
in this function. You manufacture the generator in other ways; for example using a standalone function or using a generator expression it is possible:
def f(flag):
n = 10
if flag:
return (i for i in range(n))
else:
return range(n)
Now no is yield
used in f
, and it will no longer create a generator object directly. Instead, the generator expression (i for i in range(n))
produces it, but only conditionally.
source to share
You can get around this by using a nested function that actually uses yield
:
def f(flag):
def gen():
for i in range(n):
yield i
n = 10
if flag:
return gen()
else:
return range(n)
>>> f(True)
<generator object gen at 0x7f62017e3730>
>>> f(False)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
As Martijn points out, any containing function yield
will always return a generator object, so if in some cases you want the body to f
actually execute on call f()
, not just execute on iteration, you should use this approach.
The standard library instance method map
from concurrent.Futures.ProcessPoolExecutor
/ concurrent.Futures.ThreadPoolExecutor
uses this to ensure that futures are dispatched as soon as it map
is called, not just when you try to fetch results from it, e.g .:
def map(self, fn, *iterables, timeout=None):
if timeout is not None:
end_time = timeout + time.time()
fs = [self.submit(fn, *args) for args in zip(*iterables)]
# Yield must be hidden in closure so that the futures are submitted
# before the first iterator value is required.
def result_iterator():
try:
for future in fs:
if timeout is None:
yield future.result()
else:
yield future.result(end_time - time.time())
finally:
for future in fs:
future.cancel()
return result_iterator()
source to share