Prev: Advanced Dictionary
Next: Wing IDE 3.2.9 released: Fixes debugger for Python 2.4.x and earlier
From: Jérôme Mainka on 16 Jun 2010 08:03
Hello,

I try to experiment with coroutines and I don't understand why this
snippet doesn't work as expected... In python 2.5 and python 2.6 I get
the following output:

0
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<generator object at 0x7e43f8> ignored

The TypeError exception comes from the pprint instruction...

If i replace the main part with:

==
p1 = dump()
p2 = sort(p1)
for item in my_list: p2.send(item)
==

it works as expected.

I don't understand what is goind wrong. Has someone an explanation for
this issue?

Thanks,

Jérôme


===
from functools import wraps
from pprint import pprint
import random

def coroutine(f):
@wraps(f)
def start(*args, **kwargs):
res = f(*args, **kwargs)
res.next()
return res
return start

@coroutine
def sort(target):
l = []

try:
while True:
l.append((yield))
except GeneratorExit:
l.sort()
for item in l:
target.send(item)

@coroutine
def dump():
while True:
pprint((yield))

if __name__ == "__main__":
my_list = range(100)
random.shuffle(my_list)

p = sort(dump())

for item in my_list:
p.send(item)
From: Thomas Jollans on 16 Jun 2010 08:35
On 06/16/2010 02:03 PM, J�r�me Mainka wrote:
> Hello,
>
> I try to experiment with coroutines and I don't understand why this
> snippet doesn't work as expected... In python 2.5 and python 2.6 I get
> the following output:
>
> 0
> Exception exceptions.TypeError: "'NoneType' object is not callable" in
> <generator object at 0x7e43f8> ignored
>
> The TypeError exception comes from the pprint instruction...
>
> If i replace the main part with:
>
> ==
> p1 = dump()
> p2 = sort(p1)
> for item in my_list: p2.send(item)
> ==
>
> it works as expected.

What a strange problem. I hope somebody else can shed more light on what
is going on

Anyway, it appears (to me) to be a scoping issue: if you move
"from pprint import pprint" into the dump() function body, it works. I
don't understand why this changes anything: pprint was in the global
namespace all along, and it's present and functioning exactly *once*,
after which it's None. It's still there, otherwise there'd be a
NameError, but it was set to None without any line of code doing this
explicitly as far as I can see.

Another VERY strange thing is, of course, your "fix": renaming "p" to
"p2" helps. "p", "p1", "p3", and "yyy" are just some examples of
variable names that don't work.

As for Python versions: I changed the code to work with Python 3 and get
the same odd behaviour.

>
> I don't understand what is goind wrong. Has someone an explanation for
> this issue?
>
> Thanks,
>
> J�r�me
>
>
> ===
> from functools import wraps
> from pprint import pprint
> import random
>
> def coroutine(f):
> @wraps(f)
> def start(*args, **kwargs):
> res = f(*args, **kwargs)
> res.next()
> return res
> return start
>
> @coroutine
> def sort(target):
> l = []
>
> try:
> while True:
> l.append((yield))
> except GeneratorExit:
> l.sort()
> for item in l:
> target.send(item)
>
> @coroutine
> def dump():
> while True:
> pprint((yield))
>
> if __name__ == "__main__":
> my_list = range(100)
> random.shuffle(my_list)
>
> p = sort(dump())
>
> for item in my_list:
> p.send(item)

From: Steven D'Aprano on 16 Jun 2010 12:35
On Wed, 16 Jun 2010 05:03:13 -0700, Jérôme Mainka wrote:

> Hello,
>
> I try to experiment with coroutines and I don't understand why this
> snippet doesn't work as expected... In python 2.5 and python 2.6 I get
> the following output:
>
> 0
> Exception exceptions.TypeError: "'NoneType' object is not callable" in
> <generator object at 0x7e43f8> ignored

I changed the while loop in dump() to print the value of pprint:

while True:
print type(pprint), pprint
pprint((yield))


and sure enough I get this output:

[steve(a)sylar ~]$ python2.6 test_coroutine.py
<type 'function'> <function pprint at 0xb7edc534>
0
<type 'NoneType'> None
Exception TypeError: "'NoneType' object is not callable" in <generator
object sort at 0xb7ee5c84> ignored


BUT then I wrapped the __main__ code in a function:


def f():
my_list = range(100)
random.shuffle(my_list)
p = sort(dump())
for item in my_list:
p.send(item)

if __name__ == "__main__":
f()


and the problem went away:

[steve(a)sylar ~]$ python2.6 test_coroutine.py
<type 'function'> <function pprint at 0xb7eb44fc>
0
<type 'function'> <function pprint at 0xb7eb44fc>
1
<type 'function'> <function pprint at 0xb7eb44fc>
2
[...]
<type 'function'> <function pprint at 0xb7eb44fc>
99
<type 'function'> <function pprint at 0xb7eb44fc>


How bizarre is that?


I have to say that your code is horribly opaque and unclear to me. Maybe
that's just because I'm not familiar with coroutines, but trying to
follow the program flow is giving me a headache. It's like being back in
1977 trying to follow GOTOs around the code, only without line numbers.
But if I have understood it correctly, I think the generator function
coroutine(sort) is being resumed by both the send() method and the next()
method. I draw your attention to this comment from the PEP that
introduced coroutines:


[quote]
Because yield will often be returning None, you should always check for
this case. Don't just use its value in expressions unless you're sure
that the send() method will be the only method used resume your generator
function.
[end quote]

http://docs.python.org/release/2.5/whatsnew/pep-342.html


So I wonder if that has something to do with it?



--
Steven
From: Jérôme Mainka on 16 Jun 2010 12:51
On Jun 16, 6:35 pm, Steven D'Aprano <st...(a)REMOVE-THIS-
cybersource.com.au> wrote:

> How bizarre is that?
Sure...


> I have to say that your code is horribly opaque and unclear to me.
Welcome to the coroutines world :-)

This is mainly a pipeline where each function suspends execution
waiting for data (yield), and feeding other functions (the sort
function's target argument) with data (.send())

-> sort() -> dump()



> Because yield will often be returning None, you should always check for
> this case. Don't just use its value in expressions unless you're sure
> that the send() method will be the only method used resume your generator
> function.

This remark deals with constructions like:

value = (yield i)

For an excellent introduction to coroutines in python, I recommend:

http://www.dabeaz.com/coroutines/


Thanks,

Jérôme
From: Thomas Jollans on 16 Jun 2010 13:25
On 06/16/2010 06:35 PM, Steven D'Aprano wrote:
> On Wed, 16 Jun 2010 05:03:13 -0700, Jérôme Mainka wrote:
>
>> Hello,
>>
>> I try to experiment with coroutines and I don't understand why this
>> snippet doesn't work as expected... In python 2.5 and python 2.6 I get
>> the following output:
>>
>> 0
>> Exception exceptions.TypeError: "'NoneType' object is not callable" in
>> <generator object at 0x7e43f8> ignored

Heureka! I think I've found the answer.

Have a look at this:

###
from itertools import izip

def gen():
print globals()
try:
while True:
yield 'things'
except GeneratorExit:
print globals()

if __name__ == "__main__":
g = gen()
g.next()
###
% python genexit.py
{'izip': <type 'itertools.izip'>, 'g': <generator object at
0x7f76c7340878>, '__builtins__': <module '__builtin__' (built-in)>,
'__file__': 'genexit.py', 'gen': <function gen at 0x7f76c7327848>,
'__name__': '__main__', '__doc__': None}
{'izip': None, 'g': None, '__builtins__': <module '__builtin__'
(built-in)>, '__file__': 'genexit.py', 'gen': <function gen at
0x7f76c7327848>, '__name__': '__main__', '__doc__': None}
###

Note that izip (the thing I imported, could just as well have been
pprint) is None the second time I print globals(). why?

This second print happens when GeneratorExit is raised. Just like all
occurences of dump.send(..) in the OP's code are underneath
"except GeneratorExit:"
When does GeneratorExit happen? Simple: When the generator is destroyed.
And when is it destroyed? *looks at the end of the file*
Since it's a global variable, it's destroyed when the module is
destroyed. Python sees the end of the script, and del's the module,
which means all the items in the module's __dict__ are del'd one-by-one
and, apparently, to avoide causing havoc and mayhem, replaced by None.
Fair enough.

So why did the "solutions" we found work?

(1) changing the variable name:
I expect Python del's the globals in some particular order, and this
order is probably influenced by the key in the globals() dict. One key
happens to be deleted before pprint (then it works, pprint is still
there), another one happens to come after pprint. For all I know, it
might not even be deterministic.

(2) moving pprint into the dump() function
As long as there's a reference to the dump generator around, there will
be a reference to pprint around. sort() has a local reference to
target == dump(), so no problem there.

(3) wrapping the code in a function
As the generator is now a local object, it gets destroyed with the
locals, when the function exits. The module gets destroyed afterwards.


What should you do to fix it then?

If you really want to keep the except GeneratorExit: approach, make sure
you exit it manually. Though really, you should do something like
p.send(None) at the end, and check for that in the generator: recieving
None would mean "we're done here, do post processing!"

-- Thomas
 |  Next  |  Last
Pages: 1 2
Prev: Advanced Dictionary
Next: Wing IDE 3.2.9 released: Fixes debugger for Python 2.4.x and earlier