Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: Lie Ryan on 13 Dec 2009 21:50 On 12/14/2009 9:45 AM, exarkun(a)twistedmatrix.com wrote: > On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >> On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>> StopIteration is intended to be used only within the .__next__ >>>> method of >>>> iterators. The devs know that other 'off-label' use results in the >>>> inconsistency you noted, but their and my view is 'don't do that'. >>> >>> Which is unfortunate, because it's not that hard to get StopIteration >>> without explicitly raising it yourself and this behavior makes it >>> difficult to debug such situations. >> >> I can't think of any way to get StopIteration without explicitly raising >> it yourself. It's not like built-ins or common data structures routinely >> raise StopIteration. I don't think I've *ever* seen a StopIteration that >> I didn't raise myself. > > Call next on an iterator. For example: iter(()).next() ..next() is not meant to be called directly, that's why it's renamed to ..__next__() in python 3. Just like .__add__() is never meant to be called directly since you'll then have to handle NotImplemented. If you really need to call .__next__() you will call next() builtin function instead which has a second argument to return a sentinel value instead of StopIteration. IMNSHO next()'s sentinel should always be specified except if you're implementing __next__() or if the sequence is an infinite iterator. >>> What's with this view, exactly? Is it just that it's hard to implement >>> the more desirable behavior? >> >> What is that "more desirable behaviour"? That StopIteration is used to >> signal that Python should stop iterating except when you want it to be >> ignored? Unfortunately, yes, it's quite hard to implement "do what the >> caller actually wants, not what he asked for" behaviour -- and even if it >> were possible, it goes against the grain of the Zen of Python. >> >> If you've ever had to debug faulty "Do What I Mean" software, you'd see >> this as a good thing. > > I have plenty of experience developing and debugging software, Steven. > Your argument is specious, as it presupposes that only two possibilities > exist: the current behavior of some kind of magical faerie land behavior. > > I'm surprised to hear you say that the magical faerie land behavior > isn't desirable either, though. I'd love a tool that did what I wanted, > not what I asked. The only serious argument against this, I think, is > that it is beyond our current ability to create (and so anyone claiming > to be able to do it is probably mistaken). In your world, this is what happens: >>> list = [a, b, c] >>> # print list >>> print list ["a", "b", "c"] >>> # make a copy of list >>> alist = list(llst) # oops a mistype >>> alist = alist - "]" + ", "d"]" >>> print alist ["a", "b", "c", "d"] >>> alist[:6] + "i", + alist[6:] >>> print alist ["a", "i", "b", "c", "d"] >>> print alist >>> # hearing the sound of my deskjet printer... >>> C:\fikle.text.write(alist) >>> print open("C:\file.txt").read() <h1>a</h1> <ul> <li>i</li> <li>b</li> <li>c d</li> >>> # great, exactly what I needed > You chopped out all the sections of this thread which discussed the more > desirable behavior. You can go back and read them in earlier messages if > you need to be reminded. I'm not talking about anything beyond what's > already been raised. > > I'm pretty sure I know the answer to my question, though - it's hard to > implement, so it's not implemented. > > Jean-Paul
From: exarkun on 13 Dec 2009 22:29 On 02:50 am, lie.1296(a)gmail.com wrote: >On 12/14/2009 9:45 AM, exarkun(a)twistedmatrix.com wrote: >>On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>>>StopIteration is intended to be used only within the .__next__ >>>>>method of >>>>>iterators. The devs know that other 'off-label' use results in the >>>>>inconsistency you noted, but their and my view is 'don't do that'. >>>> >>>>Which is unfortunate, because it's not that hard to get >>>>StopIteration >>>>without explicitly raising it yourself and this behavior makes it >>>>difficult to debug such situations. >>> >>>I can't think of any way to get StopIteration without explicitly >>>raising >>>it yourself. It's not like built-ins or common data structures >>>routinely >>>raise StopIteration. I don't think I've *ever* seen a StopIteration >>>that >>>I didn't raise myself. >> >>Call next on an iterator. For example: iter(()).next() > >.next() is not meant to be called directly Doesn't matter. Sometimes it makes sense to call it directly. And I was just giving an example of a way to get StopIteration raised without doing it yourself - which is what Steve said he couldn't think of. >> >>I'm surprised to hear you say that the magical faerie land behavior >>isn't desirable either, though. I'd love a tool that did what I >>wanted, >>not what I asked. The only serious argument against this, I think, is >>that it is beyond our current ability to create (and so anyone >>claiming >>to be able to do it is probably mistaken). > >In your world, this is what happens: > >>> list = [a, b, c] > >>> # print list > >>> print list >["a", "b", "c"] > >>> # make a copy of list > >>> alist = list(llst) # oops a mistype > >>> alist = alist - "]" + ", "d"]" > >>> print alist >["a", "b", "c", "d"] > >>> alist[:6] + "i", + alist[6:] > >>> print alist >["a", "i", "b", "c", "d"] > >>> print alist > >>> # hearing the sound of my deskjet printer... > >>> C:\fikle.text.write(alist) > >>> print open("C:\file.txt").read() ><h1>a</h1> ><ul> ><li>i</li> ><li>b</li> ><li>c d</li> > >>> # great, exactly what I needed I don't understand the point of this code listing, sorry. I suspect you didn't completely understand the magical faerie land I was describing - where all your programs would work, no matter what mistakes you made while writing them. Jean-Paul
From: Steven D'Aprano on 13 Dec 2009 23:11 On Sun, 13 Dec 2009 22:45:58 +0000, exarkun wrote: > On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>>StopIteration is intended to be used only within the .__next__ method >>>>of >>>>iterators. The devs know that other 'off-label' use results in the >>>>inconsistency you noted, but their and my view is 'don't do that'. >>> >>>Which is unfortunate, because it's not that hard to get StopIteration >>>without explicitly raising it yourself and this behavior makes it >>>difficult to debug such situations. >> >>I can't think of any way to get StopIteration without explicitly raising >>it yourself. It's not like built-ins or common data structures routinely >>raise StopIteration. I don't think I've *ever* seen a StopIteration that >>I didn't raise myself. > > Call next on an iterator. For example: iter(()).next() Or in more recent versions of Python, next(iter(())). Good example. But next() is a special case, and since next() is documented as raising StopIteration if you call it and it raises StopIteration, you have raised it yourself. Just not explicitly. >>>What's with this view, exactly? Is it just that it's hard to implement >>>the more desirable behavior? >> >>What is that "more desirable behaviour"? That StopIteration is used to >>signal that Python should stop iterating except when you want it to be >>ignored? Unfortunately, yes, it's quite hard to implement "do what the >>caller actually wants, not what he asked for" behaviour -- and even if >>it were possible, it goes against the grain of the Zen of Python. >> >>If you've ever had to debug faulty "Do What I Mean" software, you'd see >>this as a good thing. > > I have plenty of experience developing and debugging software, Steven. > Your argument is specious, as it presupposes that only two possibilities > exist: the current behavior of some kind of magical faerie land > behavior. > > I'm surprised to hear you say that the magical faerie land behavior > isn't desirable either, though. I'd love a tool that did what I wanted, > not what I asked. The only serious argument against this, I think, is > that it is beyond our current ability to create (and so anyone claiming > to be able to do it is probably mistaken). I'd argue that tools that do what you want rather than what you ask for are not just currently impossible, but always will be -- no matter how good the state of the art of artificial intelligent mind-reading software becomes. > You chopped out all the sections of this thread which discussed the more > desirable behavior. You can go back and read them in earlier messages > if you need to be reminded. I'm not talking about anything beyond > what's already been raised. I'm glad for you. But would you mind explaining for those of us aren't mind-readers what YOU consider the "more desirable behaviour"? If you're talking the list constructor and list comprehensions treating StopIteration the same, then I don't think it is at all self-evident that the current behaviour is a bad thing, nor that the only reason for it is that to do otherwise would be hard. (I don't think it would be hard to have list comps swallow a StopIteration.) > I'm pretty sure I know the answer to my question, though - it's hard to > implement, so it's not implemented. > > Jean-Paul -- Steven
From: exarkun on 13 Dec 2009 23:33 On 04:11 am, steven(a)remove.this.cybersource.com.au wrote: >On Sun, 13 Dec 2009 22:45:58 +0000, exarkun wrote: >>On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>>>StopIteration is intended to be used only within the .__next__ >>>>>method >>>>>of >>>>>iterators. The devs know that other 'off-label' use results in the >>>>>inconsistency you noted, but their and my view is 'don't do that'. >>>> >>>>Which is unfortunate, because it's not that hard to get >>>>StopIteration >>>>without explicitly raising it yourself and this behavior makes it >>>>difficult to debug such situations. >>> >>>I can't think of any way to get StopIteration without explicitly >>>raising >>>it yourself. It's not like built-ins or common data structures >>>routinely >>>raise StopIteration. I don't think I've *ever* seen a StopIteration >>>that >>>I didn't raise myself. >> >>Call next on an iterator. For example: iter(()).next() > >Or in more recent versions of Python, next(iter(())). > >Good example. But next() is a special case, and since next() is >documented as raising StopIteration if you call it and it raises >StopIteration, you have raised it yourself. Just not explicitly. But if you mistakenly don't catch it, and you're trying to debug your code to find this mistake, you probably won't be aided in this pursuit by the exception-swallowing behavior of generator expressions. > >>>>What's with this view, exactly? Is it just that it's hard to >>>>implement >>>>the more desirable behavior? >>> >>>What is that "more desirable behaviour"? That StopIteration is used >>>to >>>signal that Python should stop iterating except when you want it to >>>be >>>ignored? Unfortunately, yes, it's quite hard to implement "do what >>>the >>>caller actually wants, not what he asked for" behaviour -- and even >>>if >>>it were possible, it goes against the grain of the Zen of Python. >>> >>>If you've ever had to debug faulty "Do What I Mean" software, you'd >>>see >>>this as a good thing. >> >>I have plenty of experience developing and debugging software, Steven. >>Your argument is specious, as it presupposes that only two >>possibilities >>exist: the current behavior of some kind of magical faerie land >>behavior. >> >>I'm surprised to hear you say that the magical faerie land behavior >>isn't desirable either, though. I'd love a tool that did what I >>wanted, >>not what I asked. The only serious argument against this, I think, is >>that it is beyond our current ability to create (and so anyone >>claiming >>to be able to do it is probably mistaken). > >I'd argue that tools that do what you want rather than what you ask for >are not just currently impossible, but always will be -- no matter how >good the state of the art of artificial intelligent mind-reading >software >becomes. That may be true. I won't try to make any predictions about the arbitrarily distant future, though. >>You chopped out all the sections of this thread which discussed the >>more >>desirable behavior. You can go back and read them in earlier messages >>if you need to be reminded. I'm not talking about anything beyond >>what's already been raised. > >I'm glad for you. But would you mind explaining for those of us aren't >mind-readers what YOU consider the "more desirable behaviour"? The behavior of list comprehensions is pretty good. The behavior of constructing a list out of a generator expression isn't as good. The behavior which is more desirable is for a StopIteration raised out of the `expression` part of a `generator_expression` to not be treated identically to the way a StopIteration raised out of the `genexpr_for` part is. This could provide behavior roughly equivalent to the behavior of a list comprehension. > >If you're talking the list constructor and list comprehensions treating >StopIteration the same, then I don't think it is at all self-evident >that >the current behaviour is a bad thing, nor that the only reason for it >is >that to do otherwise would be hard. I don't expect it to be self-evident. I wasn't even trying to convince anyone that it's desirable (although I did claim it, so I won't fault anyone for making counter-arguments). The only thing I asked was what the motivation for the current behavior is. If the motivation is that it is self-evident that the current behavior is the best possible behavior, then someone just needs to say that and my question is answered. :) Jean-Paul
From: Terry Reedy on 14 Dec 2009 01:46
On 12/13/2009 10:29 PM, exarkun(a)twistedmatrix.com wrote: > Doesn't matter. Sometimes it makes sense to call it directly. It only makes sense to call next (or .__next__) when you are prepared to explicitly catch StopIteration within a try..except construct. You did not catch it, so it stopped execution. Let me repeat: StopIteration is intended only for stopping iteration. Outside that use, it is a normal exception with no special meaning. Terry Jan Reedy |