A "scopeguard" for Python
in [Python]
|
From: Robert Kern on 4 Mar 2010 23:01 On 2010-03-04 17:52 , Alf P. Steinbach wrote: > * Robert Kern: >> On 2010-03-04 12:37 PM, Alf P. Steinbach wrote: >>> * Robert Kern: >>>> On 2010-03-04 10:56 AM, Alf P. Steinbach wrote: >>>>> * Robert Kern: >>>>>> On 2010-03-03 18:49 PM, Alf P. Steinbach wrote: >>> [snippety] >>>>>> >>>>>>> If you call the possibly failing operation "A", then that systematic >>>>>>> approach goes like this: if A fails, then it has cleaned up its own >>>>>>> mess, but if A succeeds, then it's the responsibility of the calling >>>>>>> code to clean up if the higher level (multiple statements) operation >>>>>>> that A is embedded in, fails. >>>>>>> >>>>>>> And that's what Marginean's original C++ ScopeGuard was designed >>>>>>> for, >>>>>>> and what the corresponding Python Cleanup class is designed for. >>>>>> >>>>>> And try: finally:, for that matter. >>>>> >>>>> Not to mention "with". >>>>> >>>>> Some other poster made the same error recently in this thread; it is a >>>>> common fallacy in discussions about programming, to assume that since >>>>> the same can be expressed using lower level constructs, those are all >>>>> that are required. >>>>> >>>>> If adopted as true it ultimately means the removal of all control >>>>> structures above the level of "if" and "goto" (except Python doesn't >>>>> have "goto"). >>>> >>>> What I'm trying to explain is that the with: statement has a use even >>>> if Cleanup doesn't. Arguing that Cleanup doesn't improve on try: >>>> finally: does not mean that the with: statement doesn't improve on >>>> try: finally:. >>> >>> That's a different argument, essentially that you see no advantage for >>> your current coding patterns. >>> >>> It's unconnected to the argument I responded to. >>> >>> The argument that I responded to, that the possibility of expressing >>> things at the level of try:finally: means that a higher level construct >>> is superfluous, is still meaningless. >> >> I am attacking your premise that the "with Cleanup():" construct is >> higher level than try: finally:. It isn't. It provides the same level >> of abstraction as try: finally:. >> >> This is distinct from the accepted uses of the with: statement which >> *are* higher level than try: finally: and which do confer practical >> benefits over using try: finally: despite being syntactical sugar for >> try: finally:. >> >>>>>>>> Both formulations can be correct (and both work perfectly fine with >>>>>>>> the chdir() example being used). Sometimes one is better than the >>>>>>>> other, and sometimes not. You can achieve both ways with either >>>>>>>> your >>>>>>>> Cleanup class or with try: finally:. >>>>>>>> >>>>>>>> I am still of the opinion that Cleanup is not an improvement over >>>>>>>> try: >>>>>>>> finally: and has the significant ugliness of forcing cleanup code >>>>>>>> into >>>>>>>> callables. This significantly limits what you can do in your >>>>>>>> cleanup >>>>>>>> code. >>>>>>> >>>>>>> Uhm, not really. :-) As I see it. >>>>>> >>>>>> Well, not being able to affect the namespace is a significant >>>>>> limitation. Sometimes you need to delete objects from the >>>>>> namespace in >>>>>> order to ensure that their refcounts go to zero and their cleanup >>>>>> code >>>>>> gets executed. >>>>> >>>>> Just a nit (I agree that a lambda can't do this, but as to what's >>>>> required): assigning None is sufficient for that[1]. >>>> >>>> Yes, but no callable is going to allow you to assign None to names in >>>> that namespace, either. Not without sys._getframe() hackery, in any >>>> case. >>>> >>>>> However, note that the current language doesn't guarantee such >>>>> cleanup, >>>>> at least as far as I know. >>>>> >>>>> So while it's good practice to support it, to do everything to let it >>>>> happen, it's presumably bad practice to rely on it happening. >>>>> >>>>> >>>>>> Tracebacks will keep the namespace alive and all objects in it. >>>>> >>>>> Thanks!, I hadn't thought of connecting that to general cleanup >>>>> actions. >>>>> >>>>> It limits the use of general "with" in the same way. >>>> >>>> Not really. >>> >>> Sorry, it limits general 'with' in /exactly/ the same way. >>> >>>> It's easy to write context managers that do that [delete objects from >>>> the namespace]. >>> >>> Sorry, no can do, as far as I know; your following example quoted below >>> is an example of /something else/. >> >> Okay, so what do you mean by 'the use of general "with"'? I'm talking >> about writing a context manager or using the @contextmanager decorator >> to do some initialization and then later cleaning up that >> initialization. That cleaning up may entail deleting an object. You >> are correct that the context manager can't affect the namespace of the >> with: clause, but that's not the initialization that it would need to >> clean up. >> >> Yes, you can write code with a with: statement where you try to clean >> up stuff that happened inside of the clause (you did), but that's not >> how the with: statement was ever intended to be used nor is it good >> practice to do so because of that limitation. Context managers are >> designed to initialize specific things, then clean them up. I thought >> you were talking about the uses of the with: statement as described in >> PEP-343, not every possible misuse of the with: statement. > > I'm not the one talking about removing variables or that "it's easy to > write context managers that do that". > > You are the one talking about that. > > So I have really not much to add. > > It seems that you're now agreeing with me that former is not good > practice and that the latter is impossible to do portably, but you now > argue against your earlier stand as if that was something that I had put > forward. No, I'm still saying that sometimes you do need to remove variables that you initialized in the initialization section. Writing a purpose-built context manager which encapsulates the initialization and finalization allows you to do this easily. Putting the initialization section inside the with: clause, as you do, and requiring cleanup code to be put into callables makes this hard. > It's a bit confusing when you argue against your own statements. > >>> And adding on top of irrelevancy, for the pure technical aspect it can >>> be accomplished in the same way using Cleanup (I provide an example >>> below). >>> >>> However, doing that would generally be worse than pointless since with >>> good coding practices the objects would become unreferenced anyway. >>> >>> >>>> You put the initialization code in the __enter__() method, assign >>>> whatever objects you want to keep around through the with: clause as >>>> attributes on the manager, then delete those attributes in the >>>> __exit__(). >>> >>> Analogously, if one were to do this thing, then it could be accomplished >>> using a Cleanup context manager as follows: >>> >>> foo = lambda: None >>> foo.x = create_some_object() >>> at_cleanup.call( lambda o = foo: delattr( o, "x" ) ) >>> >>> ... except that >>> >>> 1) for a once-only case this is less code :-) >> >> Not compared to a try: finally:, it isn't. > > Again, this context shifting is bewildering. As you can see, quoted > above, you were talking about a situation where you would have defined a > context manager, presumably because a 'try' would not in your opinion be > simpler for whatever it was that you had in mind. But you are responding > to the code I offered as if it was an alternative to something where you > would find a 'try' to be simplest. I have consistently put forward that for once-only cases, try: finally: is a preferable construct to "with Cleanup():". I also put forward that for repetitive cases, a purpose-built context manager is preferable. I note that for both try: finally: and a purpose-built context manager, modifying the namespace is easy to do while it is difficult and less readable to do with "with Cleanup():". Since you were claiming that the "generic with:" would have the same problem as "with Cleanup():" I was trying to explain for why all of the intended use cases of the with: statement (the purpose-built context managers), there is no problem. Capisce? >>> 2) it is a usage that I wouldn't recommend; instead I recommend adopting >>> good >>> coding practices where object references aren't kept around. >> >> Many of the use cases of the with: statement involve creating an >> object (like a lock or a transaction object), keeping it around for >> the duration of the "# Do stuff" block, and then finalizing it. >> >>>> Or, you use the @contextmanager decorator to turn a generator into a >>>> context manager, and you just assign to local variables and del them >>>> in the finally: clause. >>> >>> Uhm, you don't need a 'finally' clause when you define a context >>> manager. >> >> When you use the @contextmanager decorator, you almost always do. See >> the Examples section of PEP 343: >> >> http://www.python.org/dev/peps/pep-0343/ >> >>> Additionally, you don't need to 'del' the local variables in >>> @contextmanager decorated generator. >>> >>> The local variables cease to exist automatically. >> >> True. >> >>>> What you can't do is write a generic context manager where the >>>> initialization happens inside the with: clause and the cleanup actions >>>> are registered callables. That does not allow you to affect the >>>> namespace. >>> >>> If you mean that you can't introduce direct local variables and have >>> them deleted by "registered callables" in a portable way, then right. >>> >>> But I can't think of any example where that would be relevant; in >>> particular what matters for supporting on-destruction cleanup is whether >>> you keep any references or not, not whether you have a local variable of >>> any given name. >> >> Well, local variables keep references to objects. Variable assignment >> followed by deletion is a very readable way to keep an object around >> for a while then remove it later. If you have to go through less >> readable contortions to keep the object around when it needs to be and >> clean it up later, then that is a mark against your approach. > > Sorry, as with the places noted above, I can't understand what you're > trying to say here. I don't recommend coding practices where you keep > object references around, Then how do you clean up locks and transaction objects and similar things if you don't keep them around during the code suite? Creating an object, keeping it around while executing a specific chunk of code, and finalizing it safely is a prime use case for these kinds of constructs. > and you have twice quoted that above. I don't > have any clue what "contortions" you are talking about, it must be > something that you imagine. These are the contortions: foo = lambda: None foo.x = create_some_object() at_cleanup.call( lambda o = foo: delattr( o, "x" ) ) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
From: Jean-Michel Pichavant on 5 Mar 2010 07:20 Robert Kern wrote: > On 2010-03-04 15:12 , Mike Kent wrote: >> On Mar 4, 12:30 pm, Robert Kern<robert.k...(a)gmail.com> wrote: >> >>> He's ignorant of the use cases of the with: statement, true. >> >> <humor> Ouch! Ignorant of the use cases of the with statement, am I? >> Odd, I use it all the time.</humor> > > No, I was referring to Jean-Michel, who was not familiar with the > with: statement. > >>> Given only your >>> example of the with: statement, it is hard to fault him for thinking >>> that try: >>> finally: wouldn't suffice. >> >> <humor> Damn me with faint praise, will you?</humor> > > Also talking about Jean-Michel. :-) > I confirm, I am the ignoramus (what a strange word) in this story :-) JM
From: Alf P. Steinbach on 5 Mar 2010 08:06 * Robert Kern: > On 2010-03-04 17:52 , Alf P. Steinbach wrote: >> * Robert Kern: >>> On 2010-03-04 12:37 PM, Alf P. Steinbach wrote: >>>> * Robert Kern: >>>>> On 2010-03-04 10:56 AM, Alf P. Steinbach wrote: >>>>>> * Robert Kern: >>>>>>> On 2010-03-03 18:49 PM, Alf P. Steinbach wrote: >>>> [snippety] >>>>>>> >>>>>>>> If you call the possibly failing operation "A", then that >>>>>>>> systematic >>>>>>>> approach goes like this: if A fails, then it has cleaned up its own >>>>>>>> mess, but if A succeeds, then it's the responsibility of the >>>>>>>> calling >>>>>>>> code to clean up if the higher level (multiple statements) >>>>>>>> operation >>>>>>>> that A is embedded in, fails. >>>>>>>> >>>>>>>> And that's what Marginean's original C++ ScopeGuard was designed >>>>>>>> for, >>>>>>>> and what the corresponding Python Cleanup class is designed for. >>>>>>> >>>>>>> And try: finally:, for that matter. >>>>>> >>>>>> Not to mention "with". >>>>>> >>>>>> Some other poster made the same error recently in this thread; it >>>>>> is a >>>>>> common fallacy in discussions about programming, to assume that since >>>>>> the same can be expressed using lower level constructs, those are all >>>>>> that are required. >>>>>> >>>>>> If adopted as true it ultimately means the removal of all control >>>>>> structures above the level of "if" and "goto" (except Python doesn't >>>>>> have "goto"). >>>>> >>>>> What I'm trying to explain is that the with: statement has a use even >>>>> if Cleanup doesn't. Arguing that Cleanup doesn't improve on try: >>>>> finally: does not mean that the with: statement doesn't improve on >>>>> try: finally:. >>>> >>>> That's a different argument, essentially that you see no advantage for >>>> your current coding patterns. >>>> >>>> It's unconnected to the argument I responded to. >>>> >>>> The argument that I responded to, that the possibility of expressing >>>> things at the level of try:finally: means that a higher level construct >>>> is superfluous, is still meaningless. >>> >>> I am attacking your premise that the "with Cleanup():" construct is >>> higher level than try: finally:. It isn't. It provides the same level >>> of abstraction as try: finally:. >>> >>> This is distinct from the accepted uses of the with: statement which >>> *are* higher level than try: finally: and which do confer practical >>> benefits over using try: finally: despite being syntactical sugar for >>> try: finally:. >>> >>>>>>>>> Both formulations can be correct (and both work perfectly fine >>>>>>>>> with >>>>>>>>> the chdir() example being used). Sometimes one is better than the >>>>>>>>> other, and sometimes not. You can achieve both ways with either >>>>>>>>> your >>>>>>>>> Cleanup class or with try: finally:. >>>>>>>>> >>>>>>>>> I am still of the opinion that Cleanup is not an improvement over >>>>>>>>> try: >>>>>>>>> finally: and has the significant ugliness of forcing cleanup code >>>>>>>>> into >>>>>>>>> callables. This significantly limits what you can do in your >>>>>>>>> cleanup >>>>>>>>> code. >>>>>>>> >>>>>>>> Uhm, not really. :-) As I see it. >>>>>>> >>>>>>> Well, not being able to affect the namespace is a significant >>>>>>> limitation. Sometimes you need to delete objects from the >>>>>>> namespace in >>>>>>> order to ensure that their refcounts go to zero and their cleanup >>>>>>> code >>>>>>> gets executed. >>>>>> >>>>>> Just a nit (I agree that a lambda can't do this, but as to what's >>>>>> required): assigning None is sufficient for that[1]. >>>>> >>>>> Yes, but no callable is going to allow you to assign None to names in >>>>> that namespace, either. Not without sys._getframe() hackery, in any >>>>> case. >>>>> >>>>>> However, note that the current language doesn't guarantee such >>>>>> cleanup, >>>>>> at least as far as I know. >>>>>> >>>>>> So while it's good practice to support it, to do everything to let it >>>>>> happen, it's presumably bad practice to rely on it happening. >>>>>> >>>>>> >>>>>>> Tracebacks will keep the namespace alive and all objects in it. >>>>>> >>>>>> Thanks!, I hadn't thought of connecting that to general cleanup >>>>>> actions. >>>>>> >>>>>> It limits the use of general "with" in the same way. >>>>> >>>>> Not really. >>>> >>>> Sorry, it limits general 'with' in /exactly/ the same way. >>>> >>>>> It's easy to write context managers that do that [delete objects from >>>>> the namespace]. >>>> >>>> Sorry, no can do, as far as I know; your following example quoted below >>>> is an example of /something else/. >>> >>> Okay, so what do you mean by 'the use of general "with"'? I'm talking >>> about writing a context manager or using the @contextmanager decorator >>> to do some initialization and then later cleaning up that >>> initialization. That cleaning up may entail deleting an object. You >>> are correct that the context manager can't affect the namespace of the >>> with: clause, but that's not the initialization that it would need to >>> clean up. >>> >>> Yes, you can write code with a with: statement where you try to clean >>> up stuff that happened inside of the clause (you did), but that's not >>> how the with: statement was ever intended to be used nor is it good >>> practice to do so because of that limitation. Context managers are >>> designed to initialize specific things, then clean them up. I thought >>> you were talking about the uses of the with: statement as described in >>> PEP-343, not every possible misuse of the with: statement. >> >> I'm not the one talking about removing variables or that "it's easy to >> write context managers that do that". >> >> You are the one talking about that. >> >> So I have really not much to add. >> >> It seems that you're now agreeing with me that former is not good >> practice and that the latter is impossible to do portably, but you now >> argue against your earlier stand as if that was something that I had put >> forward. > > No, I'm still saying that sometimes you do need to remove variables that > you initialized in the initialization section. Writing a purpose-built > context manager which encapsulates the initialization and finalization > allows you to do this easily. Putting the initialization section inside > the with: clause, as you do, and requiring cleanup code to be put into > callables makes this hard. > >> It's a bit confusing when you argue against your own statements. >> >>>> And adding on top of irrelevancy, for the pure technical aspect it can >>>> be accomplished in the same way using Cleanup (I provide an example >>>> below). >>>> >>>> However, doing that would generally be worse than pointless since with >>>> good coding practices the objects would become unreferenced anyway. >>>> >>>> >>>>> You put the initialization code in the __enter__() method, assign >>>>> whatever objects you want to keep around through the with: clause as >>>>> attributes on the manager, then delete those attributes in the >>>>> __exit__(). >>>> >>>> Analogously, if one were to do this thing, then it could be >>>> accomplished >>>> using a Cleanup context manager as follows: >>>> >>>> foo = lambda: None >>>> foo.x = create_some_object() >>>> at_cleanup.call( lambda o = foo: delattr( o, "x" ) ) >>>> >>>> ... except that >>>> >>>> 1) for a once-only case this is less code :-) >>> >>> Not compared to a try: finally:, it isn't. >> >> Again, this context shifting is bewildering. As you can see, quoted >> above, you were talking about a situation where you would have defined a >> context manager, presumably because a 'try' would not in your opinion be >> simpler for whatever it was that you had in mind. But you are responding >> to the code I offered as if it was an alternative to something where you >> would find a 'try' to be simplest. > > I have consistently put forward that for once-only cases, try: finally: > is a preferable construct to "with Cleanup():". I also put forward that > for repetitive cases, a purpose-built context manager is preferable. I > note that for both try: finally: and a purpose-built context manager, > modifying the namespace is easy to do while it is difficult and less > readable to do with "with Cleanup():". Since you were claiming that the > "generic with:" would have the same problem as "with Cleanup():" I was > trying to explain for why all of the intended use cases of the with: > statement (the purpose-built context managers), there is no problem. > Capisce? > >>>> 2) it is a usage that I wouldn't recommend; instead I recommend >>>> adopting >>>> good >>>> coding practices where object references aren't kept around. >>> >>> Many of the use cases of the with: statement involve creating an >>> object (like a lock or a transaction object), keeping it around for >>> the duration of the "# Do stuff" block, and then finalizing it. >>> >>>>> Or, you use the @contextmanager decorator to turn a generator into a >>>>> context manager, and you just assign to local variables and del them >>>>> in the finally: clause. >>>> >>>> Uhm, you don't need a 'finally' clause when you define a context >>>> manager. >>> >>> When you use the @contextmanager decorator, you almost always do. See >>> the Examples section of PEP 343: >>> >>> http://www.python.org/dev/peps/pep-0343/ >>> >>>> Additionally, you don't need to 'del' the local variables in >>>> @contextmanager decorated generator. >>>> >>>> The local variables cease to exist automatically. >>> >>> True. >>> >>>>> What you can't do is write a generic context manager where the >>>>> initialization happens inside the with: clause and the cleanup actions >>>>> are registered callables. That does not allow you to affect the >>>>> namespace. >>>> >>>> If you mean that you can't introduce direct local variables and have >>>> them deleted by "registered callables" in a portable way, then right. >>>> >>>> But I can't think of any example where that would be relevant; in >>>> particular what matters for supporting on-destruction cleanup is >>>> whether >>>> you keep any references or not, not whether you have a local >>>> variable of >>>> any given name. >>> >>> Well, local variables keep references to objects. Variable assignment >>> followed by deletion is a very readable way to keep an object around >>> for a while then remove it later. If you have to go through less >>> readable contortions to keep the object around when it needs to be and >>> clean it up later, then that is a mark against your approach. >> >> Sorry, as with the places noted above, I can't understand what you're >> trying to say here. I don't recommend coding practices where you keep >> object references around, > > Then how do you clean up locks and transaction objects and similar > things if you don't keep them around during the code suite? Creating an > object, keeping it around while executing a specific chunk of code, and > finalizing it safely is a prime use case for these kinds of constructs. Where you need a scope, create a scope. That is, put the logic in a routine. However, most objects aren't of the sort the requiring to become unreferenced. Those that require cleanup can be cleaned up and left as zombies, like calling 'close' on a file. And so in the general case what you're discussing, how to get rid of object references, is a non-issue -- irrelevant. >> and you have twice quoted that above. I don't >> have any clue what "contortions" you are talking about, it must be >> something that you imagine. > > These are the contortions: > > foo = lambda: None > foo.x = create_some_object() > at_cleanup.call( lambda o = foo: delattr( o, "x" ) ) That's code that demonstrates something very different, in response to your description of doing this. Since I half suspected that it could be taken out of context I followed that immediately with <quote> 2) it is a usage that I wouldn't recommend; instead I recommend adopting good coding practices where object references aren't kept around. </quote> I'm sorry but I don't know how to make that more clear. Cheers & hth., - Alf
From: Alf P. Steinbach on 5 Mar 2010 08:09 * Robert Kern: > On 2010-03-04 16:27 , Alf P. Steinbach wrote: >> * Mike Kent: > >>> However, I fail to understand his response that I must have meant try/ >>> else instead, as this, as Mr. Kern pointed out, is invalid syntax. >>> Perhaps Mr. Steinbach would like to give an example? >> >> OK. >> >> Assuming that you wanted the chdir to be within a try block (which it >> was in your code), then to get code equivalent to my code, for the >> purpose of a comparision of codes that do the same, you'd have to write >> something like ... >> >> original_dir = os.getcwd() >> try: >> os.chdir(somewhere) >> except Whatever: >> # E.g. log it. >> raise >> else: >> try: >> # Do other stuff >> finally: >> os.chdir(original_dir) >> # Do other cleanup >> >> ... which would be a more general case. >> >> I've also given this example in response to Robert earlier in the >> thread. Although I haven't tried it I believe it's syntactically valid. >> If not, then the relevant typo should just be fixed. :-) >> >> I have no idea which construct Robert thought was syntactically invalid. >> I think that if he's written that, then it must have been something he >> thought of. > > I was just trying to interpret what you meant by "Changing 'finally' to > 'else' could make it equivalent." Oh yes, in the article where I gave the example of that, shown above. Hey, discussing previous discussion is silly. Cheers, - ALf
From: Mike Kent on 5 Mar 2010 11:29
On Mar 4, 8:04 pm, Robert Kern <robert.k...(a)gmail.com> wrote: > No, the try: finally: is not implicit. See the source for > contextlib.GeneratorContextManager. When __exit__() gets an exception from the > with: block, it will push it into the generator using its .throw() method.. This > raises the exception inside the generator at the yield statement. Wow, I just learned something new. My understanding of context managers was that the __exit__ method was guaranteed to be executed regardless of how the context was left. I have often written my own context manager classes, giving them the __enter__ and __exit__ methods. I had mistakenly assumed that the @contextmanager decorator turned a generator function into a context manager with the same behavior as the equivalent context manager class. Now I learn that, no, in order to have the 'undo' code executed in the presence of an exception, you must write your own try/finally block in the generator function. This raises the question in my mind: What's the use case for using @contextmanager rather than wrapping your code in a context manager class that defines __enter__ and __exit__, if you still have to manager your own try/finally block? |