How to guard against bugs like this one?
in [Python]
|
From: Grant Edwards on 2 Feb 2010 10:00 On 2010-02-02, Roy Smith <roy(a)panix.com> wrote: > In article <hk82uv$8kn$1(a)reader1.panix.com>, kj <no.email(a)please.post> > wrote: > >> Through a *lot* of trial an error I finally discovered that the >> root cause of the problem was the fact that, in the same directory >> as buggy.py, there is *another* innocuous little script, totally >> unrelated, whose name happens to be numbers.py. >> [...] >> It turns out that buggy.py imports psycopg2, as you can see, and >> apparently psycopg2 (or something imported by psycopg2) tries to >> import some standard Python module called numbers; instead it ends >> up importing the innocent myscript/numbers.py, resulting in *absolute >> mayhem*. > > I feel your pain, but this is not a Python problem, per-se. I think it is. There should be different syntax to import from "standard" places and from "current directory". Similar to the difference between "foo.h" and <foo.h> in cpp. > The general > pattern is: > > 1) You have something which refers to a resource by name. > > 2) There is a sequence of places which are searched for this > name. Searching the current directory by default is the problem. Nobody in their right mind has "." in the shell PATH and IMO it shouldn't be in Python's import path either. Even those wreckless souls who do put "." in their path put it at the end so they don't accidentally override system commands. -- Grant Edwards grante Yow! So this is what it at feels like to be potato visi.com salad
From: Nobody on 2 Feb 2010 12:02 On Tue, 02 Feb 2010 15:00:28 +0000, Grant Edwards wrote: >>> It turns out that buggy.py imports psycopg2, as you can see, and >>> apparently psycopg2 (or something imported by psycopg2) tries to >>> import some standard Python module called numbers; instead it ends >>> up importing the innocent myscript/numbers.py, resulting in *absolute >>> mayhem*. >> >> I feel your pain, but this is not a Python problem, per-se. > > I think it is. I agree. > There should be different syntax to import from > "standard" places and from "current directory". Similar to the > difference between "foo.h" and <foo.h> in cpp. I don't know if that's necessary. Only supporting the "foo.h" case would work fine if Python behaved like gcc, i.e. if the "current directory" referred to the directory contain the file performing the import rather than in the process' CWD. As it stands, imports are dynamically scoped, when they should be lexically scoped. >> The general >> pattern is: >> >> 1) You have something which refers to a resource by name. >> >> 2) There is a sequence of places which are searched for this >> name. > > Searching the current directory by default is the problem. > Nobody in their right mind has "." in the shell PATH and IMO it > shouldn't be in Python's import path either. Even those > wreckless souls who do put "." in their path put it at the end > so they don't accidentally override system commands. Except, what should be happening here is that it should be searching the directory containing the file performing the import *first*. If foo.py contains "import bar", and there's a bar.py in the same directory as foo.py, that's the one it should be using. The existing behaviour is simply wrong, and there's no excuse for it ("but it's easier to implement" isn't a legitimate argument). The only situation where the process' CWD should be used is for an import statement in a non-file source (i.e. stdin or the argument to the -c switch).
From: Alf P. Steinbach on 2 Feb 2010 12:20 * Nobody: > On Tue, 02 Feb 2010 15:00:28 +0000, Grant Edwards wrote: > >>>> It turns out that buggy.py imports psycopg2, as you can see, and >>>> apparently psycopg2 (or something imported by psycopg2) tries to >>>> import some standard Python module called numbers; instead it ends >>>> up importing the innocent myscript/numbers.py, resulting in *absolute >>>> mayhem*. >>> I feel your pain, but this is not a Python problem, per-se. >> I think it is. > > I agree. > >> There should be different syntax to import from >> "standard" places and from "current directory". Similar to the >> difference between "foo.h" and <foo.h> in cpp. > > I don't know if that's necessary. Only supporting the "foo.h" case would > work fine if Python behaved like gcc, i.e. if the "current directory" > referred to the directory contain the file performing the import rather > than in the process' CWD. > > As it stands, imports are dynamically scoped, when they should be > lexically scoped. > >>> The general >>> pattern is: >>> >>> 1) You have something which refers to a resource by name. >>> >>> 2) There is a sequence of places which are searched for this >>> name. >> Searching the current directory by default is the problem. >> Nobody in their right mind has "." in the shell PATH and IMO it >> shouldn't be in Python's import path either. Even those >> wreckless souls who do put "." in their path put it at the end >> so they don't accidentally override system commands. > > Except, what should be happening here is that it should be searching the > directory containing the file performing the import *first*. If foo.py > contains "import bar", and there's a bar.py in the same directory as > foo.py, that's the one it should be using. > > The existing behaviour is simply wrong, and there's no excuse for it > ("but it's easier to implement" isn't a legitimate argument). +1 > The only situation where the process' CWD should be used is for an import > statement in a non-file source (i.e. stdin or the argument to the -c > switch). Hm, not sure about that last. Cheers, - Alf
From: Terry Reedy on 2 Feb 2010 13:29 On 2/2/2010 9:13 AM, kj wrote: >> As for fixing it, unfortunately it's not quite so simple to fix without >> breaking backwards-compatibility. The opportunity to do so for Python 3.0 >> was missed. > > This last point is to me the most befuddling of all. Does anyone > know why this opportunity was missed for 3.0? Anyone out there > with the inside scoop on this? Was the fixing of this problem > discussed in some PEP or some mailing list thread? (I've tried > Googling this but did not hit on the right keywords to bring up > the deliberations I'm looking for.) There was a proposal to put the whole stdlib into a gigantic package, so that import itertools would become, for instance import std.itertools. Guido rejected that. I believe he both did not like it and was concerned about making upgrade to 3.x even harder. The discussion was probably on the now closed py3k list. Terry Jan Reedy
From: Carl Banks on 2 Feb 2010 13:38
On Feb 2, 9:02 am, Nobody <nob...(a)nowhere.com> wrote: > I don't know if that's necessary. Only supporting the "foo.h" case would > work fine if Python behaved like gcc, i.e. if the "current directory" > referred to the directory contain the file performing the import rather > than in the process' CWD. > > As it stands, imports are dynamically scoped, when they should be > lexically scoped. Mostly incorrect. The CWD is in sys.path only for interactive sessions, and when started with -c switch. When running scripts, the directory where the script is located is used instead, not the process's working directory. So, no, it isn't anything like dynamic scoping. > The only situation where the process' CWD should be used is for an import > statement in a non-file source (i.e. stdin or the argument to the -c > switch). It already is that way, chief. I think you're misunderstanding what's wrong here; the CWD doesn't have anything to do with it. Even if CWD isn't in the path you still get the bad behavior kj noted. So now what? Python's importing can be improved but there's no foolproof way to get rid of the fundamental problem of name clashes. Carl Banks |