Prev: Can't find _sqlite3.so in lib-dynload
Next: max time wait for a function
From: paragk on 17 May 2010 22:02
Hi,

I am unable to figure out the cause of python pickle unable to find
the collection module.

I am getting

__import__(module)
ImportError: No module named collections

when I try to load a pickled object.

Details:

Python version:

Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit
(Intel)] on win32


[CODE]

import sys
import glob
import collections
import pickle

....

SomeListDict = collections.defaultdict(list)

.... # Populate SomeListDict

DictFile = open (options.saveDict, 'w')
pickle.dump(SomeListDict , DictFile, -1)
DictFile.close()

....

# Try loading the dictionary

LoadDictFile = open(options.loadDict, 'rb')
SomeListDict = pickle.load(LoadDictFile)
LoadDictFile.close()

....
[CODE END]

The pickle.dump(CallGraphDict, DictFile, -1) works fine. The
dictionary is saved.

The interesting thing is, the dump function is able to find the
collections module.
try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)

The above try block succeeds.

module
str: collections

mod
module: <module 'collections' from 'C:\Python26\Lib\collections.pyc'>

klass
type: <type 'collections.defaultdict'>


The SomeListDict = pickle.load(LoadDictFile) fails with

__import__(module)
ImportError: No module named collections


The sequence of calls is:

def load_global(self):
module = self.readline()[:-1]
name = self.readline()[:-1]
klass = self.find_class(module, name)
self.append(klass)
dispatch[GLOBAL] = load_global

Variables:

module
str: collections

name
str: defaultdict


The code where the exception occurs is:

def find_class(self, module, name):
# Subclasses may override this
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
return klass


From the above analysis, clearly the collection module exists, since
the dump works.

What am I missing?
From: Steven D'Aprano on 17 May 2010 22:07
On Mon, 17 May 2010 19:02:18 -0700, paragk wrote:

> Hi,
>
> I am unable to figure out the cause of python pickle unable to find the
> collection module.
>
> I am getting
>
> __import__(module)
> ImportError: No module named collections

What is the actual value of module? Here's an obvious failure mode:

>>> __import__("collections ")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named collections


> module
> str: collections

Try printing repr(str) to see what's actually there.


It may also help if you post a copy-and-paste of the complete traceback.



--
Steven
From: Peter Otten on 18 May 2010 03:10
paragk wrote:

> Hi,
>
> I am unable to figure out the cause of python pickle unable to find
> the collection module.
>
> I am getting
>
> __import__(module)
> ImportError: No module named collections
>
> when I try to load a pickled object.
>
> Details:
>
> Python version:
>
> Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit
> (Intel)] on win32
>
>
> [CODE]
>
> import sys
> import glob
> import collections
> import pickle
>
> ...
>
> SomeListDict = collections.defaultdict(list)
>
> ... # Populate SomeListDict
>
> DictFile = open (options.saveDict, 'w')
> pickle.dump(SomeListDict , DictFile, -1)
> DictFile.close()

[...]

> From the above analysis, clearly the collection module exists, since
> the dump works.
>
> What am I missing?

You have to open the file in binary mode "wb".

>>> import collections
>>> import pickle
>>> data = pickle.dumps(collections.defaultdict())
>>> pickle.loads(data)
defaultdict(None, {})

Now simulate the effect of writing in text mode and reading in binary mode:

>>> garbled_data = data.replace("\n", "\r\n")
>>> pickle.loads(garbled_data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/usr/lib/python2.6/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
__import__(module)
ImportError: No module named collections

>>> try: pickle.loads(garbled_data)
.... except ImportError as e:
.... e
....
ImportError('No module named collections\r',)

So as Steven suspected there was a whitespace char in the module name:
pickle.load() was looking for the "collections\r" module.

Peter
From: paragk on 18 May 2010 04:26
On May 18, 12:10 am, Peter Otten <__pete...(a)web.de> wrote:
> paragk wrote:
> > Hi,
>
> > I am unable to figure out the cause of python pickle unable to find
> > the collection module.
>
> > I am getting
>
> >     __import__(module)
> > ImportError: No module named collections
>
> > when I try to load a pickled object.
>
> > Details:
>
> > Python version:
>
> > Python 2.6 (r26:66721, Oct  2 2008, 11:35:03) [MSC v.1500 32 bit
> > (Intel)] on win32
>
> > [CODE]
>
> > import sys
> > import glob
> > import collections
> > import pickle
>
> > ...
>
> > SomeListDict = collections.defaultdict(list)
>
> > ... # Populate SomeListDict
>
> > DictFile = open (options.saveDict, 'w')
> > pickle.dump(SomeListDict , DictFile, -1)
> > DictFile.close()
>
> [...]
>
> > From the above analysis, clearly the collection module exists, since
> > the dump works.
>
> > What am I missing?
>
> You have to open the file in binary mode "wb".
>
> >>> import collections
> >>> import pickle
> >>> data = pickle.dumps(collections.defaultdict())
> >>> pickle.loads(data)
>
> defaultdict(None, {})
>
> Now simulate the effect of writing in text mode and reading in binary mode:
>
> >>> garbled_data = data.replace("\n", "\r\n")
> >>> pickle.loads(garbled_data)
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python2.6/pickle.py", line 1374, in loads
>     return Unpickler(file).load()
>   File "/usr/lib/python2.6/pickle.py", line 858, in load
>     dispatch[key](self)
>   File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
>     klass = self.find_class(module, name)
>   File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
>     __import__(module)
> ImportError: No module named collections
>
> >>> try: pickle.loads(garbled_data)
>
> ... except ImportError as e:
> ...     e
> ...
> ImportError('No module named collections\r',)
>
> So as Steven suspected there was a whitespace char in the module name:
> pickle.load() was looking for the "collections\r" module.
>
> Peter

Thank you for the suggestions. Opening the file in 'wb' mode worked.

Thanks,
Parag
From: Terry Reedy on 18 May 2010 13:07
On 5/18/2010 3:10 AM, Peter Otten wrote:
> Now simulate the effect of writing in text mode and reading in binary mode:
>
>>>> >>> garbled_data = data.replace("\n", "\r\n")
>>>> >>> pickle.loads(garbled_data)
> Traceback (most recent call last):
....
> ImportError: No module named collections
>
>>>> >>> try: pickle.loads(garbled_data)
> ... except ImportError as e:
> ... e
> ...
> ImportError('No module named collections\r',)

Nice catch, Peter. I filed a request that the message quote the name it
cannot import, so that the OP would have seen the more informative

ImportError: No module named 'collections\r'

http://bugs.python.org/issue8754

Terry Jan Reedy


 |  Next  |  Last
Pages: 1 2
Prev: Can't find _sqlite3.so in lib-dynload
Next: max time wait for a function