Prev: Is there any way to minimize str()/unicode() objects memory usage [Python 2.6.4] ?
Next: Is there any way to minimize str()/unicode() objects memory usage[Python 2.6.4] ?
From: Steven D'Aprano on 6 Aug 2010 21:02
On Fri, 06 Aug 2010 17:45:31 -0700, dmtr wrote:

> I'm running into some performance / memory bottlenecks on large lists.
> Is there any easy way to minimize/optimize memory usage?

Yes, lots of ways. For example, do you *need* large lists? Often a better
design is to use generators and iterators to lazily generate data when
you need it, rather than creating a large list all at once.

An optimization that sometimes may help is to intern strings, so that
there's only a single copy of common strings rather than multiple copies
of the same one.

Can you compress the data and use that? Without knowing what you are
trying to do, and why, it's really difficult to advise a better way to do
it (other than vague suggestions like "use generators instead of lists").

Very often, it is cheaper and faster to just put more memory in the
machine than to try optimizing memory use. Memory is cheap, your time and
effort is not.

[...]
> Well... 63 bytes per item for very short unicode strings... Is there
> any way to do better than that? Perhaps some compact unicode objects?

If you think that unicode objects are going to be *smaller* than byte
strings, I think you're badly informed about the nature of unicode.

Python is not a low-level language, and it trades off memory compactness
for ease of use. Python strings are high-level rich objects, not merely a
contiguous series of bytes. If all else fails, you might have to use
something like the array module, or even implement your own data type in
C.

But as a general rule, as I mentioned above, the best way to minimize the
memory used by a large list is to not use a large list. I can't emphasise
that enough -- look into generators and iterators, and lazily handle your
data whenever possible.


--
Steven
 | 
Pages: 1
Prev: Is there any way to minimize str()/unicode() objects memory usage [Python 2.6.4] ?
Next: Is there any way to minimize str()/unicode() objects memory usage[Python 2.6.4] ?