Prev: socked and bytes operation
Next: [Python3] Reading a binary file and wrtiting the bytes verbatim in an utf-8 file
From: Steven D'Aprano on 22 Apr 2010 05:10
On Thu, 22 Apr 2010 09:31:12 +0200, candide wrote:

> Alf P. Steinbach a écrit :
>> * candide:
>>> Suppose a and b are lists.
>>>
>>> What is more efficient in order to extend the list a by appending all
>>> the items in the list b ?
>>>
>>>
>>> I imagine a.extend(b)to be more efficient for only appendinding the
>>> items from b while a+=b creates a copy of a before appending, right ?
>>
>> No.
>>
>> But in general, if you're concerned about efficiency, *measure*.
>>
>>
>
> But my question refers to memory management rather than to time
> execution. Imagine foo is a big size list and bar is a small one. It
> would be a waste of memory to copy list foo somewhere in memory before
> appending the items in bar.



Yes, unfortunately measuring Python's memory use from within Python is
not easy... however the fact that += and extend take approximately the
same time suggests that they are doing approximately the same amount of
work, and therefore it's unlikely that one is radically more wasteful of
memory than the other.

Keep in mind that since Python uses duck-typing, you may not necessarily
be dealing with actual built-in lists, but some other list-like object.
If you're programming a library, and don't control your input, any
conclusion you draw about lists may not apply in practice. If somebody
provides you a sequence like this, what are you going to do?

class StupidList(list):
def extend(self, other):
new = []
for item in self:
new = new[:] + item
for item in other:
new = new[:] + item
return new
def __iadd__(self, other):
self.extend(other)
return self

You can't defend against the caller doing something stupid, so it's not
worth spending too much effort trying.

In general the warnings against premature optimization apply just as
strongly against memory optimizations as against speed optimizations.



--
Steven
From: Raymond Hettinger on 22 Apr 2010 06:20
On Apr 22, 12:10 am, candide <cand...(a)free.invalid> wrote:
> Suppose a and b are lists.
>
> What is more efficient in order to extend the list a by appending all
> the items in the list b ?
>
> I imagine a.extend(b)to be more efficient for only appendinding the
> items from b while a+=b creates a copy of a before appending, right ?

The a+=b form invokes list.__iadd__() which is implemented using
list.extend(), so the two are basically the same. Looking at the
source in http://svn.python.org/view/python/trunk/Objects/listobject.c?revision=78522&view=markup
we see:

static PyObject *
list_inplace_concat(PyListObject *self, PyObject *other)
{
PyObject *result;

result = listextend(self, other);
if (result == NULL)
return result;
Py_DECREF(result);
Py_INCREF(self);
return (PyObject *)self;
}

There is a slight and constant difference is the overhead for making
the call. The a.extend(b) does a dictionary lookup for the "extend"
method and creates a bound method. Using a+=b is a little more
direct.


Raymond
First  |  Prev  | 
Pages: 1 2
Prev: socked and bytes operation
Next: [Python3] Reading a binary file and wrtiting the bytes verbatim in an utf-8 file