Prev: os.fdopen() issue in Python 3.1?
Next: freebsd and multiprocessing
From: mk on 2 Mar 2010 16:54
Daniel Stutzbach wrote:
> On Tue, Mar 2, 2010 at 1:58 PM, Martin P. Hellwig
> <martin.hellwig(a)dcuktec.org <mailto:martin.hellwig(a)dcuktec.org>> wrote:
>
> What actually happens if multiple threads at the same time, write to
> a shared dictionary (Not using the same key)?

> All of Python's built-in types are thread safe. Both updates will happen.

No need to use synchro primitives like locks?

I know that it may work, but that strikes me as somehow wrong... I'm
used to using things like Lock().acquire() and Lock().release() when
accessing shared data structures, whatever they are.

Although trying to do the "right thing" may indeed get one in trouble in
case of deadlock caused by a bug in one's own program.

Regards,
mk


From: John Krukoff on 2 Mar 2010 17:08
On Tue, 2010-03-02 at 22:54 +0100, mk wrote:
<snip>
> No need to use synchro primitives like locks?
>
> I know that it may work, but that strikes me as somehow wrong... I'm
> used to using things like Lock().acquire() and Lock().release() when
> accessing shared data structures, whatever they are.
<snip>

This is one of those places where the GIL is a good thing, and makes
your life simpler. You could consider it that the interpreter does the
locking for you for such primitive operations, if you like.
--
John Krukoff <jkrukoff(a)ltgc.com>
Land Title Guarantee Company

From: MRAB on 2 Mar 2010 20:00
John Krukoff wrote:
> On Tue, 2010-03-02 at 22:54 +0100, mk wrote:
> <snip>
>> No need to use synchro primitives like locks?
>>
>> I know that it may work, but that strikes me as somehow wrong... I'm
>> used to using things like Lock().acquire() and Lock().release() when
>> accessing shared data structures, whatever they are.
> <snip>
>
> This is one of those places where the GIL is a good thing, and makes
> your life simpler. You could consider it that the interpreter does the
> locking for you for such primitive operations, if you like.

I suppose it depends on the complexity of the data structure. A dict's
methods are threadsafe, for example, but if you have a data structure
where access leads to multiple method calls then collectively they need
a lock.
From: Gregory Ewing on 3 Mar 2010 01:14
MRAB wrote:

> I suppose it depends on the complexity of the data structure. A dict's
> methods are threadsafe, for example, but if you have a data structure
> where access leads to multiple method calls then collectively they need
> a lock.

It also depends on the nature of the objects being used
as dict keys. Dict operations invoke the hashing and
comparison methods of the keys, which in general can
execute arbitrary Python code.

If the keys are elementary built-in types such as
strings or ints, then dict operations will *probably*
be atomic. But I think I'd use a lock anyway, to be
on the safe side.

--
Greg
From: mk on 3 Mar 2010 11:10
Veloz wrote:

> Unless I missed where you guys were going, I think we got off the main
> point. The main question at hand was this: what's the best way (heck,
> any way) to implement a sort of "peek" whereby a number of processes
> can write results to some common "object" and some other process can
> "peek" into this object, looking for specific items they're interested
> in?

I have used threads so far and not multiprocessing (much), so I'll
describe it using threads:

If I were you, I'd do it the simplest possible way, with dictionary and
lock, in the following way:

1. Have shared "queue" of tasks, say, a dictionary.

2. Each worker thread and central "queue manager" access the queue in
the same manner:

....do your stuff, prepare yourself

acquire a lock

update or read value from queue

....don't do much here to be able release lock quickly

release the lock

....process value

If you had your dictionary simple, it could work like Daniel Stutzbach
wrote, that updates are thread safe and perhaps you could even dispense
with locks and just read the dictionary; and if you wanted to use locks,
for "peek"ing you could write a function that could e.g. acquire a lock,
do a copy of the queue, release the lock, and return the copy to the
object wanting to examine the queue.

I have used this approach (although with a list, not a dictionary -- I
haven't had the need to do extensive searching by keys) and so far it
has worked for me.

Regards,
mk

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: os.fdopen() issue in Python 3.1?
Next: freebsd and multiprocessing