Prev: multiprocessing callbacks?
Next: Interesting things of 'getattr' and 'setattr'
From: makobu on 15 Dec 2009 03:56
I have a function that makes two subprocess.Popen() calls on a file.

I have 8 cores. I need 8 instances of that function running in
parallel at any given time till all the files are worked on.
Can the multiprocessing module do this? If so, whats the best method?

A technical overview of how the multiprocessing module actually works
would also be really helpful.

regards,
mak.
From: Aahz on 31 Dec 2009 17:22
In article <79854c42-b2af-4adb-8967-3dc5e4ac0d2a(a)l13g2000yqb.googlegroups.com>,
makobu <makobu.mwambiriro(a)gmail.com> wrote:
>
>I have a function that makes two subprocess.Popen() calls on a file.
>
>I have 8 cores. I need 8 instances of that function running in
>parallel at any given time till all the files are worked on.
>Can the multiprocessing module do this? If so, whats the best method?

You don't quite explicitly say so, but it sounds like you have multiple
files. In which case, yes, it should be reasonably straightforward to
use multiprocessing; I haven't used it myself, but what you want must be
whatever is equivalent to Queue.Queue().

>A technical overview of how the multiprocessing module actually works
>would also be really helpful.

Not having any clue what you already know makes that rather a daunting
prospect, as indicated by the lack of replies. Maybe you could
summarize?
--
Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.
From: Glazner on 1 Jan 2010 02:26
On Dec 15 2009, 10:56 am, makobu <makobu.mwambir...(a)gmail.com> wrote:
> I have a function that makes two subprocess.Popen() calls on a file.
>
> I have 8 cores. I need 8 instances of that function running in
> parallel at any given time till all the files are worked on.
> Can the multiprocessing module do this? If so, whats the best method?
>
> A technical overview of how the multiprocessing module actually works
> would also be really helpful.
>
> regards,
> mak.

I guess you need the Map-Reduce pattern.
It appears like multiprocessing.Pool will do the trick.

def doSomething(f):
pass

pool = Pool(8)# 8 processes 1 per core

files = ['pop.txt','looper.txt','foo.bar']
results = pool.map(doSomething,files) #this does all the job

got it?
 | 
Pages: 1
Prev: multiprocessing callbacks?
Next: Interesting things of 'getattr' and 'setattr'