Prev: Could not import external source folder in Pydev
Next: Efficiency/style issues of import <module> vs. from <module>import <name>, ...
From: Matteo Landi on 17 Jun 2010 09:23
Some weeks ago, here on the mailing list I read about picloud[1], a
python library used for cloud-computing; I was impressed by its
simplicity, here is an example:

>>>import cloud
>>>def square(x):
.... return x * x
>>>cloud.call(square, 10)
>>>cloud.result()
100

So, I tried to figure out how to achieve the same result, i.e. define a
local generic function and send it somewhere for remote execution, and
the get the result back.
So I thought about serialization (pickle): I made a few tries and it
seemed to work.. but I was wrong, because I never tried to load pickled
data from another session different from the one used to pickle data
itself. If you try and pickle a function, it is not pickled as a whole,
indeed, once you unpickle it, it will raise an exception telling you
that the target function was not found in the current module.

So I'm here, with nothing in my hands; how would you implement this?

Thanks in advance.

[1] http://www.picloud.com/

--
Matteo Landi
http://www.matteolandi.net

From: Bruno Desthuilliers on 17 Jun 2010 10:01
Matteo Landi a écrit :
> Some weeks ago, here on the mailing list I read about picloud[1], a
> python library used for cloud-computing; I was impressed by its
> simplicity, here is an example:
>
>>>> import cloud
>>>> def square(x):
> ... return x * x
>>>> cloud.call(square, 10)
>>>> cloud.result()
> 100
>
> So, I tried to figure out how to achieve the same result, i.e. define a
> local generic function and send it somewhere for remote execution, and
> the get the result back.
> So I thought about serialization (pickle): I made a few tries and it
> seemed to work.. but I was wrong, because I never tried to load pickled
> data from another session different from the one used to pickle data
> itself. If you try and pickle a function, it is not pickled as a whole,
> indeed, once you unpickle it, it will raise an exception telling you
> that the target function was not found in the current module.
>
> So I'm here, with nothing in my hands; how would you implement this?

Hint found on picloud's website:

"""
PiCloud's programming library, cloud, automatically transfers the
necessary source code, byte code, execution state, and data to run your
function on our cluster.
"""

From: Paul Rubin on 17 Jun 2010 10:37
Matteo Landi <landimatte(a)gmail.com> writes:
> If you try and pickle a function, it is not pickled as a whole,
> indeed, once you unpickle it, it will raise an exception telling you
> that the target function was not found in the current module.
>
> So I'm here, with nothing in my hands; how would you implement this?

Use marshal rather than pickle. Note that requires both ends to be
running the same python version. Be aware of the obvious security
issues of running code that came from remote machine, either because the
remote machine itself may be untrusted or compromised, or because the
network between the two machines may be compromised (remote machine is
being spoofed or commnications are being modified).
From: Matteo Landi on 17 Jun 2010 11:06
On Thu, 2010-06-17 at 07:37 -0700, Paul Rubin wrote:
> Matteo Landi <landimatte(a)gmail.com> writes:
> > If you try and pickle a function, it is not pickled as a whole,
> > indeed, once you unpickle it, it will raise an exception telling you
> > that the target function was not found in the current module.
> >
> > So I'm here, with nothing in my hands; how would you implement this?
>
> Use marshal rather than pickle. Note that requires both ends to be

I could be wrong, but it seems functions are not marshable objects, is
it right?

> running the same python version. Be aware of the obvious security
> issues of running code that came from remote machine, either because the
> remote machine itself may be untrusted or compromised, or because the
> network between the two machines may be compromised (remote machine is
> being spoofed or commnications are being modified).

--
Matteo Landi
http://www.matteolandi.net

From: Paul Rubin on 17 Jun 2010 11:18
Matteo Landi <landimatte(a)gmail.com> writes:
> I could be wrong, but it seems functions are not marshable objects, is
> it right?

Hmm, you're right, you can marshal code objects, but you can't marshal a
function directly. It's been a while since I've used marshal and I
forgot how it works. You might be able to concoct something around it,
but it could be messy. It may be simplest to send the other side a
python source file that it can import.
 |  Next  |  Last
Pages: 1 2 3
Prev: Could not import external source folder in Pydev
Next: Efficiency/style issues of import <module> vs. from <module>import <name>, ...