Prev: Deditor -- pythonic text-editor
Next: text to DB
From: Mikael Liljeroth on 13 Aug 2010 07:22
Hi, this is my first contact with the python community so I hope this
is the right place for my two questions.
They are both related to embedding and extending python.

I'm writing a program that dynamically loads c++ modules that extend
the functionality of my program.
These modules are controlled by scripts. Currently I have embedded the
v8 javascript engine but I would also like to support scripts written
in python. I have a special interface between the c++ modules and the
scripts to be able to support different scripting languages. Scripts
are separated into different 'applications' that are executed in one
process triggered by external events (no threading, since I'm working
in an event driven system). The scripts should execute independently
of each other. I solved this with v8 by switching between the
applications' contexts. However, it seems like python has some issues
with this.

1. To do some basic context switching I assume I should use a sub
interpreter ie create a new thread state (without any actual threads).
I cannot find any information on how to do this except creating the
new thread state. Is this the preferred way of doing this, if so, how
do I switch between different thread states and execute scripts in
them? I assume scripts running in one thread state are isolated from
scripts running in other thread states, like global variables and such
(except for the GIL thing?) ? I do not care much about security
problems like file descriptor issues and extension issues right now.

2. When a new C++ module is loaded into my program I want to create a
representation of it in Python. Not as a type, but rather just return
a single object instance reference of a new type representing the
exported attributes and methods of the module's C++ class. In other
words, I want to be able to do something like this obj =
application.getObject("objectId"); where objectId is the name of an
instance of that c++ module type. I do not want the script to be able
to create new instances of this type, since this is done via xml in my
program. Hence I would also like to be able to associate my C++ object
with my returned python object to be able to manipulate it when
methods are called on 'obj'. So how do I define a new type on the fly
and create an instance of it, associated with a c++ pointer and return
it in the c function representing the python method 'getObject' in my
application module?

Hope this makes any sense, let me know if something is unclear.

Regards
Mikael Liljeroth
From: Thomas Jollans on 13 Aug 2010 12:50
On 2010-08-13 13:22, Mikael Liljeroth wrote:
> Hi, this is my first contact with the python community so I hope this
> is the right place for my two questions.
> They are both related to embedding and extending python.
>
> I'm writing a program that dynamically loads c++ modules that extend
> the functionality of my program.
> These modules are controlled by scripts. Currently I have embedded the
> v8 javascript engine but I would also like to support scripts written
> in python. I have a special interface between the c++ modules and the
> scripts to be able to support different scripting languages. Scripts
> are separated into different 'applications' that are executed in one
> process triggered by external events (no threading, since I'm working
> in an event driven system). The scripts should execute independently
> of each other. I solved this with v8 by switching between the
> applications' contexts. However, it seems like python has some issues
> with this.
>
> 1. To do some basic context switching I assume I should use a sub
> interpreter ie create a new thread state (without any actual threads).
> I cannot find any information on how to do this except creating the
> new thread state. Is this the preferred way of doing this, if so, how
> do I switch between different thread states and execute scripts in
> them? I assume scripts running in one thread state are isolated from
> scripts running in other thread states, like global variables and such
> (except for the GIL thing?) ? I do not care much about security
> problems like file descriptor issues and extension issues right now.
>
Py_NewInterpreter does indeed sound like what you want. I have a feeling
you're going to run into problems here sooner or later, say, when
extension modules that aren't yours use static variables.

As for switching interpreters, after skimming the docs, I'm guessing
you're looking for PyEval_AcquireThread

http://docs.python.org/py3k/c-api/init.html#PyEval_AcquireThread

> 2. When a new C++ module is loaded into my program I want to create a
> representation of it in Python. Not as a type, but rather just return
> a single object instance reference of a new type representing the
> exported attributes and methods of the module's C++ class. In other
> words, I want to be able to do something like this obj =
> application.getObject("objectId"); where objectId is the name of an
> instance of that c++ module type. I do not want the script to be able
> to create new instances of this type, since this is done via xml in my
> program. Hence I would also like to be able to associate my C++ object
> with my returned python object to be able to manipulate it when
> methods are called on 'obj'. So how do I define a new type on the fly
> and create an instance of it, associated with a c++ pointer and return
> it in the c function representing the python method 'getObject' in my
> application module?
>

How difficult this is obviously depends somewhat on how varied the types
you create are, but in principle, there's no difference between creating
a type statically and creating one on-the-fly: all you need to do to
create a type is create a PyTypeObject structure with suitable function
pointers. If you want to make it impossible to create a second instance
of your type, you can simply create a tp_new or tp_init that raises an
exception.

If you have a good look at
http://docs.python.org/py3k/extending/newtypes.html , you'll notice that
most of the "creating a type" involves writing a static PyTypeObject
structure -- you'll just have to allocate that yourself, and you should
be good.

 | 
Pages: 1
Prev: Deditor -- pythonic text-editor
Next: text to DB