Skip to contentSkip to conversation

: add libxl python binding

PATCH Superseded Gianni Tedesco 2 participants 7 messages Sep 10, 2010 – Sep 14, 2010 1 message not in the archive Arm/actions semi architecture

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 : add libxl python binding

Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

--

 tools/python/genwrap.py             |  375 ++++++++++++++++++++++++++
 tools/python/xen/lowlevel/xl/xl.c   |  512 ++++++++++++++++++++++++++++++++++++
 tools/python/Makefile               |    9 
 tools/python/setup.py               |   19 +
 4 files changed, 912 insertions(+), 3 deletions(-)

--

tools/python/Makefile +8 -1
diff -r b8c925673644 tools/python/Makefile
@@ -19,7 +19,12 @@ genpath-target = $(call buildmakevars2fi
 $(eval $(genpath-target))
 
 .PHONY: build buildpy
-buildpy: genpath 
+buildpy: genpath genwrap.py $(XEN_ROOT)/tools/libxl/libxl.idl \
+		$(XEN_ROOT)/tools/libxl/libxltypes.py
+	PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \
+		$(XEN_ROOT)/tools/libxl/libxl.idl \
+		xen/lowlevel/xl/_pyxl_types.h \
+		xen/lowlevel/xl/_pyxl_types.c
 	CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build
 
 build: buildpy refresh-pot refresh-po $(CATALOGS)
@@ -85,6 +90,8 @@ test:
 clean:
 	rm -f $(XENPATH)
 	rm -rf build *.pyc *.pyo *.o *.a *~ $(CATALOGS) xen/util/auxbin.pyc
+	rm -f xen/lowlevel/xl/_pyxl_types.h
+	rm -f xen/lowlevel/xl/_pyxl_types.c
 	rm -f $(DEPS)
 
 -include $(DEPS)
tools/python/genwrap.py new file +375 -0

375 lines not on this page · raw message

tools/python/setup.py +17 -2
diff -r b8c925673644 tools/python/setup.py
@@ -9,14 +9,23 @@ extra_compile_args  = [ "-fno-strict-ali
 include_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
                  XEN_ROOT + "/tools/include",
+                 XEN_ROOT + "/tools/libxl",
                  ]
 
 library_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
+                 XEN_ROOT + "/tools/libxl",
+                 XEN_ROOT + "/tools/blktap2/control",
                  ]
 
 libraries = [ "xenctrl", "xenguest", "xenstore" ]
 
+plat = os.uname()[0]
+if plat == 'Linux':
+    uuid_libs = ["uuid"]
+else:
+    uuid_libs = []
+
 xc = Extension("xc",
                extra_compile_args = extra_compile_args,
                include_dirs       = include_dirs + [ "xen/lowlevel/xc" ],
@@ -83,8 +92,14 @@ netlink = Extension("netlink",
                     sources            = [ "xen/lowlevel/netlink/netlink.c",
                                            "xen/lowlevel/netlink/libnetlink.c"])
 
-modules = [ xc, xs, ptsname, acm, flask ]
-plat = os.uname()[0]
+xl = Extension("xl",
+               extra_compile_args = extra_compile_args,
+               include_dirs       = include_dirs + [ "xen/lowlevel/xl" ],
+               library_dirs       = library_dirs,
+               libraries          = libraries + ["xenlight", "blktapctl" ] + uuid_libs,
+               sources            = [ "xen/lowlevel/xl/xl.c", "xen/lowlevel/xl/_pyxl_types.c" ])
+
+modules = [ xc, xs, ptsname, acm, flask, xl ]
 if plat == 'SunOS':
     modules.extend([ scf, process ])
 if plat == 'Linux':
tools/python/xen/lowlevel/xl/xl.c new file +512 -0

512 lines not on this page · raw message

#1 lore raw
Quoted 17 lines · Gianni Tedesco

Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Quoted 5 lines

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.

One perhaps interesting tip:

tools/python/genwrap.py line 100

Quoted 3 lines
tools/python/genwrap.py new file +3 -0
+    funcs="""static void Py%s_dealloc(Py_%s *self)
+{
+%s    self->ob_type->tp_free((PyObject *)self);

[...etc...]

Quoted 2 lines

+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))

String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)

e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}

etc

I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a pyxl.h and/or xl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.

Ian.

#2 lore raw
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco

Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Quoted 5 lines

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.

One perhaps interesting tip:

Review

Quoted 3 lines
(quoted diff) +3 -0
+    funcs="""static void Py%s_dealloc(Py_%s *self)
+{
+%s    self->ob_type->tp_free((PyObject *)self);

[...etc...]

Quoted 8 lines
Quoted 2 lines

+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))

String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)

Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.

Quoted 11 lines

e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}

etc

I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a pyxl.h and/or xl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.

Yeah, the definitions of the genwrap__*_(get|set) funcs can move to xl.c which makes it a lot tidier.

With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in libxltypes.c so that we could, for example, say

x = Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)

which would allow, eg: pci = xl.device_pci(); pci.parse("00:11.2") or allow a non keyword initialiser, eg: pci = xl.device_pci("00:11.2")

Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

--

tools/python/Makefile +8 -1
diff -r 57eaf7872ad9 tools/python/Makefile
@@ -19,7 +19,12 @@ genpath-target = $(call buildmakevars2fi
 $(eval $(genpath-target))
 
 .PHONY: build buildpy
-buildpy: genpath 
+buildpy: genpath genwrap.py $(XEN_ROOT)/tools/libxl/libxl.idl \
+		$(XEN_ROOT)/tools/libxl/libxltypes.py
+	PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \
+		$(XEN_ROOT)/tools/libxl/libxl.idl \
+		xen/lowlevel/xl/_pyxl_types.h \
+		xen/lowlevel/xl/_pyxl_types.c
 	CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build
 
 build: buildpy refresh-pot refresh-po $(CATALOGS)
@@ -85,6 +90,8 @@ test:
 clean:
 	rm -f $(XENPATH)
 	rm -rf build *.pyc *.pyo *.o *.a *~ $(CATALOGS) xen/util/auxbin.pyc
+	rm -f xen/lowlevel/xl/_pyxl_types.h
+	rm -f xen/lowlevel/xl/_pyxl_types.c
 	rm -f $(DEPS)
 
 -include $(DEPS)
tools/python/genwrap.py new file +280 -0

280 lines not on this page · raw message

tools/python/setup.py +17 -2
diff -r 57eaf7872ad9 tools/python/setup.py
@@ -9,14 +9,23 @@ extra_compile_args  = [ "-fno-strict-ali
 include_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
                  XEN_ROOT + "/tools/include",
+                 XEN_ROOT + "/tools/libxl",
                  ]
 
 library_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
+                 XEN_ROOT + "/tools/libxl",
+                 XEN_ROOT + "/tools/blktap2/control",
                  ]
 
 libraries = [ "xenctrl", "xenguest", "xenstore" ]
 
+plat = os.uname()[0]
+if plat == 'Linux':
+    uuid_libs = ["uuid"]
+else:
+    uuid_libs = []
+
 xc = Extension("xc",
                extra_compile_args = extra_compile_args,
                include_dirs       = include_dirs + [ "xen/lowlevel/xc" ],
@@ -83,8 +92,14 @@ netlink = Extension("netlink",
                     sources            = [ "xen/lowlevel/netlink/netlink.c",
                                            "xen/lowlevel/netlink/libnetlink.c"])
 
-modules = [ xc, xs, ptsname, acm, flask ]
-plat = os.uname()[0]
+xl = Extension("xl",
+               extra_compile_args = extra_compile_args,
+               include_dirs       = include_dirs + [ "xen/lowlevel/xl" ],
+               library_dirs       = library_dirs,
+               libraries          = libraries + ["xenlight", "blktapctl" ] + uuid_libs,
+               sources            = [ "xen/lowlevel/xl/xl.c", "xen/lowlevel/xl/_pyxl_types.c" ])
+
+modules = [ xc, xs, ptsname, acm, flask, xl ]
 if plat == 'SunOS':
     modules.extend([ scf, process ])
 if plat == 'Linux':
tools/python/xen/lowlevel/xl/xl.c new file +609 -0

609 lines not on this page · raw message

#3 lore raw
Quoted 61 lines · Gianni Tedesco
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco

Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Quoted 5 lines

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.

One perhaps interesting tip:

tools/python/genwrap.py line 100

Quoted 3 lines
tools/python/genwrap.py new file +3 -0
+    funcs="""static void Py%s_dealloc(Py_%s *self)
+{
+%s    self->ob_type->tp_free((PyObject *)self);

[...etc...]

Quoted 8 lines
Quoted 2 lines

+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))

String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)

Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.

Quoted 11 lines

e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}

etc

I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a pyxl.h and/or xl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.

Yeah, the definitions of the genwrap__*_(get|set) funcs can move to xl.c which makes it a lot tidier.

Quoted 8 lines

With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in libxltypes.c so that we could, for example, say

x = Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)

which would allow, eg: pci = xl.device_pci(); pci.parse("00:11.2") or allow a non keyword initialiser, eg: pci = xl.device_pci("00:11.2")

I don't know that adding language specific stuff to the IDL is any better than just recognising specific type names which would like special handling in the generator.

Quoted 2 lines

Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.

Agreed.

Two things though, firstly the patch to Makefile didn't apply for me in any version you've sent because tabs have become spaces. Might be an issue on my end though.

Secondly:
xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_set':
xen/lowlevel/xl/xl.c:164: error: implicit declaration of function 'PyByteArray_Check' xen/lowlevel/xl/xl.c:165: error: implicit declaration of function 'PyByteArray_Size' xen/lowlevel/xl/xl.c:166: error: implicit declaration of function 'PyByteArray_AsString' xen/lowlevel/xl/xl.c:166: error: assignment makes pointer from integer without a cast xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_get':
xen/lowlevel/xl/xl.c:195: error: implicit declaration of function 'PyByteArray_FromStringAndSize' xen/lowlevel/xl/xl.c:195: error: return makes pointer from integer without a cast
error: command 'gcc' failed with exit status 1

This is with Python 2.5 (default on Debian Lenny so, unfortunately, still worth supporting). I guess PyByteArray is new in >2.5? I don't see a cunning way around it other than making MAC and UUID first class objects instead of wrappers around byte array -- does that make the syntax in the user icky?

Oh, and I think you want to add the new generated files to .hgignore.

Ian.

#4 lore raw
Quoted 8 lines · Ian Campbell
Quoted 2 lines

Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.

Agreed.

Two things though, firstly the patch to Makefile didn't apply for me in any version you've sent because tabs have become spaces. Might be an issue on my end though.

Hmm, not sure what the culprit is but it's happened to me before, could be evolution or could be xclip... I should probably hg mail more often but find it a bit cumbersome tbh. Wierd because the ocaml stuff is tabs and I assume that worked fine. It may even be vim come to think of it.

Quoted 11 lines

Secondly:
xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_set':
xen/lowlevel/xl/xl.c:164: error: implicit declaration of function 'PyByteArray_Check' xen/lowlevel/xl/xl.c:165: error: implicit declaration of function 'PyByteArray_Size' xen/lowlevel/xl/xl.c:166: error: implicit declaration of function 'PyByteArray_AsString' xen/lowlevel/xl/xl.c:166: error: assignment makes pointer from integer without a cast xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_get':
xen/lowlevel/xl/xl.c:195: error: implicit declaration of function 'PyByteArray_FromStringAndSize' xen/lowlevel/xl/xl.c:195: error: return makes pointer from integer without a cast
error: command 'gcc' failed with exit status 1

Quoted 5 lines

This is with Python 2.5 (default on Debian Lenny so, unfortunately, still worth supporting). I guess PyByteArray is new in >2.5? I don't see a cunning way around it other than making MAC and UUID first class objects instead of wrappers around byte array -- does that make the syntax in the user icky?

Well, a binary string will work just as well, making these a first class object just means adding a parse and unparse method which will be doing the same thing anyway. I'm pretty sure there'll be a define to check availability.

Quoted 1 line

Oh, and I think you want to add the new generated files to .hgignore.

Ah yes, of course

#5 lore raw
In reply to Ian Campbell
Quoted 66 lines · Ian Campbell
Quoted 65 lines · Gianni Tedesco
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco

Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.

Quoted 5 lines

Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.

One perhaps interesting tip:

tools/python/genwrap.py line 100

Quoted 3 lines
tools/python/genwrap.py new file +3 -0
+    funcs="""static void Py%s_dealloc(Py_%s *self)
+{
+%s    self->ob_type->tp_free((PyObject *)self);

[...etc...]

Quoted 8 lines
Quoted 2 lines

+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))

String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)

Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.

Quoted 11 lines

e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}

etc

I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a pyxl.h and/or xl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.

Yeah, the definitions of the genwrap__*_(get|set) funcs can move to xl.c which makes it a lot tidier.

With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in libxltypes.c so that we could, for example, say

Quoted 8 lines
Quoted 4 lines

x = Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)

which would allow, eg: pci = xl.device_pci(); pci.parse("00:11.2") or allow a non keyword initialiser, eg: pci = xl.device_pci("00:11.2")

I don't know that adding language specific stuff to the IDL is any better than just recognising specific type names which would like special handling in the generator.

Actually I think enabling inheritance in the generated objects is the way to do this. It just requires a bit of glue code both in the C and in any inherited python wrappers to make it all work right.

Will have code for this in the next rev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 8 lines · Ian Campbell
Quoted 2 lines · Gianni Tedesco

+ ret = filter(lambda x:True, types) # deep copy

I think this is more normally written "ret = list(types)", "ret = types[:]", or:
from copy import copy
ret = copy(types)

I think i prefer types[:] (forgot about that) - copy module is a nightmare and IMO nastiest part of python language is semantics viz deep vs shallow copies. Does list() even do deep copy?? (I had no idea)

Quoted 3 lines

Although given that the caller does:
types = tree_frob(types)
why is the deep copy necessary? tree_frob could work in-place.

Well, it's going to end up doing the tree flattening for the keyed union type that needs to be handled. Just makes it simpler than keeping around an original list that may or may not contain what you think it does at that point.