: add libxl python binding
PATCH v2 1/1 : add libxl python binding
Changes since v1:
- Interpret keyword arguments in initializers: eg. xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del and pci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention
-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.
--
tools/python/genwrap.py | 375 ++++++++++++++++++++++++++ tools/python/xen/lowlevel/xl/xl.c | 512 ++++++++++++++++++++++++++++++++++++ tools/python/Makefile | 9 tools/python/setup.py | 19 + 4 files changed, 912 insertions(+), 3 deletions(-)
--
tools/python/Makefile +8 -1
@@ -19,7 +19,12 @@ genpath-target = $(call buildmakevars2fi $(eval $(genpath-target)) .PHONY: build buildpy -buildpy: genpath +buildpy: genpath genwrap.py $(XEN_ROOT)/tools/libxl/libxl.idl \ + $(XEN_ROOT)/tools/libxl/libxltypes.py + PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \ + $(XEN_ROOT)/tools/libxl/libxl.idl \ + xen/lowlevel/xl/_pyxl_types.h \ + xen/lowlevel/xl/_pyxl_types.c CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build build: buildpy refresh-pot refresh-po $(CATALOGS)
@@ -85,6 +90,8 @@ test: clean: rm -f $(XENPATH) rm -rf build *.pyc *.pyo *.o *.a *~ $(CATALOGS) xen/util/auxbin.pyc + rm -f xen/lowlevel/xl/_pyxl_types.h + rm -f xen/lowlevel/xl/_pyxl_types.c rm -f $(DEPS) -include $(DEPS)
tools/python/genwrap.py new file +375 -0
375 lines not on this page · raw message
tools/python/setup.py +17 -2
@@ -9,14 +9,23 @@ extra_compile_args = [ "-fno-strict-ali include_dirs = [ XEN_ROOT + "/tools/libxc", XEN_ROOT + "/tools/xenstore", XEN_ROOT + "/tools/include", + XEN_ROOT + "/tools/libxl", ] library_dirs = [ XEN_ROOT + "/tools/libxc", XEN_ROOT + "/tools/xenstore", + XEN_ROOT + "/tools/libxl", + XEN_ROOT + "/tools/blktap2/control", ] libraries = [ "xenctrl", "xenguest", "xenstore" ] +plat = os.uname()[0] +if plat == 'Linux': + uuid_libs = ["uuid"] +else: + uuid_libs = [] + xc = Extension("xc", extra_compile_args = extra_compile_args, include_dirs = include_dirs + [ "xen/lowlevel/xc" ],
@@ -83,8 +92,14 @@ netlink = Extension("netlink", sources = [ "xen/lowlevel/netlink/netlink.c", "xen/lowlevel/netlink/libnetlink.c"]) -modules = [ xc, xs, ptsname, acm, flask ] -plat = os.uname()[0] +xl = Extension("xl", + extra_compile_args = extra_compile_args, + include_dirs = include_dirs + [ "xen/lowlevel/xl" ], + library_dirs = library_dirs, + libraries = libraries + ["xenlight", "blktapctl" ] + uuid_libs, + sources = [ "xen/lowlevel/xl/xl.c", "xen/lowlevel/xl/_pyxl_types.c" ]) + +modules = [ xc, xs, ptsname, acm, flask, xl ] if plat == 'SunOS': modules.extend([ scf, process ]) if plat == 'Linux':
tools/python/xen/lowlevel/xl/xl.c new file +512 -0
512 lines not on this page · raw message
Quoted 17 lines · Gianni Tedesco
Changes since v1:
- Interpret keyword arguments in initializers: eg.xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del andpci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Quoted 5 lines
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between
libxl_file_referenceand a python file object is required.
I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.
One perhaps interesting tip:
tools/python/genwrap.py line 100
Quoted 3 lines
tools/python/genwrap.py new file +3
-0+ funcs="""static void Py%s_dealloc(Py_%s *self) +{ +%s self->ob_type->tp_free((PyObject *)self);
[...etc...]
Quoted 2 lines
+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))
String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)
e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}
etc
I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a pyxl.h and/or xl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.
Ian.
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco
Changes since v1:
- Interpret keyword arguments in initializers: eg.xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del andpci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Quoted 5 lines
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between
libxl_file_referenceand a python file object is required.I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.
One perhaps interesting tip:
Review
Quoted 3 lines
(quoted diff) +3
-0+ funcs="""static void Py%s_dealloc(Py_%s *self) +{ +%s self->ob_type->tp_free((PyObject *)self);[...etc...]
Quoted 8 lines
Quoted 2 lines
+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)
Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.
Quoted 11 lines
e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}etc
I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a
pyxl.hand/orxl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.
Yeah, the definitions of the genwrap__*_(get|set) funcs can move to xl.c which makes it a lot tidier.
With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in libxltypes.c so that we could, for example, say
x = Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)
which would allow, eg: pci = xl.device_pci(); pci.parse("00:11.2") or allow a non keyword initialiser, eg: pci = xl.device_pci("00:11.2")
Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.
-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.
--
tools/python/Makefile +8 -1
@@ -19,7 +19,12 @@ genpath-target = $(call buildmakevars2fi $(eval $(genpath-target)) .PHONY: build buildpy -buildpy: genpath +buildpy: genpath genwrap.py $(XEN_ROOT)/tools/libxl/libxl.idl \ + $(XEN_ROOT)/tools/libxl/libxltypes.py + PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \ + $(XEN_ROOT)/tools/libxl/libxl.idl \ + xen/lowlevel/xl/_pyxl_types.h \ + xen/lowlevel/xl/_pyxl_types.c CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build build: buildpy refresh-pot refresh-po $(CATALOGS)
@@ -85,6 +90,8 @@ test: clean: rm -f $(XENPATH) rm -rf build *.pyc *.pyo *.o *.a *~ $(CATALOGS) xen/util/auxbin.pyc + rm -f xen/lowlevel/xl/_pyxl_types.h + rm -f xen/lowlevel/xl/_pyxl_types.c rm -f $(DEPS) -include $(DEPS)
tools/python/genwrap.py new file +280 -0
280 lines not on this page · raw message
tools/python/setup.py +17 -2
@@ -9,14 +9,23 @@ extra_compile_args = [ "-fno-strict-ali include_dirs = [ XEN_ROOT + "/tools/libxc", XEN_ROOT + "/tools/xenstore", XEN_ROOT + "/tools/include", + XEN_ROOT + "/tools/libxl", ] library_dirs = [ XEN_ROOT + "/tools/libxc", XEN_ROOT + "/tools/xenstore", + XEN_ROOT + "/tools/libxl", + XEN_ROOT + "/tools/blktap2/control", ] libraries = [ "xenctrl", "xenguest", "xenstore" ] +plat = os.uname()[0] +if plat == 'Linux': + uuid_libs = ["uuid"] +else: + uuid_libs = [] + xc = Extension("xc", extra_compile_args = extra_compile_args, include_dirs = include_dirs + [ "xen/lowlevel/xc" ],
@@ -83,8 +92,14 @@ netlink = Extension("netlink", sources = [ "xen/lowlevel/netlink/netlink.c", "xen/lowlevel/netlink/libnetlink.c"]) -modules = [ xc, xs, ptsname, acm, flask ] -plat = os.uname()[0] +xl = Extension("xl", + extra_compile_args = extra_compile_args, + include_dirs = include_dirs + [ "xen/lowlevel/xl" ], + library_dirs = library_dirs, + libraries = libraries + ["xenlight", "blktapctl" ] + uuid_libs, + sources = [ "xen/lowlevel/xl/xl.c", "xen/lowlevel/xl/_pyxl_types.c" ]) + +modules = [ xc, xs, ptsname, acm, flask, xl ] if plat == 'SunOS': modules.extend([ scf, process ]) if plat == 'Linux':
tools/python/xen/lowlevel/xl/xl.c new file +609 -0
609 lines not on this page · raw message
Quoted 61 lines · Gianni Tedesco
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco
Changes since v1:
- Interpret keyword arguments in initializers: eg.xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del andpci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Quoted 5 lines
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between
libxl_file_referenceand a python file object is required.I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.
One perhaps interesting tip:
tools/python/genwrap.py line 100
Quoted 3 lines
tools/python/genwrap.py new file +3
-0+ funcs="""static void Py%s_dealloc(Py_%s *self) +{ +%s self->ob_type->tp_free((PyObject *)self);[...etc...]
Quoted 8 lines
Quoted 2 lines
+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)
Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.
Quoted 11 lines
e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}etc
I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a
pyxl.hand/orxl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.Yeah, the definitions of the genwrap__*_(get|set) funcs can move to
xl.cwhich makes it a lot tidier.
Quoted 8 lines
With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in
libxltypes.cso that we could, for example, sayx =
Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)which would allow, eg: pci =
xl.device_pci();pci.parse("00:11.2")or allow a non keyword initialiser, eg: pci =xl.device_pci("00:11.2")
I don't know that adding language specific stuff to the IDL is any better than just recognising specific type names which would like special handling in the generator.
Quoted 2 lines
Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.
Agreed.
Two things though, firstly the patch to Makefile didn't apply for me in any version you've sent because tabs have become spaces. Might be an issue on my end though.
Secondly:xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_set':xen/lowlevel/xl/xl.c:164: error: implicit declaration of function 'PyByteArray_Check' xen/lowlevel/xl/xl.c:165: error: implicit declaration of function 'PyByteArray_Size' xen/lowlevel/xl/xl.c:166: error: implicit declaration of function 'PyByteArray_AsString' xen/lowlevel/xl/xl.c:166: error: assignment makes pointer from integer without a cast xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_get':xen/lowlevel/xl/xl.c:195: error: implicit declaration of function 'PyByteArray_FromStringAndSize' xen/lowlevel/xl/xl.c:195: error: return makes pointer from integer without a cast
error: command 'gcc' failed with exit status 1
This is with Python 2.5 (default on Debian Lenny so, unfortunately, still worth supporting). I guess PyByteArray is new in >2.5? I don't see a cunning way around it other than making MAC and UUID first class objects instead of wrappers around byte array -- does that make the syntax in the user icky?
Oh, and I think you want to add the new generated files to .hgignore.
Ian.
Quoted 8 lines · Ian Campbell
Quoted 2 lines
Anyway, I don't think this stops us getting the code in the tree cos such things can be done in a future patch.
Agreed.
Two things though, firstly the patch to
Makefiledidn't apply for me in any version you've sent because tabs have become spaces. Might be an issue on my end though.
Hmm, not sure what the culprit is but it's happened to me before, could be evolution or could be xclip... I should probably hg mail more often but find it a bit cumbersome tbh. Wierd because the ocaml stuff is tabs and I assume that worked fine. It may even be vim come to think of it.
Quoted 11 lines
Secondly:
xen/lowlevel/xl/xl.c: In function 'fixed_bytearray_set':xen/lowlevel/xl/xl.c:164: error: implicit declaration of function 'PyByteArray_Check'xen/lowlevel/xl/xl.c:165: error: implicit declaration of function 'PyByteArray_Size'xen/lowlevel/xl/xl.c:166: error: implicit declaration of function 'PyByteArray_AsString'xen/lowlevel/xl/xl.c:166: error: assignment makes pointer from integer without a castxen/lowlevel/xl/xl.c: In function 'fixed_bytearray_get':xen/lowlevel/xl/xl.c:195: error: implicit declaration of function 'PyByteArray_FromStringAndSize'xen/lowlevel/xl/xl.c:195: error: return makes pointer from integer without a cast
error: command 'gcc' failed with exit status 1
Quoted 5 lines
This is with Python 2.5 (default on Debian Lenny so, unfortunately, still worth supporting). I guess PyByteArray is new in >2.5? I don't see a cunning way around it other than making MAC and UUID first class objects instead of wrappers around byte array -- does that make the syntax in the user icky?
Well, a binary string will work just as well, making these a first class object just means adding a parse and unparse method which will be doing the same thing anyway. I'm pretty sure there'll be a define to check availability.
Quoted 1 line
Oh, and I think you want to add the new generated files to .hgignore.
Ah yes, of course
Quoted 66 lines · Ian Campbell
Quoted 65 lines · Gianni Tedesco
Quoted 32 lines · Ian Campbell
Quoted 17 lines · Gianni Tedesco
Changes since v1:
- Interpret keyword arguments in initializers: eg.xl.device_pci(bus=1, dev=2, fn=3)
- Implement domain pause/unpause/rename/pci_add/pci_del andpci_parse
- Proper type-checking in all (I hope) code-paths
- Improve API for accessing auto-generated code and types
Changes since RFC:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but serveral methods are implemented and tested. Those which are implemented provide examples of the two or three basic patterns that most future methods should follow.
Quoted 5 lines
Over 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between
libxl_file_referenceand a python file object is required.I'm not qualified to comment on the actual generated bindings themselves but the tiny bit which remains looks good to me.
One perhaps interesting tip:
tools/python/genwrap.py line 100
Quoted 3 lines
tools/python/genwrap.py new file +3
-0+ funcs="""static void Py%s_dealloc(Py_%s *self) +{ +%s self->ob_type->tp_free((PyObject *)self);[...etc...]
Quoted 8 lines
Quoted 2 lines
+}
+"""%((ty.rawname, ty.rawname, dtor) + tuple(ty.rawname for x in range(5)))String formatting in python can take a dictionary and then %(foo)s will expand by looking for the key 'foo' in the dict which would likely simplify (and help self-document) stuff with this pattern and get rid of the "x in range 5" stuff. I bet you had a hell of a job lining up all the %s's with their arguments correctly ;-)
Yeah, well, apart from the one case where it was the same thing 15 times. But yes, I was aware of that usage but never used it before and tbh I hacked this up real quick so there's a few warts like that.
Quoted 11 lines
e.g.
"""static void Py%(rawname)s_dealloc(Py_%(rawname)s... etc """ % {'rawname': ty.rawname, 'dtor': dtor... etc}etc
I also think you could get rid of the long """ string containing hand written helpers by putting all that stuff in a
pyxl.hand/orxl.c. I'd much prefer to edit the hand-written C code in a .h file where syntax highlighting will work properly etc. If you generate the relevant prototypes correctly then declaration shouldn't be too much of an issue.Yeah, the definitions of the genwrap__*_(get|set) funcs can move to
xl.cwhich makes it a lot tidier.With that bit of code motion done (see patch below) it could make sense to add a few python_* attributes in
libxltypes.cso that we could, for example, say
Quoted 8 lines
Quoted 4 lines
x =
Type(..., python_methods=PyFoo_Methods, python_init=Foobar_Init)which would allow, eg: pci =
xl.device_pci();pci.parse("00:11.2")or allow a non keyword initialiser, eg: pci =xl.device_pci("00:11.2")I don't know that adding language specific stuff to the IDL is any better than just recognising specific type names which would like special handling in the generator.
Actually I think enabling inheritance in the generated objects is the way to do this. It just requires a bit of glue code both in the C and in any inherited python wrappers to make it all work right.
Will have code for this in the next rev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Ian Campbell
Quoted 2 lines · Gianni Tedesco
+ ret =
filter(lambda x:True, types)# deep copyI think this is more normally written "ret = list(types)", "ret = types[:]", or:
from copy import copy
ret = copy(types)
I think i prefer types[:] (forgot about that) - copy module is a nightmare and IMO nastiest part of python language is semantics viz deep vs shallow copies. Does list() even do deep copy?? (I had no idea)
Quoted 3 lines
Although given that the caller does:
types =tree_frob(types)
why is the deep copy necessary?tree_frobcould work in-place.
Well, it's going to end up doing the tree flattening for the keyed union type that needs to be handled. Just makes it simpler than keeping around an original list that may or may not contain what you think it does at that point.