Skip to contentSkip to conversation

: add libxl python binding

PATCH Superseded Gianni Tedesco 2 participants 3 messages Sep 9, 2010 Arm/actions semi architecture

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 : add libxl python binding

Changes since last time:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

Please consider and apply :)

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but list_domains, domid_to_name, domain_shutdown and domain_destroy methods are implemented and tested. These functions provide examples of the two basic patterns that all future methods should follow.

About 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

There are minor changes to libxltypes.py and the libxl.idl file and the new rules are documented. I have also removed the use of a nasty union in libxl_device_pci structure which was making the binding more complicated than necessary and for no good reason (exporting random bits of low level ABI that libxl would rather hide anyway). Finally there is a corresponding (untested) change to the ocaml binding to fix a compile error and maintain previous API in ocaml.

--

 tools/python/genwrap.py             |  346 +++++++++++++++++++++++++++++++
 tools/python/xen/lowlevel/xl/xl.c   |  397 ++++++++++++++++++++++++++++++++++++
 tools/libxl/idl.txt                 |    7 
 tools/libxl/libxl.idl               |   28 --
 tools/libxl/libxl_pci.c             |   13 -
 tools/libxl/libxltypes.py           |   27 +-
 tools/ocaml/libs/xl/xl_stubs.c      |   10 
 tools/python/Makefile               |    6 
 tools/python/setup.py               |   19 +
 9 files changed, 823 insertions(+), 30 deletions(-)

--

tools/libxl/idl.txt +6 -1
diff -r b19856f6dd76 tools/libxl/idl.txt
@@ -13,7 +13,8 @@ contain the initial namespace element (e
 how to specify a namespace.
 
 The Type.typename contains the C name of the type _including_ the
-namespace element.
+namespace element while Type.rawname is always set to the 'base' name
+of the type.
 
 The libxltypes.Type base class has several other properties which
 apply to all types. The properties are set by passing a named
@@ -48,6 +49,10 @@ Type.autogenerate_destructor: (default: 
  Indicates if the above named Type.destructor_fn should be
  autogenerated.
 
+Type.internal (default: False):
+ Indicates that type is not suitable for export, for example, via accessors
+ in auto-generated language bindings.
+
 Other simple type-Classes
 -------------------------
 
tools/libxl/libxl.idl +11 -17
diff -r b19856f6dd76 tools/libxl/libxl.idl
@@ -6,11 +6,11 @@
 libxl_ctx = Builtin("ctx")
 libxl_uuid = Builtin("uuid")
 libxl_mac = Builtin("mac")
-libxl_qemu_machine_type = Builtin("qemu_machine_type")
-libxl_console_consback = Builtin("console_consback")
-libxl_console_constype = Builtin("console_constype")
-libxl_disk_phystype = Builtin("disk_phystype")
-libxl_nic_type = Builtin("nic_type")
+libxl_qemu_machine_type = Number("qemu_machine_type", namespace="libxl_")
+libxl_console_consback = Number("console_consback", namespace="libxl_")
+libxl_console_constype = Number("console_constype", namespace="libxl_")
+libxl_disk_phystype = Number("disk_phystype", namespace="libxl_")
+libxl_nic_type = Number("nic_type", namespace="libxl_")
 
 libxl_string_list = Builtin("string_list", destructor_fn="libxl_string_list_destroy", passby=PASS_BY_REFERENCE)
 libxl_key_value_list = Builtin("key_value_list", destructor_fn="libxl_key_value_list_destroy", passby=PASS_BY_REFERENCE)
@@ -85,7 +85,7 @@ libxl_file_reference = Struct("file_refe
 mapped is true then the actual file may already be unlinked."""),
     ("mapped", integer),
     ("data", void),
-    ("size", size_t)], autogenerate_destructor=False)
+    ("size", size_t)], autogenerate_destructor=False, internal=True)
 
 libxl_domain_build_info = Struct("domain_build_info",[
     ("max_vcpus",       integer),
@@ -240,17 +240,11 @@ libxl_device_net2 = Struct("device_net2"
     ])
 
 libxl_device_pci = Struct("device_pci", [
-    (None, Union(None, [("value", unsigned_integer),
-                        (None, Struct(None,[("reserved1", BitField(unsigned_integer, 2)),
-                                            ("reg",       BitField(unsigned_integer, 6)),
-                                            ("func",      BitField(unsigned_integer, 3)),
-                                            ("dev",       BitField(unsigned_integer, 5)),
-                                            ("bus",       BitField(unsigned_integer, 8)),
-                                            ("reserved2", BitField(unsigned_integer, 7)),
-                                            ("enable",    BitField(unsigned_integer, 1)),
-                                             ])),
-                        ])
-     ),
+    ("reg",       uint8),
+    ("func",      uint8),
+    ("dev",       uint8),
+    ("bus",       uint8),
+    ("enable",    bool),
     ("domain", unsigned_integer),
     ("vdevfn", unsigned_integer),
     ("vfunc_mask", unsigned_integer),
tools/libxl/libxl_pci.c +11 -2
diff -r b19856f6dd76 tools/libxl/libxl_pci.c
@@ -41,6 +41,15 @@
 #define PCI_BDF_SHORT          "%02x:%02x.%01x"
 #define PCI_BDF_VDEVFN         "%04x:%02x:%02x.%01x@%02x"
 
+static unsigned int pcidev_value(libxl_device_pci *pcidev)
+{
+    return ((pcidev->reg & 0x3f) << 24) |
+           ((pcidev->func & 0x7) << 21) |
+           ((pcidev->dev & 0x1f) << 16) |
+           ((pcidev->bus & 0xff) << 8) |
+           ((pcidev->enable) ? 1 : 0);
+}
+
 static int pcidev_init(libxl_device_pci *pcidev, unsigned int domain,
                           unsigned int bus, unsigned int dev,
                           unsigned int func, unsigned int vdevfn)
@@ -699,7 +708,7 @@ static int do_pci_add(libxl__gc *gc, uin
     }
 out:
     if (!libxl_is_stubdom(ctx, domid, NULL)) {
-        rc = xc_assign_device(ctx->xch, domid, pcidev->value);
+        rc = xc_assign_device(ctx->xch, domid, pcidev_value(pcidev));
         if (rc < 0 && (hvm || errno != ENOSYS)) {
             LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, rc, "xc_assign_device failed");
             return ERROR_FAIL;
@@ -915,7 +924,7 @@ out:
     }
 
     if (!libxl_is_stubdom(ctx, domid, NULL)) {
-        rc = xc_deassign_device(ctx->xch, domid, pcidev->value);
+        rc = xc_deassign_device(ctx->xch, domid, pcidev_value(pcidev));
         if (rc < 0 && (hvm || errno != ENOSYS))
             LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, rc, "xc_deassign_device failed");
     }
tools/libxl/libxltypes.py +20 -7
diff -r b19856f6dd76 tools/libxl/libxltypes.py
@@ -14,10 +14,13 @@ class Type(object):
 
         if typename is None: # Anonymous type
             self.typename = None
+            self.rawname = None
         elif self.namespace is None: # e.g. system provided types
             self.typename = typename
+            self.rawname = typename
         else:
             self.typename = self.namespace + typename
+            self.rawname = typename
 
         if self.typename is not None:
             self.destructor_fn = kwargs.setdefault('destructor_fn', self.typename + "_destroy")
@@ -26,17 +29,27 @@ class Type(object):
 
         self.autogenerate_destructor = kwargs.setdefault('autogenerate_destructor', True)
 
+        self.internal = kwargs.setdefault('internal', False)
+
 class Builtin(Type):
     """Builtin type"""
     def __init__(self, typename, **kwargs):
         kwargs.setdefault('destructor_fn', None)
         Type.__init__(self, typename, **kwargs)
 
-class UInt(Type):
+class Number(Builtin):
+    def __init__(self, ctype, **kwargs):
+        kwargs.setdefault('namespace', None)
+        kwargs.setdefault('destructor_fn', None)
+        kwargs.setdefault('signed', False)
+        self.signed = kwargs['signed']
+        Builtin.__init__(self, ctype, **kwargs)
+
+class UInt(Number):
     def __init__(self, w, **kwargs):
         kwargs.setdefault('namespace', None)
         kwargs.setdefault('destructor_fn', None)
-        Type.__init__(self, "uint%d_t" % w, **kwargs)
+        Number.__init__(self, "uint%d_t" % w, **kwargs)
 
         self.width = w
 
@@ -128,12 +141,12 @@ class Reference(Type):
 
 void = Builtin("void *", namespace = None)
 bool = Builtin("bool", namespace = None)
-size_t = Builtin("size_t", namespace = None)
+size_t = Number("size_t", namespace = None)
 
-integer = Builtin("int", namespace = None)
-unsigned_integer = Builtin("unsigned int", namespace = None)
-unsigned = Builtin("unsigned int", namespace = None)
-unsigned_long = Builtin("unsigned long", namespace = None)
+integer = Number("int", namespace = None, signed = True)
+unsigned_integer = Number("unsigned int", namespace = None)
+unsigned = Number("unsigned int", namespace = None)
+unsigned_long = Number("unsigned long", namespace = None)
 
 uint8 = UInt(8)
 uint16 = UInt(16)
tools/ocaml/libs/xl/xl_stubs.c +9 -1
diff -r b19856f6dd76 tools/ocaml/libs/xl/xl_stubs.c
@@ -269,9 +269,17 @@ static int device_vfb_val(caml_gc *gc, l
 
 static int device_pci_val(caml_gc *gc, libxl_device_pci *c_val, value v)
 {
+    unsigned int bitmask;
 	CAMLparam1(v);
 
-	c_val->value = Int_val(Field(v, 0));
+    /* FIXME: propagate API change to ocaml */
+    bitmask = Int_val(Field(v, 0));
+    c_val->reg = (bitmask & (0x3f << 24)) >> 24;
+    c_val->func = (bitmask & (0x7 << 21)) >> 21;
+    c_val->dev = (bitmask & (0x1f << 16)) >> 16;
+    c_val->bus = (bitmask & (0xff << 8)) >> 8;
+    c_val->enable = bitmask & 1;
+
 	c_val->domain = Int_val(Field(v, 1));
 	c_val->vdevfn = Int_val(Field(v, 2));
 	c_val->msitranslate = Bool_val(Field(v, 3));
tools/python/Makefile +6 -0
diff -r b19856f6dd76 tools/python/Makefile
@@ -20,6 +20,10 @@ genpath-target = $(call buildmakevars2fi
 
 .PHONY: build buildpy
 buildpy: genpath 
+	PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \
+		$(XEN_ROOT)/tools/libxl/libxl.idl \
+		xen/lowlevel/xl/_pyxl_types.h \
+		xen/lowlevel/xl/_pyxl_types.c
 	CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build
 
 build: buildpy refresh-pot refresh-po $(CATALOGS)
@@ -85,6 +89,8 @@ test:
 clean:
 	rm -f $(XENPATH)
 	rm -rf build *.pyc *.pyo *.o *.a *~ $(CATALOGS) xen/util/auxbin.pyc
+	rm -f xen/lowlevel/xl/_pyxl_types.h
+	rm -f xen/lowlevel/xl/_pyxl_types.c
 	rm -f $(DEPS)
 
 -include $(DEPS)
tools/python/genwrap.py new file +346 -0

346 lines not on this page · raw message

tools/python/setup.py +17 -2
diff -r b19856f6dd76 tools/python/setup.py
@@ -9,14 +9,23 @@ extra_compile_args  = [ "-fno-strict-ali
 include_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
                  XEN_ROOT + "/tools/include",
+                 XEN_ROOT + "/tools/libxl",
                  ]
 
 library_dirs = [ XEN_ROOT + "/tools/libxc",
                  XEN_ROOT + "/tools/xenstore",
+                 XEN_ROOT + "/tools/libxl",
+                 XEN_ROOT + "/tools/blktap2/control",
                  ]
 
 libraries = [ "xenctrl", "xenguest", "xenstore" ]
 
+plat = os.uname()[0]
+if plat == 'Linux':
+    uuid_libs = ["uuid"]
+else:
+    uuid_libs = []
+
 xc = Extension("xc",
                extra_compile_args = extra_compile_args,
                include_dirs       = include_dirs + [ "xen/lowlevel/xc" ],
@@ -83,8 +92,14 @@ netlink = Extension("netlink",
                     sources            = [ "xen/lowlevel/netlink/netlink.c",
                                            "xen/lowlevel/netlink/libnetlink.c"])
 
-modules = [ xc, xs, ptsname, acm, flask ]
-plat = os.uname()[0]
+xl = Extension("xl",
+               extra_compile_args = extra_compile_args,
+               include_dirs       = include_dirs + [ "xen/lowlevel/xl" ],
+               library_dirs       = library_dirs,
+               libraries          = libraries + ["xenlight", "blktapctl" ] + uuid_libs,
+               sources            = [ "xen/lowlevel/xl/xl.c", "xen/lowlevel/xl/_pyxl_types.c" ])
+
+modules = [ xc, xs, ptsname, acm, flask, xl ]
 if plat == 'SunOS':
     modules.extend([ scf, process ])
 if plat == 'Linux':
tools/python/xen/lowlevel/xl/xl.c new file +397 -0

397 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · Gianni Tedesco

Changes since last time:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

Please consider and apply :)

Like it.

I think it'd be nice to separate out stuff like the PCI API change and IDL changes into preparatory patches. Patches which do one logical thing are generally easier to review (and useful for bisecting and such too).

Quoted 11 lines

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but list_domains, domid_to_name, domain_shutdown and domain_destroy methods are implemented and tested. These functions provide examples of the two basic patterns that all future methods should follow.

About 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I think this is acceptable for Builtin() types and is really part of the meaning of builtin. (except as I read on I realise that libxl_file_reference isn't actually a Builtin like I thought, more on that later!)

Quoted 8 lines

There are minor changes to libxltypes.py and the libxl.idl file and the new rules are documented. I have also removed the use of a nasty union in libxl_device_pci structure which was making the binding more complicated than necessary and for no good reason (exporting random bits of low level ABI that libxl would rather hide anyway). Finally there is a corresponding (untested) change to the ocaml binding to fix a compile error and maintain previous API in ocaml.

Quoted 5 lines

+Type.internal (default: False):
+ Indicates that type is not suitable for export, for example, via accessors
+ in auto-generated language bindings.

Maybe the issue here is that libxl_file_reference should really be a Builtin? It's suspicious that it is the only non-Builtin to which both this flag and autogenerate_destructor=False apply.

If that doesn't make sense then perhaps a new sub-class of Builtin might be more appropriate than an attribute (not sure what it would be called though, SuperBuiltin ;-)). I'm not sure that a non-Builtin should ever be internal and this would help prevent that.

tools/python/Makefile Thu Sep 09 12:06:50 2010 +0100 genpath-target = $(call buildmakevars2fi · line 27

Quoted 12 lines

diff -r b19856f6dd76 tools/python/Makefile

tools/python/Makefile Thu Sep 09 12:06:50 2010 +0100 renamed from tools/python/Makefile Thu Sep 09 09:24:24 2010 +0100 +4 -0
@@ -20,6 +20,10 @@ genpath-target = $(call buildmakevars2fi
 
 .PHONY: build buildpy
 buildpy: genpath
+       PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \
+               $(XEN_ROOT)/tools/libxl/libxl.idl \
+               xen/lowlevel/xl/_pyxl_types.h \
+               xen/lowlevel/xl/_pyxl_types.c
        CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build

Needs some additional dependencies on genwrap.py and $(XE_ROOT)/tools/libxl/libxl.idl?

Ian.

#2 lore raw
Quoted 7 lines · Ian Campbell
Quoted 6 lines · Gianni Tedesco

Changes since last time:
- split auto-generated code in to c and h files
- un-break the build system
- fix ocaml binding due to libxl API change
- lot's of tidy-ups too numerous to mention

Quoted 7 lines
Quoted 1 line

Please consider and apply :)

Like it.

I think it'd be nice to separate out stuff like the PCI API change and IDL changes into preparatory patches. Patches which do one logical thing are generally easier to review (and useful for bisecting and such too).

Yeah, that's not a problem, in this case I can just split it out by filename.

Quoted 27 lines
Quoted 11 lines

-----8<--------------------------------------------------------------- Introduce python binding for libxl. The binding is not yet complete but list_domains, domid_to_name, domain_shutdown and domain_destroy methods are implemented and tested. These functions provide examples of the two basic patterns that all future methods should follow.

About 5,000 lines of boilerplate is automatically generated to wrap and export all relevant libxl structure definitions. There are a few places where such code cannot be fully auto-generated and special hooks are declared and stubbed where, for example, conversion between libxl_file_reference and a python file object is required.

I think this is acceptable for Builtin() types and is really part of the meaning of builtin. (except as I read on I realise that libxl_file_reference isn't actually a Builtin like I thought, more on that later!)

Quoted 9 lines

There are minor changes to libxltypes.py and the libxl.idl file and the new rules are documented. I have also removed the use of a nasty union in libxl_device_pci structure which was making the binding more complicated than necessary and for no good reason (exporting random bits of low level ABI that libxl would rather hide anyway). Finally there is a corresponding (untested) change to the ocaml binding to fix a compile error and maintain previous API in ocaml.

Quoted 7 lines
Quoted 3 lines

+Type.internal (default: False):
+ Indicates that type is not suitable for export, for example, via accessors
+ in auto-generated language bindings.

Maybe the issue here is that libxl_file_reference should really be a Builtin? It's suspicious that it is the only non-Builtin to which both this flag and autogenerate_destructor=False apply.

I think your analysis is right and this in fact works for me. I had misunderstood full meaning of Builtins even though you had mentioned the (builtins,types) tuple thing. Fixed this and it works fine.

Quoted 4 lines

If that doesn't make sense then perhaps a new sub-class of Builtin might be more appropriate than an attribute (not sure what it would be called though, SuperBuiltin ;-)). I'm not sure that a non-Builtin should ever be internal and this would help prevent that.

Meh

Quoted 5 lines
Quoted 5 lines

diff -r b19856f6dd76 tools/python/Makefile

tools/python/Makefile Thu Sep 09 12:06:50 2010 +0100 renamed from tools/python/Makefile Thu Sep 09 09:24:24 2010 +0100 +0 -0
@@ -20,6 +20,10 @@ genpath-target = $(call buildmakevars2fi
 
Quoted 10 lines

tools/python/Makefile Thu Sep 09 12:06:50 2010 +0100 genpath-target = $(call buildmakevars2fi · line 27

Quoted 7 lines
tools/python/Makefile Thu Sep 09 12:06:50 2010 +0100 renamed from tools/python/Makefile Thu Sep 09 09:24:24 2010 +0100 +4 -0
 .PHONY: build buildpy
 buildpy: genpath
+       PYTHONPATH=$(XEN_ROOT)/tools/libxl $(PYTHON) genwrap.py \
+               $(XEN_ROOT)/tools/libxl/libxl.idl \
+               xen/lowlevel/xl/_pyxl_types.h \
+               xen/lowlevel/xl/_pyxl_types.c
        CC="$(CC)" CFLAGS="$(CFLAGS)" $(PYTHON) setup.py build

Needs some additional dependencies on genwrap.py and $(XE_ROOT)/tools/libxl/libxl.idl?

Yes, and libxltypes.py, fixed.