Skip to contentSkip to conversation

generated Go libxl bindings using IDL

PATCH In review Nick Rosbrook 2 participants 9 messages Jan 4, 2020 – Jan 16, 2020

lore
Share & tools
lore
#0 lore raw

v5 cover generated Go libxl bindings using IDL

After Xen summit, we started the discussion in this[1] RFC thread to figure out how to generate Go bindings for libxl. This series implements that Go code generation using the existing IDL, and updates the existing hand-written code in xenlight.go to use the generated code.

The goal of this series is to provide a good foundation for continued development of the Go package.

The v1 series can be found on my GitHub branch[2].

Changes in v2:
- GitHub branch for v2 [3].
- Drop patch 01/24 from v1 since was committed as a bug fix for 4.13.
- The Makefile changes in 24/24 from v1 have been moved to the patches where the build changes are introduced.

Changes in v3:
- GitHub branch for v3 [4].
- Simplify a pattern for iterating over builtin types in their fromC functions.
- Try not to duplicate as much code in gengotypes.py, and use consistent style in generated code when calling fromC.

Changes in v4:
- GitHub branch for v4 [5].
- The first 16 patches were commited after v3.

Changes in v5:
- GitHub branch for v5 [6].
- The only 3 patches left are for generating toC marshaling functions.

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02259.html
[2] https://github.com/enr0n/xen/tree/golang-patches-v1
[3] https://github.com/enr0n/xen/tree/golang-patches-v2
[4] https://github.com/enr0n/xen/tree/golang-patches-v3
[5] https://github.com/enr0n/xen/tree/golang-patches-v4
[6] https://github.com/enr0n/xen/tree/golang-patches-v5

Nick Rosbrook (3):
golang/xenlight: begin Go to C type marshaling golang/xenlight: implement keyed union Go to C marshaling golang/xenlight: implement array Go to C marshaling

 tools/golang/xenlight/gengotypes.py  |  185 +++
 tools/golang/xenlight/helpers.gen.go | 2249 +++++++++++++++++++++++++-
 tools/golang/xenlight/xenlight.go    |   82 +-
 3 files changed, 2399 insertions(+), 117 deletions(-)
Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v5 1/3 golang/xenlight: begin Go to C type marshaling

Implement conversions for basic types such as strings and integer types in toC functions.

Modify function signatures of toC implementations for builtin types to be consistent with the signature of the generated toC functions.

---
Changes in v5:
- Define xenlight_golang_convert_to_C so that field conversion code can be easily re-used.
- Check for err in defer'd func within toC to determine if the dispose function needs to be called.
- Pass a reference to the C type in toC, rather than returning a copy of the C variable.
- Update the existing toC functions for builtin types to be consistent with the generated functions.
- Only call CString if the Go string is non-empty.

---
 tools/golang/xenlight/gengotypes.py  |   81 ++
 tools/golang/xenlight/helpers.gen.go | 1506 ++++++++++++++++++++++++++
 tools/golang/xenlight/xenlight.go    |   82 +-
 3 files changed, 1621 insertions(+), 48 deletions(-)
tools/golang/xenlight/gengotypes.py +81 -0
index 8857e2e8fd..15aa4be220 100644
@@ -234,6 +234,9 @@ def xenlight_golang_generate_helpers(path = None, types = None, comment = None):
                 f.write(extra)
                 f.write('\n')
 
+            f.write(xenlight_golang_define_to_C(ty))
+            f.write('\n')
+
     go_fmt(path)
 
 def xenlight_golang_define_from_C(ty = None):
@@ -434,6 +437,84 @@ def xenlight_golang_array_from_C(ty = None):
 
     return s
 
+def xenlight_golang_define_to_C(ty = None, typename = None, nested = False):
+    """
+    Define the toC marshaling function for the type
+    represented by ty.
+    """
+    func = 'func (x *{}) toC(xc *C.{}) (err error){{{}\n return nil \n }}\n'
+    body = ''
+
+    if ty.dispose_fn is not None:
+        body += 'defer func(){{\nif err != nil{{\nC.{}(xc)}}\n}}()\n\n'.format(ty.dispose_fn)
+
+    goname = xenlight_golang_fmt_name(ty.typename)
+    cname  = ty.typename
+
+    for f in ty.fields:
+        if f.type.typename is not None:
+            if isinstance(f.type, idl.Array):
+                # TODO
+                continue
+
+            body += xenlight_golang_convert_to_C(f)
+
+        elif isinstance(f.type, idl.Struct):
+            for nf in f.type.fields:
+                body += xenlight_golang_convert_to_C(nf, outer_name=f.name)
+
+        elif isinstance(f.type, idl.KeyedUnion):
+            # TODO
+            pass
+
+        else:
+            raise Exception('type {} not supported'.format(f.type))
+
+    return func.format(goname, cname, body)
+
+def xenlight_golang_convert_to_C(ty = None, outer_name = None):
+    """
+    Returns a line of Go code that converts the Go type represented
+    by ty to its corresponding Go type.
+
+    If outer_name is set, the type is treated as nested within another field
+    named outer_name.
+    """
+    s = ''
+
+    gotypename = xenlight_golang_fmt_name(ty.type.typename)
+    ctypename  = ty.type.typename
+    goname     = xenlight_golang_fmt_name(ty.name)
+    cname      = ty.name
+
+    # In cgo, C names that conflict with Go keywords can be
+    # accessed by prepending an underscore to the name.
+    if cname in go_keywords:
+        cname = '_' + cname
+
+    # If outer_name is set, treat this as nested.
+    if outer_name is not None:
+        goname = '{}.{}'.format(xenlight_golang_fmt_name(outer_name), goname)
+        cname  = '{}.{}'.format(outer_name, cname)
+
+    is_castable = (ty.type.json_parse_type == 'JSON_INTEGER' or
+                   isinstance(ty.type, idl.Enumeration) or
+                   gotypename in go_builtin_types)
+
+    if not is_castable:
+        s += 'if err := x.{}.toC(&xc.{}); err != nil {{\n'.format(goname,cname)
+        s += 'return err\n}\n'
+
+    elif gotypename == 'string':
+        # Use the cgo helper for converting C strings.
+        s += 'if x.{} != "" {{\n'.format(goname)
+        s += 'xc.{} = C.CString(x.{})}}\n'.format(cname,goname)
+
+    else:
+        s += 'xc.{} = C.{}(x.{})\n'.format(cname,ctypename,goname)
+
+    return s
+
 def xenlight_golang_fmt_name(name, exported = True):
     """
     Take a given type name and return an
tools/golang/xenlight/helpers.gen.go +1506 -0

1,863 lines not on this page · raw message

tools/golang/xenlight/xenlight.go +34 -48

198 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v5 1/3
#2 lore raw

Re: [Xen-devel] [PATCH v5 1/3] golang/xenlight: begin Go to C type marshaling

Quoted 8 lines · Nick Rosbrook

Implement conversions for basic types such as strings and integer types in toC functions.

Modify function signatures of toC implementations for builtin types to be consistent with the signature of the generated toC functions.

Quoted 11 lines

---
Changes in v5:
- Define xenlight_golang_convert_to_C so that field conversion code can be easily re-used.
- Check for err in defer'd func within toC to determine if the dispose function needs to be called.
- Pass a reference to the C type in toC, rather than returning a copy of the C variable.
- Update the existing toC functions for builtin types to be consistent with the generated functions.
- Only call CString if the Go string is non-empty.

Looks good! Only one question:

tools/golang/xenlight/gengotypes.py def xenlight_golang_array_from_C(ty = None): · line 505

Quoted 2 lines
tools/golang/xenlight/gengotypes.py +2 -0
+    if not is_castable:
+        s += 'if err := x.{}.toC(&xc.{}); err != nil {{\n'.format(goname,cname)

Err should be defined function-wide at this point. Are you using := on purpose for some reason? Would it make sense to make this = instead?

If we want to, I can change that on check-in; so either way:

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH v5 1/3] golang/xenlight: begin Go to C type marshaling

Quoted 7 lines

Looks good! Only one question:

tools/golang/xenlight/gengotypes.py def xenlight_golang_array_from_C(ty = None): · line 505

Quoted 2 lines
tools/golang/xenlight/gengotypes.py +2 -0
+    if not is_castable:
+        s += 'if err := x.{}.toC(&xc.{}); err != nil {{\n'.format(goname,cname)

Err should be defined function-wide at this point. Are you using := on purpose for some reason? Would it make sense to make this = instead?

This is on purpose. IMO it's better to keep the variable scoped to the if block when using short statement if syntax. If the desire is to assign to the same err variable, then I would not use the short statement. Reason being I think it helps readability; the short statement syntax is handy, but it is easy to glance over and not realize an assignment is being made.

Thanks,
-NR

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: [Xen-devel] [PATCH v5 1/3] golang/xenlight: begin Go to C type marshaling

Quoted 8 lines · Nick Rosbrook
Quoted 7 lines

Looks good! Only one question:

tools/golang/xenlight/gengotypes.py def xenlight_golang_array_from_C(ty = None): · line 505

Quoted 2 lines
tools/golang/xenlight/gengotypes.py +2 -0
+    if not is_castable:
+        s += 'if err := x.{}.toC(&xc.{}); err != nil {{\n'.format(goname,cname)

Err should be defined function-wide at this point. Are you using := on purpose for some reason? Would it make sense to make this = instead?

Quoted 6 lines

This is on purpose. IMO it's better to keep the variable scoped to the if block when using short statement if syntax. If the desire is to assign to the same err variable, then I would not use the short statement. Reason being I think it helps readability; the short statement syntax is handy, but it is easy to glance over and not realize an assignment is being made.

Fair enough. I'll check this series in once I get a chance to test it again.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

PATCH v5 2/3 golang/xenlight: implement keyed union Go to C marshaling

In reply to Nick Rosbrook

Since the C union cannot be directly populated, populate the fields of the corresponding C struct defined in the cgo preamble, and then copy that struct as bytes into the byte slice that Go uses as the union.

---
Changes in v5:
- Make use of xenlight_golang_convert_to_C to convert fields.
- Remove dispose_fn from parameter list in xenlight_golang_union_to_C since it's no longer needed.

---
 tools/golang/xenlight/gengotypes.py  |  79 ++++++-
 tools/golang/xenlight/helpers.gen.go | 294 +++++++++++++++++++++++++++
 2 files changed, 366 insertions(+), 7 deletions(-)
tools/golang/xenlight/gengotypes.py +72 -7
index 15aa4be220..8ab6768e61 100644
@@ -464,15 +464,15 @@ def xenlight_golang_define_to_C(ty = None, typename = None, nested = False):
                 body += xenlight_golang_convert_to_C(nf, outer_name=f.name)
 
         elif isinstance(f.type, idl.KeyedUnion):
-            # TODO
-            pass
+            body += xenlight_golang_union_to_C(f.type, f.name, ty.typename)
 
         else:
             raise Exception('type {} not supported'.format(f.type))
 
     return func.format(goname, cname, body)
 
-def xenlight_golang_convert_to_C(ty = None, outer_name = None):
+def xenlight_golang_convert_to_C(ty = None, outer_name = None,
+                                 govarname = None, cvarname = None):
     """
     Returns a line of Go code that converts the Go type represented
     by ty to its corresponding Go type.
@@ -482,6 +482,14 @@ def xenlight_golang_convert_to_C(ty = None, outer_name = None):
     """
     s = ''
 
+    # Use 'xc' as the name for the C variable unless otherwise specified.
+    if cvarname is None:
+        cvarname = 'xc'
+
+    # Use 'x' as the name for the Go variable unless otherwise specified.
+    if govarname is None:
+        govarname = 'x'
+
     gotypename = xenlight_golang_fmt_name(ty.type.typename)
     ctypename  = ty.type.typename
     goname     = xenlight_golang_fmt_name(ty.name)
@@ -502,16 +510,73 @@ def xenlight_golang_convert_to_C(ty = None, outer_name = None):
                    gotypename in go_builtin_types)
 
     if not is_castable:
-        s += 'if err := x.{}.toC(&xc.{}); err != nil {{\n'.format(goname,cname)
+        s += 'if err := {}.{}.toC(&{}.{}); err != nil {{\n'.format(govarname,goname,
+                                                                   cvarname,cname)
         s += 'return err\n}\n'
 
     elif gotypename == 'string':
         # Use the cgo helper for converting C strings.
-        s += 'if x.{} != "" {{\n'.format(goname)
-        s += 'xc.{} = C.CString(x.{})}}\n'.format(cname,goname)
+        s += 'if {}.{} != "" {{\n'.format(govarname,goname)
+        s += '{}.{} = C.CString({}.{})}}\n'.format(cvarname,cname,
+                                                   govarname,goname)
 
     else:
-        s += 'xc.{} = C.{}(x.{})\n'.format(cname,ctypename,goname)
+        s += '{}.{} = C.{}({}.{})\n'.format(cvarname,cname,ctypename,
+                                            govarname,goname)
+
+    return s
+
+def xenlight_golang_union_to_C(ty = None, union_name = '',
+                               struct_name = ''):
+    keyname   = ty.keyvar.name
+    gokeyname = xenlight_golang_fmt_name(keyname)
+    keytype   = ty.keyvar.type.typename
+    gokeytype = xenlight_golang_fmt_name(keytype)
+
+    interface_name = '{}_{}_union'.format(struct_name, keyname)
+    interface_name = xenlight_golang_fmt_name(interface_name, exported=False)
+
+    cgo_keyname = keyname
+    if cgo_keyname in go_keywords:
+        cgo_keyname = '_' + cgo_keyname
+
+
+    s = 'xc.{} = C.{}(x.{})\n'.format(cgo_keyname,keytype,gokeyname)
+    s += 'switch x.{}{{\n'.format(gokeyname)
+
+    # Create switch statement to determine how to populate the C union.
+    for f in ty.fields:
+        key_val = '{}_{}'.format(keytype, f.name)
+        key_val = xenlight_golang_fmt_name(key_val)
+        if f.type is None:
+            continue
+
+        s += 'case {}:\n'.format(key_val)
+        cgotype = '{}_{}_union_{}'.format(struct_name,keyname,f.name)
+        gotype  = xenlight_golang_fmt_name(cgotype)
+
+        field_name = xenlight_golang_fmt_name('{}_union'.format(keyname))
+        s += 'tmp, ok := x.{}.({})\n'.format(field_name,gotype)
+        s += 'if !ok {\n'
+        s += 'return errors.New("wrong type for union key {}")\n'.format(keyname)
+        s += '}\n'
+
+        s += 'var {} C.{}\n'.format(f.name,cgotype)
+        for uf in f.type.fields:
+            s += xenlight_golang_convert_to_C(uf,cvarname=f.name,
+                                              govarname='tmp')
+
+        # The union is still represented as Go []byte.
+        s += '{}Bytes := C.GoBytes(unsafe.Pointer(&{}),C.sizeof_{})\n'.format(f.name,
+                                                                              f.name,
+                                                                              cgotype)
+        s += 'copy(xc.{}[:],{}Bytes)\n'.format(union_name,f.name)
+
+    # End switch statement
+    s += 'default:\n'
+    err_string = '"invalid union key \'%v\'", x.{}'.format(gokeyname)
+    s += 'return fmt.Errorf({})'.format(err_string)
+    s += '}\n'
 
     return s
 
tools/golang/xenlight/helpers.gen.go +294 -0

330 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v5 2/3
#6 lore raw

Re: [Xen-devel] [PATCH v5 2/3] golang/xenlight: implement keyed union Go to C marshaling

Quoted 5 lines · Nick Rosbrook

Since the C union cannot be directly populated, populate the fields of the corresponding C struct defined in the cgo preamble, and then copy that struct as bytes into the byte slice that Go uses as the union.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

PATCH v5 3/3 golang/xenlight: implement array Go to C marshaling

In reply to Nick Rosbrook

---
Changes in v5:
- Remove dispose_fn from parameter list in xenlight_golang_array_to_C since it's no longer needed.
- Update call to toC for elements of arrays of structs so that it matches the new function signature of toC.
- Do not try to marshal zero-length arrays.

---
 tools/golang/xenlight/gengotypes.py  |  41 +++-
 tools/golang/xenlight/helpers.gen.go | 311 +++++++++++++++++++++++++++
 2 files changed, 351 insertions(+), 1 deletion(-)
tools/golang/xenlight/gengotypes.py +40 -1
index 8ab6768e61..27edf66241 100644
@@ -454,7 +454,7 @@ def xenlight_golang_define_to_C(ty = None, typename = None, nested = False):
     for f in ty.fields:
         if f.type.typename is not None:
             if isinstance(f.type, idl.Array):
-                # TODO
+                body += xenlight_golang_array_to_C(f)
                 continue
 
             body += xenlight_golang_convert_to_C(f)
@@ -580,6 +580,45 @@ def xenlight_golang_union_to_C(ty = None, union_name = '',
 
     return s
 
+def xenlight_golang_array_to_C(ty = None):
+    s = ''
+
+    gotypename = xenlight_golang_fmt_name(ty.type.elem_type.typename)
+    goname     = xenlight_golang_fmt_name(ty.name)
+    ctypename  = ty.type.elem_type.typename
+    cname      = ty.name
+    clenvar    = ty.type.lenvar.name
+    golenvar   = xenlight_golang_fmt_name(clenvar,exported=False)
+
+    is_enum = isinstance(ty.type.elem_type,idl.Enumeration)
+    if gotypename in go_builtin_types or is_enum:
+        s += 'if {} := len(x.{}); {} > 0 {{\n'.format(golenvar,goname,golenvar)
+        s += 'xc.{} = (*C.{})(C.malloc(C.size_t({}*{})))\n'.format(cname,ctypename,
+                                                                   golenvar,golenvar)
+        s += 'xc.{} = C.int({})\n'.format(clenvar,golenvar)
+        s += 'c{} := (*[1<<28]C.{})(unsafe.Pointer(xc.{}))[:{}:{}]\n'.format(goname,
+                                                                      ctypename,cname,
+                                                                      golenvar,golenvar)
+        s += 'for i,v := range x.{} {{\n'.format(goname)
+        s += 'c{}[i] = C.{}(v)\n'.format(goname,ctypename)
+        s += '}\n}\n'
+
+        return s
+
+    s += 'if {} := len(x.{}); {} > 0 {{\n'.format(golenvar,goname,golenvar)
+    s += 'xc.{} = (*C.{})(C.malloc(C.ulong({})*C.sizeof_{}))\n'.format(cname,ctypename,
+                                                                   golenvar,ctypename)
+    s += 'xc.{} = C.int({})\n'.format(clenvar,golenvar)
+    s += 'c{} := (*[1<<28]C.{})(unsafe.Pointer(xc.{}))[:{}:{}]\n'.format(goname,
+                                                                         ctypename,cname,
+                                                                         golenvar,golenvar)
+    s += 'for i,v := range x.{} {{\n'.format(goname)
+    s += 'if err := v.toC(&c{}[i]); err != nil {{\n'.format(goname)
+    s += 'return err\n'
+    s += '}\n}\n}\n'
+
+    return s
+
 def xenlight_golang_fmt_name(name, exported = True):
     """
     Take a given type name and return an
tools/golang/xenlight/helpers.gen.go +311 -0

395 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel