tools: ynl-gen: fill in the gaps in support of legacy families
Subsystems (3)
Jakub Kicinski
maintainercover tools: ynl-gen: fill in the gaps in support of legacy families
Fill in the gaps in YNL C code gen so that we can generate user space code for all genetlink families for which we have specs.
The two major changes we need are support for fixed headers and support for recursive nests.
For fixed header support - place the struct for the fixed header directly in the request struct (and don't bother generating access helpers). The member of a fixed header can't be too complex, and also are by definition not optional so the user has to fill them in. The YNL core needs a bit of a tweak to understand that the attrs may now start at a fixed offset, which is not necessarily equal to sizeof(struct genlmsghdr).
Dealing with nested sets is much harder. Previously we'd gen the nested structs as:
struct outer {
struct inner inner;
};
If structs are recursive (e.g. inner contains outer again) we must break this chain and allocate one of the structs dynamically (store a pointer rather than full struct).
Jakub Kicinski (8):
tools: ynl-gen: add missing request free helpers for dumps
tools: ynl-gen: use enum user type for members and args
tools: ynl-gen: support fixed headers in genetlink
tools: ynl-gen: fill in implementations for TypeUnused
tools: ynl-gen: record information about recursive nests
tools: ynl-gen: re-sort ignoring recursive nests
tools: ynl-gen: store recursive nests by a pointer
tools: ynl-gen: print prototypes for recursive stuff
tools/net/ynl/lib/ynl.c | 8 +- tools/net/ynl/lib/ynl.h | 1 + tools/net/ynl/ynl-gen-c.py | 190 +++++++++++++++++++++++++++++-------- 3 files changed, 158 insertions(+), 41 deletions(-)
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerPATCH 1/8 tools: ynl-gen: add missing request free helpers for dumps
The code gen generates a prototype for dump request free in the header, but no implementation in the source.
--- tools/net/ynl/ynl-gen-c.py | 1 + 1 file changed, 1 insertion(+)
tools/net/ynl/ynl-gen-c.py +1 -0
@@ -2771,6 +2771,7 @@ _C_KW = { ri = RenderInfo(cw, parsed, args.mode, op, "dump") if not ri.type_consistent: parse_rsp_msg(ri, deref=True) + print_req_free(ri) print_dump_type_free(ri) print_dump(ri) cw.nl()
Signature · 1 line
2.43.0
Re: [PATCH net-next 1/8] tools: ynl-gen: add missing request free helpers for dumps
Quoted 4 lines · Jakub Kicinski
The code gen generates a prototype for dump request free in the header, but no implementation in the source.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 2/8 tools: ynl-gen: use enum user type for members and args
In reply to Jakub KicinskiCommit 30c902001534 ("tools: ynl-gen: use enum name from the spec") added pre-cooked user type for enums. Use it to fix ignoring enum-name provided in the spec.
This changes a type in struct ethtool_tunnel_udp_entry but is generally inconsequential for current families.
--- tools/net/ynl/ynl-gen-c.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
tools/net/ynl/ynl-gen-c.py +2 -3
@@ -333,9 +333,8 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S else: self.is_bitfield = False - maybe_enum = not self.is_bitfield and 'enum' in self.attr - if maybe_enum and self.family.consts[self.attr['enum']].enum_name: - self.type_name = c_lower(f"enum {self.family.name}_{self.attr['enum']}") + if not self.is_bitfield and 'enum' in self.attr: + self.type_name = self.family.consts[self.attr['enum']].user_type elif self.is_auto_scalar: self.type_name = '__' + self.type[0] + '64' else:
Signature · 1 line
2.43.0
Re: [PATCH net-next 2/8] tools: ynl-gen: use enum user type for members and args
Quoted 8 lines · Jakub Kicinski
Commit
30c902001534("tools: ynl-gen: use enum name from the spec") added pre-cooked user type for enums. Use it to fix ignoring enum-name provided in the spec.This changes a type in struct
ethtool_tunnel_udp_entrybut is generally inconsequential for current families.Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 3/8 tools: ynl-gen: support fixed headers in genetlink
In reply to Jakub KicinskiSupport genetlink families using simple fixed headers. Assume fixed header is identical for all ops of the family for now.
Fixed headers are added to the request and reply structs as a _hdr member, and copied to/from netlink messages appropriately.
--- tools/net/ynl/lib/ynl.c | 8 +++---- tools/net/ynl/lib/ynl.h | 1 + tools/net/ynl/ynl-gen-c.py | 44 ++++++++++++++++++++++++++++++++++---- 3 files changed, 45 insertions(+), 8 deletions(-)
tools/net/ynl/lib/ynl.c +4 -4
@@ -191,12 +191,12 @@ ynl_ext_ack_check(struct ynl_sock *ys, const struct nlmsghdr *nlh, str ? " (" : ""); start =mnl_nlmsg_get_payload_offset (ys->nlh, -sizeof (struct genlmsghdr )); + ys->family->hdr_len); end =mnl_nlmsg_get_payload_tail (ys->nlh); off = ys->err.attr_offs; off -=sizeof (struct nlmsghdr ); - off -=sizeof (struct genlmsghdr ); + off -= ys->family->hdr_len; n +=ynl_err_walk (ys, start, end, off, ys->req_policy, &bad_attr[n],sizeof (bad_attr) - n,NULL );
@@ -217,14 +217,14 @@ ynl_ext_ack_check(struct ynl_sock *ys, const struct nlmsghdr *nlh, bad_attr[0] ? ", " : (str ? " (" : "")); start =mnl_nlmsg_get_payload_offset (ys->nlh, -sizeof (struct genlmsghdr )); + ys->family->hdr_len); end =mnl_nlmsg_get_payload_tail (ys->nlh); nest_pol = ys->req_policy;if (tb[NLMSGERR_ATTR_MISS_NEST ]) { off =mnl_attr_get_u32 (tb[NLMSGERR_ATTR_MISS_NEST ]); off -=sizeof (struct nlmsghdr ); - off -=sizeof (struct genlmsghdr ); + off -= ys->family->hdr_len; n += ynl_err_walk(ys, start, end, off, ys->req_policy, &miss_attr[n], sizeof(miss_attr) - n,
tools/net/ynl/lib/ynl.h +1 -0
@@ -44,6 +44,7 @@ struct ynl_error {struct ynl_family {/* private: */ const char *name; +size_t hdr_len;const struct ynl_ntf_info *ntf_info;unsigned int ntf_info_size; };
tools/net/ynl/ynl-gen-c.py +40 -4
98 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH net-next 3/8] tools: ynl-gen: support fixed headers in genetlink
Quoted 7 lines · Jakub Kicinski
Support genetlink families using simple fixed headers. Assume fixed header is identical for all ops of the family for now.
Fixed headers are added to the request and reply structs as a
_hdrmember, and copied to/from netlink messages appropriately.Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 4/8 tools: ynl-gen: fill in implementations for TypeUnused
In reply to Jakub KicinskiFill in more empty handlers for TypeUnused. When 'unused' attr gets specified in a nested set we have to cleanly skip it during code generation.
--- tools/net/ynl/ynl-gen-c.py | 9 +++++++++ 1 file changed, 9 insertions(+)
tools/net/ynl/ynl-gen-c.py +9 -0
@@ -264,6 +264,15 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S def attr_policy(self, cw): pass + def attr_put(self, ri, var): + pass + + def attr_get(self, ri, var, first): + pass + + def setter(self, ri, space, direction, deref=False, ref=None): + pass + class TypePad(Type): def presence_type(self):
Signature · 1 line
2.43.0
Re: [PATCH net-next 4/8] tools: ynl-gen: fill in implementations for TypeUnused
Quoted 5 lines · Jakub Kicinski
Fill in more empty handlers for TypeUnused. When 'unused' attr gets specified in a nested set we have to cleanly skip it during code generation.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 5/8 tools: ynl-gen: record information about recursive nests
In reply to Jakub KicinskiTrack which nests are recursive. Non-recursive nesting gets rendered in C as directly nested structs. For recursive ones we need to put a pointer in, rather than full struct.
Track this information, no change to generated code, yet.
--- tools/net/ynl/ynl-gen-c.py | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
tools/net/ynl/ynl-gen-c.py +17 -2
@@ -105,6 +105,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S def is_scalar(self): return self.type in {'u8', 'u16', 'u32', 'u64', 's32', 's64'} + def is_recursive(self): + return False + def presence_type(self): return 'bit'
@@ -529,6 +532,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S class TypeNest(Type): + def is_recursive(self): + return self.family.pure_nested_structs[self.nested_attrs].recursive + def _complex_member_type(self, ri): return self.nested_struct_type
@@ -700,9 +706,12 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S if self.nested and space_name in family.consts: self.struct_name += '_' self.ptr_name = self.struct_name + ' *' + # All attr sets this one contains, directly or multiple levels down + self.child_nests = set() self.request = False self.reply = False + self.recursive = False self.attr_list = [] self.attrs = dict()
@@ -1059,14 +1068,20 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S pns_key_seen.add(name) else: pns_key_list.append(name) - # Propagate the request / reply + # Propagate the request / reply / recursive for attr_set, struct in reversed(self.pure_nested_structs.items()): for _, spec in self.attr_sets[attr_set].items(): if 'nested-attributes' in spec: - child = self.pure_nested_structs.get(spec['nested-attributes']) + child_name = spec['nested-attributes'] + struct.child_nests.add(child_name) + child = self.pure_nested_structs.get(child_name) if child: + if not child.recursive: + struct.child_nests.update(child.child_nests) child.request |= struct.request child.reply |= struct.reply + if attr_set in struct.child_nests: + struct.recursive = True def _load_attr_use(self): for _, struct in self.pure_nested_structs.items():
Signature · 1 line
2.43.0
Re: [PATCH net-next 5/8] tools: ynl-gen: record information about recursive nests
Quoted 7 lines · Jakub Kicinski
Track which nests are recursive. Non-recursive nesting gets rendered in C as directly nested structs. For recursive ones we need to put a pointer in, rather than full struct.
Track this information, no change to generated code, yet.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 6/8 tools: ynl-gen: re-sort ignoring recursive nests
In reply to Jakub KicinskiWe try to keep the structures and helpers "topologically sorted", to avoid forward declarations. When recursive nests are at play we need to sort twice, because structs which end up being marked as recursive will get a full set of forward declarations, so we should ignore them for the purpose of sorting.
--- tools/net/ynl/ynl-gen-c.py | 52 +++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 21 deletions(-)
tools/net/ynl/ynl-gen-c.py +31 -21
@@ -1008,6 +1008,33 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S self.root_sets[op['attribute-set']]['request'].update(req_attrs) self.root_sets[op['attribute-set']]['reply'].update(rsp_attrs) + def _sort_pure_types(self): + # Try to reorder according to dependencies + pns_key_list = list(self.pure_nested_structs.keys()) + pns_key_seen = set() + rounds = len(pns_key_list) ** 2 # it's basically bubble sort + for _ in range(rounds): + if len(pns_key_list) == 0: + break + name = pns_key_list.pop(0) + finished = True + for _, spec in self.attr_sets[name].items(): + if 'nested-attributes' in spec: + nested = spec['nested-attributes'] + # If the unknown nest we hit is recursive it's fine, it'll be a pointer + if self.pure_nested_structs[nested].recursive: + continue + if nested not in pns_key_seen: + # Dicts are sorted, this will make struct last + struct = self.pure_nested_structs.pop(name) + self.pure_nested_structs[name] = struct + finished = False + break + if finished: + pns_key_seen.add(name) + else: + pns_key_list.append(name) + def _load_nested_sets(self): attr_set_queue = list(self.root_sets.keys()) attr_set_seen = set(self.root_sets.keys())
@@ -1047,27 +1074,8 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S if attr in rs_members['reply']: self.pure_nested_structs[nested].reply = True - # Try to reorder according to dependencies - pns_key_list = list(self.pure_nested_structs.keys()) - pns_key_seen = set() - rounds = len(pns_key_list)**2 # it's basically bubble sort - for _ in range(rounds): - if len(pns_key_list) == 0: - break - name = pns_key_list.pop(0) - finished = True - for _, spec in self.attr_sets[name].items(): - if 'nested-attributes' in spec: - if spec['nested-attributes'] not in pns_key_seen: - # Dicts are sorted, this will make struct last - struct = self.pure_nested_structs.pop(name) - self.pure_nested_structs[name] = struct - finished = False - break - if finished: - pns_key_seen.add(name) - else: - pns_key_list.append(name) + self._sort_pure_types() + # Propagate the request / reply / recursive for attr_set, struct in reversed(self.pure_nested_structs.items()): for _, spec in self.attr_sets[attr_set].items():
@@ -1083,6 +1091,8 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S if attr_set in struct.child_nests: struct.recursive = True + self._sort_pure_types() + def _load_attr_use(self): for _, struct in self.pure_nested_structs.items(): if struct.request:
Signature · 1 line
2.43.0
Re: [PATCH net-next 6/8] tools: ynl-gen: re-sort ignoring recursive nests
Quoted 7 lines · Jakub Kicinski
We try to keep the structures and helpers "topologically sorted", to avoid forward declarations. When recursive nests are at play we need to sort twice, because structs which end up being marked as recursive will get a full set of forward declarations, so we should ignore them for the purpose of sorting.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 7/8 tools: ynl-gen: store recursive nests by a pointer
In reply to Jakub KicinskiTo avoid infinite nesting store recursive structs by pointer. If recursive struct is placed in the op directly - the first instance can be stored by value. That makes the code much less of a pain for majority of practical uses.
--- tools/net/ynl/ynl-gen-c.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
tools/net/ynl/ynl-gen-c.py +14 -2
@@ -108,6 +108,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S def is_recursive(self): return False + def is_recursive_for_op(self, ri): + return self.is_recursive() and not ri.op + def presence_type(self): return 'bit'
@@ -148,6 +151,8 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S member = self._complex_member_type(ri) if member: ptr = '*' if self.is_multi_val() else '' + if self.is_recursive_for_op(ri): + ptr = '*' ri.cw.p(f"{member} {ptr}{self.c_name};") return members = self.arg_member(ri)
@@ -539,7 +544,11 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S return self.nested_struct_type def free(self, ri, var, ref): - ri.cw.p(f'{self.nested_render_name}_free(&{var}->{ref}{self.c_name});') + at = '&' + if self.is_recursive_for_op(ri): + at = '' + ri.cw.p(f'if ({var}->{ref}{self.c_name})') + ri.cw.p(f'{self.nested_render_name}_free({at}{var}->{ref}{self.c_name});') def _attr_typol(self): return f'.type = YNL_PT_NEST, .nest = &{self.nested_render_name}_nest, '
@@ -548,8 +557,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S return 'NLA_POLICY_NESTED(' + self.nested_render_name + '_nl_policy)' def attr_put(self, ri, var): + at = '' if self.is_recursive_for_op(ri) else '&' self._attr_put_line(ri, var, f"{self.nested_render_name}_put(nlh, " + - f"{self.enum_name}, &{var}->{self.c_name})") + f"{self.enum_name}, {at}{var}->{self.c_name})") def _attr_get(self, ri, var): get_lines = [f"if ({self.nested_render_name}_parse(&parg, attr))",
@@ -562,6 +572,8 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S ref = (ref if ref else []) + [self.c_name] for _, attr in ri.family.pure_nested_structs[self.nested_attrs].member_list(): + if attr.is_recursive(): + continue attr.setter(ri, self.nested_attrs, direction, deref=deref, ref=ref)
Signature · 1 line
2.43.0
Re: [PATCH net-next 7/8] tools: ynl-gen: store recursive nests by a pointer
Quoted 6 lines · Jakub Kicinski
To avoid infinite nesting store recursive structs by pointer. If recursive struct is placed in the op directly - the first instance can be stored by value. That makes the code much less of a pain for majority of practical uses.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH 8/8 tools: ynl-gen: print prototypes for recursive stuff
In reply to Jakub KicinskiWe avoid printing forward declarations and prototypes for most types by sorting things topologically. But if structs nest we do need the forward declarations, there's no other way.
--- tools/net/ynl/ynl-gen-c.py | 44 +++++++++++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 5 deletions(-)
tools/net/ynl/ynl-gen-c.py +39 -5
@@ -1521,6 +1521,10 @@ _C_KW = { print_prototype(ri, "request") +def put_typol_fwd(cw, struct): + cw.p(f'extern struct ynl_policy_nest {struct.render_name}_nest;') + + def put_typol(cw, struct): type_max = struct.attr_set.max_name cw.block_start(line=f'struct ynl_policy_attr {struct.render_name}_policy[{type_max} + 1] =')
@@ -1594,12 +1598,17 @@ _C_KW = { _put_enum_to_str_helper(cw, enum.render_name, map_name, 'value', enum=enum) -def put_req_nested(ri, struct): +def put_req_nested_prototype(ri, struct, suffix=';'): func_args = ['struct nlmsghdr *nlh', 'unsigned int attr_type', f'{struct.ptr_name}obj'] - ri.cw.write_func_prot('int', f'{struct.render_name}_put', func_args) + ri.cw.write_func_prot('int', f'{struct.render_name}_put', func_args, + suffix=suffix) + + +def put_req_nested(ri, struct): + put_req_nested_prototype(ri, struct, suffix='') ri.cw.block_start() ri.cw.write_func_lvar('struct nlattr *nest;')
@@ -1726,18 +1735,23 @@ _C_KW = { ri.cw.nl() -def parse_rsp_nested(ri, struct): +def parse_rsp_nested_prototype(ri, struct, suffix=';'): func_args = ['struct ynl_parse_arg *yarg', 'const struct nlattr *nested'] for arg in struct.inherited: func_args.append('__u32 ' + arg) + ri.cw.write_func_prot('int', f'{struct.render_name}_parse', func_args, + suffix=suffix) + + +def parse_rsp_nested(ri, struct): + parse_rsp_nested_prototype(ri, struct, suffix='') + local_vars = ['const struct nlattr *attr;', f'{struct.ptr_name}dst = yarg->data;'] init_lines = [] - ri.cw.write_func_prot('int', f'{struct.render_name}_parse', func_args) - _multi_parse(ri, struct, init_lines, local_vars)
@@ -2051,6 +2065,10 @@ _C_KW = { ri.cw.nl() +def free_rsp_nested_prototype(ri): + print_free_prototype(ri, "") + + def free_rsp_nested(ri, struct): _free_type(ri, "", struct)
@@ -2818,7 +2836,14 @@ _C_KW = { put_enum_to_str(parsed, cw, const) cw.nl() + has_recursive_nests = False cw.p('/* Policies */') + for struct in parsed.pure_nested_structs.values(): + if struct.recursive: + put_typol_fwd(cw, struct) + has_recursive_nests = True + if has_recursive_nests: + cw.nl() for name in parsed.pure_nested_structs: struct = Struct(parsed, name) put_typol(cw, struct)
@@ -2827,6 +2852,15 @@ _C_KW = { put_typol(cw, struct) cw.p('/* Common nested types */') + if has_recursive_nests: + for attr_set, struct in parsed.pure_nested_structs.items(): + ri = RenderInfo(cw, parsed, args.mode, "", "", attr_set) + free_rsp_nested_prototype(ri) + if struct.request: + put_req_nested_prototype(ri, struct) + if struct.reply: + parse_rsp_nested_prototype(ri, struct) + cw.nl() for attr_set, struct in parsed.pure_nested_structs.items(): ri = RenderInfo(cw, parsed, args.mode, "", "", attr_set)
Signature · 1 line
2.43.0
Re: [PATCH net-next 8/8] tools: ynl-gen: print prototypes for recursive stuff
Quoted 5 lines · Jakub Kicinski
We avoid printing forward declarations and prototypes for most types by sorting things topologically. But if structs nest we do need the forward declarations, there's no other way.
Signed-off-by Jakub Kicinski <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 14 lines · On Wed, 13 Dec 2023 15:14:24 -0800 you wrote:
Fill in the gaps in YNL C code gen so that we can generate user space code for all genetlink families for which we have specs.
The two major changes we need are support for fixed headers and support for recursive nests.
For fixed header support - place the struct for the fixed header directly in the request struct (and don't bother generating access helpers). The member of a fixed header can't be too complex, and also are by definition not optional so the user has to fill them in. The YNL core needs a bit of a tweak to understand that the attrs may now start at a fixed offset, which is not necessarily equal to sizeof(struct genlmsghdr).
Quoted 1 line
[...]
Here is the summary with links:
- [net-next,1/8] tools: ynl-gen: add missing request free helpers for dumps
https://git.kernel.org/netdev/net-next/c/4dc27587dcba - [net-next,2/8] tools: ynl-gen: use enum user type for members and args https://git.kernel.org/netdev/net-next/c/139c163b5b0b - [net-next,3/8] tools: ynl-gen: support fixed headers in genetlink https://git.kernel.org/netdev/net-next/c/f6805072c2aa - [net-next,4/8] tools: ynl-gen: fill in implementations for TypeUnused https://git.kernel.org/netdev/net-next/c/f967a498fce8 - [net-next,5/8] tools: ynl-gen: record information about recursive nests https://git.kernel.org/netdev/net-next/c/38329fcfb757 - [net-next,6/8] tools: ynl-gen: re-sort ignoring recursive nests https://git.kernel.org/netdev/net-next/c/aa75783b95a1 - [net-next,7/8] tools: ynl-gen: store recursive nests by a pointer https://git.kernel.org/netdev/net-next/c/461f25a2e433 - [net-next,8/8] tools: ynl-gen: print prototypes for recursive stuff https://git.kernel.org/netdev/net-next/c/7b5fe80ebc63
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html