Skip to contentSkip to conversation

tools: ynl: more docs and basic ethtool support

PATCH Applied Jakub Kicinski 2 participants 16 messages Jan 31, 2023 – Feb 1, 2023

Subsystems (5)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v2 cover tools: ynl: more docs and basic ethtool support

I got discouraged from supporting ethtool in specs, because generating the user space C code seems a little tricky. The messages are ID'ed in a "directional" way (to and from kernel are separate ID "spaces"). There is value, however, in having the spec and being able to for example use it in Python.

After paying off some technical debt - add a partial ethtool spec. Partial because the header for ethtool is almost a 1000 LoC, so converting in one sitting is tough. But adding new commands should be trivial now.

Last but not least I add more docs, I realized that I've been sending a similar "instructions" email to people working on new families. It's now intro-specs.rst.

v2:
- spelling fixes (patch 11)
- always use python3 (new patch 14)

Jakub Kicinski (14):
tools: ynl-gen: prevent do / dump reordering
tools: ynl: move the cli and netlink code around
tools: ynl: add an object hierarchy to represent parsed spec
tools: ynl: use the common YAML loading and validation code
tools: ynl: add support for types needed by ethtool
tools: ynl: support directional enum-model in CLI
tools: ynl: support multi-attr
tools: ynl: support pretty printing bad attribute names
tools: ynl: use operation names from spec on the CLI
tools: ynl: load jsonschema on demand
netlink: specs: finish up operation enum-models
netlink: specs: add partial specification for ethtool
docs: netlink: add a starting guide for working with specs
tools: net: use python3 explicitly

 Documentation/netlink/genetlink-c.yaml        |   4 +-
 Documentation/netlink/genetlink-legacy.yaml   |  11 +-
 Documentation/netlink/genetlink.yaml          |   4 +-
 Documentation/netlink/specs/ethtool.yaml      | 392 ++++++++++++++++++
 .../netlink/genetlink-legacy.rst              |  82 ++++
 Documentation/userspace-api/netlink/index.rst |   1 +
 .../userspace-api/netlink/intro-specs.rst     |  80 ++++
 Documentation/userspace-api/netlink/specs.rst |   3 +
 tools/net/ynl/{samples => }/cli.py            |  17 +-
 tools/net/ynl/lib/__init__.py                 |   7 +
 tools/net/ynl/lib/nlspec.py                   | 310 ++++++++++++++
 tools/net/ynl/{samples => lib}/ynl.py         | 192 +++++----
 tools/net/ynl/ynl-gen-c.py                    | 262 ++++++------
 13 files changed, 1109 insertions(+), 256 deletions(-)
 create mode 100644 Documentation/netlink/specs/ethtool.yaml
 create mode 100644 Documentation/userspace-api/netlink/intro-specs.rst
 rename tools/net/ynl/{samples => }/cli.py (76%)
 create mode 100644 tools/net/ynl/lib/__init__.py
 create mode 100644 tools/net/ynl/lib/nlspec.py
 rename tools/net/ynl/{samples => lib}/ynl.py (79%)
Signature · 1 line
2.39.1
author maintainer
#1 lore raw

PATCH v2 1/14 tools: ynl-gen: prevent do / dump reordering

An earlier fix tried to address generated code jumping around one code-gen run to another. Turns out dict()s are already ordered since Python 3.7, the problem is that we iterate over operation modes using a set(). Sets are unordered in Python.

---
 tools/net/ynl/ynl-gen-c.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/net/ynl/ynl-gen-c.py +2 -2
index 1aa872e582ab..e5002d420961 100755
@@ -933,7 +933,7 @@ import yaml
             if attr_set_name != op['attribute-set']:
                 raise Exception('For a global policy all ops must use the same set')
 
-            for op_mode in {'do', 'dump'}:
+            for op_mode in ['do', 'dump']:
                 if op_mode in op:
                     global_set.update(op[op_mode].get('request', []))
 
@@ -2244,7 +2244,7 @@ _C_KW = {
 
             for op_name, op in parsed.ops.items():
                 if parsed.kernel_policy in {'per-op', 'split'}:
-                    for op_mode in {'do', 'dump'}:
+                    for op_mode in ['do', 'dump']:
                         if op_mode in op and 'request' in op[op_mode]:
                             cw.p(f"/* {op.enum_name} - {op_mode} */")
                             ri = RenderInfo(cw, parsed, args.mode, op, op_name, op_mode)
Signature · 1 line
2.39.1
author maintainer
#2 lore raw

PATCH v2 2/14 tools: ynl: move the cli and netlink code around

In reply to Jakub Kicinski

Move the CLI code out of samples/ and the library part of it into tools/net/ynl/lib/. This way we can start sharing some code with the code gen.

Initially I thought that code gen is too C-specific to share anything but basic stuff like calculating values for enums can easily be shared.

---
 tools/net/ynl/{samples => }/cli.py    | 2 +-
 tools/net/ynl/lib/__init__.py         | 5 +++++
 tools/net/ynl/{samples => lib}/ynl.py | 0
 3 files changed, 6 insertions(+), 1 deletion(-)
 rename tools/net/ynl/{samples => }/cli.py (97%)
 create mode 100644 tools/net/ynl/lib/__init__.py
 rename tools/net/ynl/{samples => lib}/ynl.py (100%)
tools/net/ynl/cli.py renamed from tools/net/ynl/samples/cli.py +1 -1
similarity index 97%
rename from tools/net/ynl/samples/cli.py
rename to tools/net/ynl/cli.py
index b27159c70710..5c4eb5a68514 100755
@@ -6,7 +6,7 @@ import json
 import pprint
 import time
 
-from ynl import YnlFamily
+from lib import YnlFamily
 
 
 def main():
tools/net/ynl/lib/__init__.py new file +5 -0
new file mode 100644
index 000000000000..0a6102758ebe
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-3-Clause
+
+from .ynl import YnlFamily
+
+__all__ = ["YnlFamily"]
tools/net/ynl/lib/ynl.py renamed from tools/net/ynl/samples/ynl.py +0 -0
similarity index 100%
rename from tools/net/ynl/samples/ynl.py
rename to tools/net/ynl/lib/ynl.py
Signature · 1 line
2.39.1
author maintainer
#3 lore raw

PATCH v2 3/14 tools: ynl: add an object hierarchy to represent parsed spec

In reply to Jakub Kicinski

There's a lot of copy and pasting going on between the "cli" and code gen when it comes to representing the parsed spec. Create a library which both can use.

---
 tools/net/ynl/lib/__init__.py |   4 +-
 tools/net/ynl/lib/nlspec.py   | 301 ++++++++++++++++++++++++++++++++++
 2 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 tools/net/ynl/lib/nlspec.py
tools/net/ynl/lib/__init__.py +3 -1
index 0a6102758ebe..3c73f59eabab 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 
+from .nlspec import SpecAttr, SpecAttrSet, SpecFamily, SpecOperation
 from .ynl import YnlFamily
 
-__all__ = ["YnlFamily"]
+__all__ = ["SpecAttr", "SpecAttrSet", "SpecFamily", "SpecOperation",
+           "YnlFamily"]
tools/net/ynl/lib/nlspec.py new file +301 -0

301 lines not on this page · raw message

Signature · 1 line
2.39.1
author maintainer
#4 lore raw

PATCH v2 4/14 tools: ynl: use the common YAML loading and validation code

In reply to Jakub Kicinski

Adapt the common object hierarchy in code gen and CLI.

---
 tools/net/ynl/lib/ynl.py   | 118 ++++-------------
 tools/net/ynl/ynl-gen-c.py | 256 +++++++++++++++++--------------------
 2 files changed, 142 insertions(+), 232 deletions(-)
tools/net/ynl/lib/ynl.py +25 -93
index b71523d71d46..0ceb627ba686 100644
@@ -1,13 +1,14 @@
 # SPDX-License-Identifier: BSD-3-Clause
 
 import functools
-import jsonschema
 import os
 import random
 import socket
 import struct
 import yaml
 
+from .nlspec import SpecFamily
+
 #
 # Generic Netlink code which should really be in some library, but I can't quickly find one.
 #
@@ -158,8 +159,8 @@ import yaml
                 # We don't have the ability to parse nests yet, so only do global
                 if 'miss-type' in self.extack and 'miss-nest' not in self.extack:
                     miss_type = self.extack['miss-type']
-                    if len(attr_space.attr_list) > miss_type:
-                        spec = attr_space.attr_list[miss_type]
+                    if miss_type in attr_space.attrs_by_val:
+                        spec = attr_space.attrs_by_val[miss_type]
                         desc = spec['name']
                         if 'doc' in spec:
                             desc += f" ({spec['doc']})"
@@ -289,100 +290,31 @@ genl_family_name_to_id = None
 #
 
 
-class YnlAttrSpace:
-    def __init__(self, family, yaml):
-        self.yaml = yaml
-
-        self.attrs = dict()
-        self.name = self.yaml['name']
-        self.subspace_of = self.yaml['subset-of'] if 'subspace-of' in self.yaml else None
-
-        val = 0
-        max_val = 0
-        for elem in self.yaml['attributes']:
-            if 'value' in elem:
-                val = elem['value']
-            else:
-                elem['value'] = val
-            if val > max_val:
-                max_val = val
-            val += 1
-
-            self.attrs[elem['name']] = elem
-
-        self.attr_list = [None] * (max_val + 1)
-        for elem in self.yaml['attributes']:
-            self.attr_list[elem['value']] = elem
-
-    def __getitem__(self, key):
-        return self.attrs[key]
-
-    def __contains__(self, key):
-        return key in self.yaml
-
-    def __iter__(self):
-        yield from self.attrs
-
-    def items(self):
-        return self.attrs.items()
-
-
-class YnlFamily:
+class YnlFamily(SpecFamily):
     def __init__(self, def_path, schema=None):
-        self.include_raw = False
+        super().__init__(def_path, schema)
 
-        with open(def_path, "r") as stream:
-            self.yaml = yaml.safe_load(stream)
-
-        if schema:
-            with open(schema, "r") as stream:
-                schema = yaml.safe_load(stream)
-
-            jsonschema.validate(self.yaml, schema)
+        self.include_raw = False
 
         self.sock = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, Netlink.NETLINK_GENERIC)
         self.sock.setsockopt(Netlink.SOL_NETLINK, Netlink.NETLINK_CAP_ACK, 1)
         self.sock.setsockopt(Netlink.SOL_NETLINK, Netlink.NETLINK_EXT_ACK, 1)
 
-        self._ops = dict()
-        self._spaces = dict()
         self._types = dict()
 
-        for elem in self.yaml['attribute-sets']:
-            self._spaces[elem['name']] = YnlAttrSpace(self, elem)
-
         for elem in self.yaml['definitions']:
             self._types[elem['name']] = elem
 
-        async_separation = 'async-prefix' in self.yaml['operations']
         self.async_msg_ids = set()
         self.async_msg_queue = []
-        val = 0
-        max_val = 0
-        for elem in self.yaml['operations']['list']:
-            if not (async_separation and ('notify' in elem or 'event' in elem)):
-                if 'value' in elem:
-                    val = elem['value']
-                else:
-                    elem['value'] = val
-                val += 1
-                max_val = max(val, max_val)
-
-            if 'notify' in elem or 'event' in elem:
-                self.async_msg_ids.add(elem['value'])
-
-            self._ops[elem['name']] = elem
-
-            op_name = elem['name'].replace('-', '_')
 
-            bound_f = functools.partial(self._op, elem['name'])
-            setattr(self, op_name, bound_f)
+        for msg in self.msgs.values():
+            if msg.is_async:
+                self.async_msg_ids.add(msg.value)
 
-        self._op_array = [None] * max_val
-        for _, op in self._ops.items():
-            self._op_array[op['value']] = op
-            if 'notify' in op:
-                op['attribute-set'] = self._ops[op['notify']]['attribute-set']
+        for op_name, op in self.ops.items():
+            bound_f = functools.partial(self._op, op_name)
+            setattr(self, op.ident_name, bound_f)
 
         self.family = GenlFamily(self.yaml['name'])
 
@@ -395,8 +327,8 @@ genl_family_name_to_id = None
                              self.family.genl_family['mcast'][mcast_name])
 
     def _add_attr(self, space, name, value):
-        attr = self._spaces[space][name]
-        nl_type = attr['value']
+        attr = self.attr_sets[space][name]
+        nl_type = attr.value
         if attr["type"] == 'nest':
             nl_type |= Netlink.NLA_F_NESTED
             attr_payload = b''
@@ -430,10 +362,10 @@ genl_family_name_to_id = None
         rsp[attr_spec['name']] = value
 
     def _decode(self, attrs, space):
-        attr_space = self._spaces[space]
+        attr_space = self.attr_sets[space]
         rsp = dict()
         for attr in attrs:
-            attr_spec = attr_space.attr_list[attr.type]
+            attr_spec = attr_space.attrs_by_val[attr.type]
             if attr_spec["type"] == 'nest':
                 subdict = self._decode(NlAttrs(attr.raw), attr_spec['nested-attributes'])
                 rsp[attr_spec['name']] = subdict
@@ -457,9 +389,9 @@ genl_family_name_to_id = None
         if self.include_raw:
             msg['nlmsg'] = nl_msg
             msg['genlmsg'] = genl_msg
-        op = self._op_array[genl_msg.genl_cmd]
+        op = self.msgs_by_value[genl_msg.genl_cmd]
         msg['name'] = op['name']
-        msg['msg'] = self._decode(genl_msg.raw_attrs, op['attribute-set'])
+        msg['msg'] = self._decode(genl_msg.raw_attrs, op.attr_set.name)
         self.async_msg_queue.append(msg)
 
     def check_ntf(self):
@@ -487,16 +419,16 @@ genl_family_name_to_id = None
                 self.handle_ntf(nl_msg, gm)
 
     def _op(self, method, vals, dump=False):
-        op = self._ops[method]
+        op = self.ops[method]
 
         nl_flags = Netlink.NLM_F_REQUEST | Netlink.NLM_F_ACK
         if dump:
             nl_flags |= Netlink.NLM_F_DUMP
 
         req_seq = random.randint(1024, 65535)
-        msg = _genl_msg(self.family.family_id, nl_flags, op['value'], 1, req_seq)
+        msg = _genl_msg(self.family.family_id, nl_flags, op.value, 1, req_seq)
         for name, value in vals.items():
-            msg += self._add_attr(op['attribute-set'], name, value)
+            msg += self._add_attr(op.attr_set.name, name, value)
         msg = _genl_msg_finalize(msg)
 
         self.sock.send(msg, 0)
@@ -505,7 +437,7 @@ genl_family_name_to_id = None
         rsp = []
         while not done:
             reply = self.sock.recv(128 * 1024)
-            nms = NlMsgs(reply, attr_space=self._spaces[op['attribute-set']])
+            nms = NlMsgs(reply, attr_space=op.attr_set)
             for nl_msg in nms:
                 if nl_msg.error:
                     print("Netlink error:", os.strerror(-nl_msg.error))
@@ -517,7 +449,7 @@ genl_family_name_to_id = None
 
                 gm = GenlMsg(nl_msg)
                 # Check if this is a reply to our request
-                if nl_msg.nl_seq != req_seq or gm.genl_cmd != op['value']:
+                if nl_msg.nl_seq != req_seq or gm.genl_cmd != op.value:
                     if gm.genl_cmd in self.async_msg_ids:
                         self.handle_ntf(nl_msg, gm)
                         continue
@@ -525,7 +457,7 @@ genl_family_name_to_id = None
                         print('Unexpected message: ' + repr(gm))
                         continue
 
-                rsp.append(self._decode(gm.raw_attrs, op['attribute-set']))
+                rsp.append(self._decode(gm.raw_attrs, op.attr_set.name))
 
         if not rsp:
             return None
tools/net/ynl/ynl-gen-c.py +117 -139

390 lines not on this page · raw message

Signature · 1 line
2.39.1
author maintainer
#5 lore raw

PATCH v2 5/14 tools: ynl: add support for types needed by ethtool

In reply to Jakub Kicinski

Ethtool needs support for handful of extra types.
It doesn't have the definitions section yet.

---
 tools/net/ynl/lib/ynl.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
tools/net/ynl/lib/ynl.py +10 -1
index 0ceb627ba686..a656b655d302 100644
@@ -75,6 +75,9 @@ from .nlspec import SpecFamily
         self.full_len = (self.payload_len + 3) & ~3
         self.raw = raw[offset + 4:offset + self.payload_len]
 
+    def as_u8(self):
+        return struct.unpack("B", self.raw)[0]
+
     def as_u16(self):
         return struct.unpack("H", self.raw)[0]
 
@@ -302,7 +305,7 @@ genl_family_name_to_id = None
 
         self._types = dict()
 
-        for elem in self.yaml['definitions']:
+        for elem in self.yaml.get('definitions', []):
             self._types[elem['name']] = elem
 
         self.async_msg_ids = set()
@@ -334,6 +337,8 @@ genl_family_name_to_id = None
             attr_payload = b''
             for subname, subvalue in value.items():
                 attr_payload += self._add_attr(attr['nested-attributes'], subname, subvalue)
+        elif attr["type"] == 'flag':
+            attr_payload = b''
         elif attr["type"] == 'u32':
             attr_payload = struct.pack("I", int(value))
         elif attr["type"] == 'string':
@@ -369,6 +374,8 @@ genl_family_name_to_id = None
             if attr_spec["type"] == 'nest':
                 subdict = self._decode(NlAttrs(attr.raw), attr_spec['nested-attributes'])
                 rsp[attr_spec['name']] = subdict
+            elif attr_spec['type'] == 'u8':
+                rsp[attr_spec['name']] = attr.as_u8()
             elif attr_spec['type'] == 'u32':
                 rsp[attr_spec['name']] = attr.as_u32()
             elif attr_spec['type'] == 'u64':
@@ -377,6 +384,8 @@ genl_family_name_to_id = None
                 rsp[attr_spec['name']] = attr.as_strz()
             elif attr_spec["type"] == 'binary':
                 rsp[attr_spec['name']] = attr.as_bin()
+            elif attr_spec["type"] == 'flag':
+                rsp[attr_spec['name']] = True
             else:
                 raise Exception(f'Unknown {attr.type} {attr_spec["name"]} {attr_spec["type"]}')
 
Signature · 1 line
2.39.1
author maintainer
#6 lore raw

PATCH v2 6/14 tools: ynl: support directional enum-model in CLI

In reply to Jakub Kicinski

Support families which use different IDs for messages to and from the kernel.

---
 tools/net/ynl/lib/ynl.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tools/net/ynl/lib/ynl.py +4 -4
index a656b655d302..690065003935 100644
@@ -313,7 +313,7 @@ genl_family_name_to_id = None
 
         for msg in self.msgs.values():
             if msg.is_async:
-                self.async_msg_ids.add(msg.value)
+                self.async_msg_ids.add(msg.rsp_value)
 
         for op_name, op in self.ops.items():
             bound_f = functools.partial(self._op, op_name)
@@ -398,7 +398,7 @@ genl_family_name_to_id = None
         if self.include_raw:
             msg['nlmsg'] = nl_msg
             msg['genlmsg'] = genl_msg
-        op = self.msgs_by_value[genl_msg.genl_cmd]
+        op = self.rsp_by_value[genl_msg.genl_cmd]
         msg['name'] = op['name']
         msg['msg'] = self._decode(genl_msg.raw_attrs, op.attr_set.name)
         self.async_msg_queue.append(msg)
@@ -435,7 +435,7 @@ genl_family_name_to_id = None
             nl_flags |= Netlink.NLM_F_DUMP
 
         req_seq = random.randint(1024, 65535)
-        msg = _genl_msg(self.family.family_id, nl_flags, op.value, 1, req_seq)
+        msg = _genl_msg(self.family.family_id, nl_flags, op.req_value, 1, req_seq)
         for name, value in vals.items():
             msg += self._add_attr(op.attr_set.name, name, value)
         msg = _genl_msg_finalize(msg)
@@ -458,7 +458,7 @@ genl_family_name_to_id = None
 
                 gm = GenlMsg(nl_msg)
                 # Check if this is a reply to our request
-                if nl_msg.nl_seq != req_seq or gm.genl_cmd != op.value:
+                if nl_msg.nl_seq != req_seq or gm.genl_cmd != op.rsp_value:
                     if gm.genl_cmd in self.async_msg_ids:
                         self.handle_ntf(nl_msg, gm)
                         continue
Signature · 1 line
2.39.1
author maintainer
#7 lore raw

PATCH v2 7/14 tools: ynl: support multi-attr

In reply to Jakub Kicinski

Ethtool uses mutli-attr, add the support to YNL.

---
 tools/net/ynl/lib/ynl.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
tools/net/ynl/lib/ynl.py +14 -7
index 690065003935..c16326495cb7 100644
@@ -373,22 +373,29 @@ genl_family_name_to_id = None
             attr_spec = attr_space.attrs_by_val[attr.type]
             if attr_spec["type"] == 'nest':
                 subdict = self._decode(NlAttrs(attr.raw), attr_spec['nested-attributes'])
-                rsp[attr_spec['name']] = subdict
+                decoded = subdict
             elif attr_spec['type'] == 'u8':
-                rsp[attr_spec['name']] = attr.as_u8()
+                decoded = attr.as_u8()
             elif attr_spec['type'] == 'u32':
-                rsp[attr_spec['name']] = attr.as_u32()
+                decoded = attr.as_u32()
             elif attr_spec['type'] == 'u64':
-                rsp[attr_spec['name']] = attr.as_u64()
+                decoded = attr.as_u64()
             elif attr_spec["type"] == 'string':
-                rsp[attr_spec['name']] = attr.as_strz()
+                decoded = attr.as_strz()
             elif attr_spec["type"] == 'binary':
-                rsp[attr_spec['name']] = attr.as_bin()
+                decoded = attr.as_bin()
             elif attr_spec["type"] == 'flag':
-                rsp[attr_spec['name']] = True
+                decoded = True
             else:
                 raise Exception(f'Unknown {attr.type} {attr_spec["name"]} {attr_spec["type"]}')
 
+            if not attr_spec.is_multi:
+                rsp[attr_spec['name']] = decoded
+            elif attr_spec.name in rsp:
+                rsp[attr_spec.name].append(decoded)
+            else:
+                rsp[attr_spec.name] = [decoded]
+
             if 'enum' in attr_spec:
                 self._decode_enum(rsp, attr_spec)
         return rsp
Signature · 1 line
2.39.1
author maintainer
#8 lore raw

PATCH v2 8/14 tools: ynl: support pretty printing bad attribute names

In reply to Jakub Kicinski

One of my favorite features of the Netlink specs is that they make decoding structured extack a ton easier. Implement pretty printing bad attribute names in YNL.

For example it will now say:

'bad-attr': '.header.flags'

rather than the useless:

'bad-attr-offs': 32

Proof:

  $ ./cli.py --spec ethtool.yaml --do rings_get \
     --json '{"header":{"dev-index":1, "flags":4}}'
  Netlink error: Invalid argument
  nl_len = 68 (52) nl_flags = 0x300 nl_type = 2
	error: -22	extack: {'msg': 'reserved bit set',
				 'bad-attr': '.header.flags'}
---
 tools/net/ynl/lib/ynl.py | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
tools/net/ynl/lib/ynl.py +40 -0
index c16326495cb7..2ff3e6dbdbf6 100644
@@ -400,6 +400,40 @@ genl_family_name_to_id = None
                 self._decode_enum(rsp, attr_spec)
         return rsp
 
+    def _decode_extack_path(self, attrs, attr_set, offset, target):
+        for attr in attrs:
+            attr_spec = attr_set.attrs_by_val[attr.type]
+            if offset > target:
+                break
+            if offset == target:
+                return '.' + attr_spec.name
+
+            if offset + attr.full_len <= target:
+                offset += attr.full_len
+                continue
+            if attr_spec['type'] != 'nest':
+                raise Exception(f"Can't dive into {attr.type} ({attr_spec['name']}) for extack")
+            offset += 4
+            subpath = self._decode_extack_path(NlAttrs(attr.raw),
+                                               self.attr_sets[attr_spec['nested-attributes']],
+                                               offset, target)
+            if subpath is None:
+                return None
+            return '.' + attr_spec.name + subpath
+
+        return None
+
+    def _decode_extack(self, request, attr_space, extack):
+        if 'bad-attr-offs' not in extack:
+            return
+
+        genl_req = GenlMsg(NlMsg(request, 0, attr_space=attr_space))
+        path = self._decode_extack_path(genl_req.raw_attrs, attr_space,
+                                        20, extack['bad-attr-offs'])
+        if path:
+            del extack['bad-attr-offs']
+            extack['bad-attr'] = path
+
     def handle_ntf(self, nl_msg, genl_msg):
         msg = dict()
         if self.include_raw:
@@ -455,11 +489,17 @@ genl_family_name_to_id = None
             reply = self.sock.recv(128 * 1024)
             nms = NlMsgs(reply, attr_space=op.attr_set)
             for nl_msg in nms:
+                if nl_msg.extack:
+                    self._decode_extack(msg, op.attr_set, nl_msg.extack)
+
                 if nl_msg.error:
                     print("Netlink error:", os.strerror(-nl_msg.error))
                     print(nl_msg)
                     return
                 if nl_msg.done:
+                    if nl_msg.extack:
+                        print("Netlink warning:")
+                        print(nl_msg)
                     done = True
                     break
 
Signature · 1 line
2.39.1
author maintainer
#9 lore raw

PATCH v2 9/14 tools: ynl: use operation names from spec on the CLI

In reply to Jakub Kicinski

When I wrote the first version of the Python code I was quite excited that we can generate class methods directly from the spec. Unfortunately we need to use valid identifiers for method names (specifically no dashes are allowed). Don't reuse those names on the CLI, it's much more natural to use the operation names exactly as listed in the spec.

Instead of:
./cli --do rings_get
use:
./cli --do rings-get

---
 tools/net/ynl/cli.py     | 9 +++++----
 tools/net/ynl/lib/ynl.py | 6 ++++++
 2 files changed, 11 insertions(+), 4 deletions(-)
tools/net/ynl/cli.py +5 -4
index 5c4eb5a68514..05d1f4069ce1 100755
@@ -32,10 +32,11 @@ from lib import YnlFamily
     if args.sleep:
         time.sleep(args.sleep)
 
-    if args.do or args.dump:
-        method = getattr(ynl, args.do if args.do else args.dump)
-
-        reply = method(attrs, dump=bool(args.dump))
+    if args.do:
+        reply = ynl.do(args.do, attrs)
+        pprint.PrettyPrinter().pprint(reply)
+    if args.dump:
+        reply = ynl.dump(args.dump, attrs)
         pprint.PrettyPrinter().pprint(reply)
 
     if args.ntf:
tools/net/ynl/lib/ynl.py +6 -0
index 2ff3e6dbdbf6..1c7411ee04dc 100644
@@ -520,3 +520,9 @@ genl_family_name_to_id = None
         if not dump and len(rsp) == 1:
             return rsp[0]
         return rsp
+
+    def do(self, method, vals):
+        return self._op(method, vals)
+
+    def dump(self, method, vals):
+        return self._op(method, vals, dump=True)
Signature · 1 line
2.39.1
author maintainer
#10 lore raw

PATCH v2 10/14 tools: ynl: load jsonschema on demand

In reply to Jakub Kicinski

The CLI script tries to validate jsonschema by default. It's seems better to validate too many times than too few. However, when copying the scripts to random servers having to install jsonschema is tedious. Load jsonschema via importlib, and let the user opt out.

---
 tools/net/ynl/cli.py        |  4 ++++
 tools/net/ynl/lib/nlspec.py | 11 ++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
tools/net/ynl/cli.py +4 -0
index 05d1f4069ce1..e64f1478764f 100755
@@ -13,6 +13,7 @@ from lib import YnlFamily
     parser = argparse.ArgumentParser(description='YNL CLI sample')
     parser.add_argument('--spec', dest='spec', type=str, required=True)
     parser.add_argument('--schema', dest='schema', type=str)
+    parser.add_argument('--no-schema', action='store_true')
     parser.add_argument('--json', dest='json_text', type=str)
     parser.add_argument('--do', dest='do', type=str)
     parser.add_argument('--dump', dest='dump', type=str)
@@ -20,6 +21,9 @@ from lib import YnlFamily
     parser.add_argument('--subscribe', dest='ntf', type=str)
     args = parser.parse_args()
 
+    if args.no_schema:
+        args.schema = ''
+
     attrs = {}
     if args.json_text:
         attrs = json.loads(args.json_text)
tools/net/ynl/lib/nlspec.py +10 -1
index 4aa3b1ad97f0..e204679ad8b7 100644
@@ -1,12 +1,16 @@
 # SPDX-License-Identifier: BSD-3-Clause
 
 import collections
-import jsonschema
+import importlib
 import os
 import traceback
 import yaml
 
 
+# To be loaded dynamically as needed
+jsonschema = None
+
+
 class SpecElement:
     """Netlink spec element.
 
@@ -197,9 +201,14 @@ import yaml
         if schema_path is None:
             schema_path = os.path.dirname(os.path.dirname(spec_path)) + f'/{self.proto}.yaml'
         if schema_path:
+            global jsonschema
+
             with open(schema_path, "r") as stream:
                 schema = yaml.safe_load(stream)
 
+            if jsonschema is None:
+                jsonschema = importlib.import_module("jsonschema")
+
             jsonschema.validate(self.yaml, schema)
 
         self.attr_sets = collections.OrderedDict()
Signature · 1 line
2.39.1
author maintainer
#11 lore raw

PATCH v2 11/14 netlink: specs: finish up operation enum-models

In reply to Jakub Kicinski

I had a (bright?) idea of introducing the concept of enum-models to account for all the weird ways families enumerate their messages. I've never finished it because generating C code for each of them is pretty daunting. But for languages which can use ID values directly the support is simple enough, so clean this up a bit.

"unified" model is what I recommend going forward. "directional" model is what ethtool uses. "notify-split" is used by the proposed DPLL code, but we can just make them use "unified", it hasn't been merged :)

---
v2: spelling fixes

---
 Documentation/netlink/genetlink-c.yaml        |  4 +-
 Documentation/netlink/genetlink-legacy.yaml   | 11 ++-
 Documentation/netlink/genetlink.yaml          |  4 +-
 .../netlink/genetlink-legacy.rst              | 82 +++++++++++++++++++
 4 files changed, 92 insertions(+), 9 deletions(-)
Documentation/netlink/genetlink-c.yaml +1 -3
index e23e3c94a932..bbcfa2472b04 100644
@@ -218,9 +218,7 @@ additionalProperties: False
           to a single enum.
           "directional" has the messages sent to the kernel and from the kernel
           enumerated separately.
-          "notify-split" has the notifications and request-response types in
-          different enums.
-        enum: [ unified, directional, notify-split ]
+        enum: [ unified ]
       name-prefix:
         description: |
           Prefix for the C enum name of the command. The name is formed by concatenating
Documentation/netlink/genetlink-legacy.yaml +8 -3
index 88db2431ef26..5642925c4ceb 100644
@@ -241,9 +241,7 @@ additionalProperties: False
           to a single enum.
           "directional" has the messages sent to the kernel and from the kernel
           enumerated separately.
-          "notify-split" has the notifications and request-response types in
-          different enums.
-        enum: [ unified, directional, notify-split ]
+        enum: [ unified, directional ] # Trim
       name-prefix:
         description: |
           Prefix for the C enum name of the command. The name is formed by concatenating
@@ -307,6 +305,13 @@ additionalProperties: False
                       type: array
                       items:
                         type: string
+                    # Start genetlink-legacy
+                    value:
+                      description: |
+                        ID of this message if value for request and response differ,
+                        i.e. requests and responses have different message enums.
+                      $ref: '#/$defs/uint'
+                    # End genetlink-legacy
                 reply: *subop-attr-list
                 pre:
                   description: Hook for a function to run before the main callback (pre_doit or start).
Documentation/netlink/genetlink.yaml +1 -3
index b5e712bbe7e7..62a922755ce2 100644
@@ -188,9 +188,7 @@ additionalProperties: False
           to a single enum.
           "directional" has the messages sent to the kernel and from the kernel
           enumerated separately.
-          "notify-split" has the notifications and request-response types in
-          different enums.
-        enum: [ unified, directional, notify-split ]
+        enum: [ unified ]
       name-prefix:
         description: |
           Prefix for the C enum name of the command. The name is formed by concatenating
Documentation/userspace-api/netlink/genetlink-legacy.rst +82 -0

88 lines not on this page · raw message

Signature · 1 line
2.39.1
author maintainer
#12 lore raw

PATCH v2 12/14 netlink: specs: add partial specification for ethtool

In reply to Jakub Kicinski

Ethtool is one of the most actively developed families. With the changes to the CLI it should be possible to use the YNL based code for easy prototyping and development. Add a partial family definition. I've tested the string set and rings. I don't have any MAC Merge implementation to test with, but I added the definition for it, anyway, because it's last. New commands can simply be added at the end without having to worry about manually providing IDs / values.

Set (with notification support - None is the response, the data is from the notification):

$ sudo ./tools/net/ynl/cli.py \
    --spec Documentation/netlink/specs/ethtool.yaml \
    --do rings-set \
    --json '{"header":{"dev-name":"enp0s31f6"}, "rx":129}' \
    --subscribe monitor
None
[{'msg': {'header': {'dev-index': 2, 'dev-name': 'enp0s31f6'},
          'rx': 136,
          'rx-max': 4096,
          'tx': 256,
          'tx-max': 4096,
          'tx-push': 0},
  'name': 'rings-ntf'}]

Do / dump (yes, the kernel requires that even for dump and even if empty - the "header" nest must be there):

$ ./tools/net/ynl/cli.py \
    --spec Documentation/netlink/specs/ethtool.yaml \
    --do rings-get \
    --json '{"header":{"dev-index": 2}}'
{'header': {'dev-index': 2, 'dev-name': 'enp0s31f6'},
 'rx': 136,
 'rx-max': 4096,
 'tx': 256,
 'tx-max': 4096,
 'tx-push': 0}
$ ./tools/net/ynl/cli.py \
    --spec Documentation/netlink/specs/ethtool.yaml \
    --dump rings-get \
    --json '{"header":{}}'
[{'header': {'dev-index': 2, 'dev-name': 'enp0s31f6'},
  'rx': 136,
  'rx-max': 4096,
  'tx': 256,
  'tx-max': 4096,
  'tx-push': 0},
 {'header': {'dev-index': 3, 'dev-name': 'wlp0s20f3'}, 'tx-push': 0},
 {'header': {'dev-index': 19, 'dev-name': 'enp58s0u1u1'},
  'rx': 100,
  'rx-max': 4096,
  'tx-push': 0}]

And error reporting:

$ ./tools/net/ynl/cli.py \
    --spec Documentation/netlink/specs/ethtool.yaml \
    --dump rings-get \
    --json '{"header":{"flags":5}}'
Netlink error: Invalid argument
nl_len = 68 (52) nl_flags = 0x300 nl_type = 2
	error: -22	extack: {'msg': 'reserved bit set',
	                         'bad-attr-offs': 24,
				 'bad-attr': '.header.flags'}
None
---
 Documentation/netlink/specs/ethtool.yaml | 392 +++++++++++++++++++++++
 1 file changed, 392 insertions(+)
 create mode 100644 Documentation/netlink/specs/ethtool.yaml
Documentation/netlink/specs/ethtool.yaml new file +392 -0
new file mode 100644
index 000000000000..82f4e6f8ddd3
@@ -0,0 +1,392 @@
+name: ethtool
+
+protocol: genetlink-legacy
+
+doc: Partial family for Ethtool Netlink.
+
+attribute-sets:
+  -
+    name: header
+    attributes:
+      -
+        name: dev-index
+        type: u32
+        value: 1
+      -
+        name: dev-name
+        type: string
+      -
+        name: flags
+        type: u32
+
+  -
+    name: bitset-bit
+    attributes:
+      -
+        name: index
+        type: u32
+        value: 1
+      -
+        name: name
+        type: string
+      -
+        name: value
+        type: flag
+  -
+    name: bitset-bits
+    attributes:
+      -
+        name: bit
+        type: nest
+        nested-attributes: bitset-bit
+        value: 1
+  -
+    name: bitset
+    attributes:
+      -
+        name: nomask
+        type: flag
+        value: 1
+      -
+        name: size
+        type: u32
+      -
+        name: bits
+        type: nest
+        nested-attributes: bitset-bits
+
+  -
+    name: string
+    attributes:
+      -
+        name: index
+        type: u32
+        value: 1
+      -
+        name: value
+        type: string
+  -
+    name: strings
+    attributes:
+      -
+        name: string
+        type: nest
+        value: 1
+        multi-attr: true
+        nested-attributes: string
+  -
+    name: stringset
+    attributes:
+      -
+        name: id
+        type: u32
+        value: 1
+      -
+        name: count
+        type: u32
+      -
+        name: strings
+        type: nest
+        multi-attr: true
+        nested-attributes: strings
+  -
+    name: stringsets
+    attributes:
+      -
+        name: stringset
+        type: nest
+        multi-attr: true
+        value: 1
+        nested-attributes: stringset
+  -
+    name: strset
+    attributes:
+      -
+        name: header
+        value: 1
+        type: nest
+        nested-attributes: header
+      -
+        name: stringsets
+        type: nest
+        nested-attributes: stringsets
+      -
+        name: counts-only
+        type: flag
+
+  -
+    name: privflags
+    attributes:
+      -
+        name: header
+        value: 1
+        type: nest
+        nested-attributes: header
+      -
+        name: flags
+        type: nest
+        nested-attributes: bitset
+
+  -
+    name: rings
+    attributes:
+      -
+        name: header
+        value: 1
+        type: nest
+        nested-attributes: header
+      -
+        name: rx-max
+        type: u32
+      -
+        name: rx-mini-max
+        type: u32
+      -
+        name: rx-jumbo-max
+        type: u32
+      -
+        name: tx-max
+        type: u32
+      -
+        name: rx
+        type: u32
+      -
+        name: rx-mini
+        type: u32
+      -
+        name: rx-jumbo
+        type: u32
+      -
+        name: tx
+        type: u32
+      -
+        name: rx-buf-len
+        type: u32
+      -
+        name: tcp-data-split
+        type: u8
+      -
+        name: cqe-size
+        type: u32
+      -
+        name: tx-push
+        type: u8
+
+  -
+    name: mm-stat
+    attributes:
+      -
+        name: pad
+        value: 1
+        type: pad
+      -
+        name: reassembly-errors
+        type: u64
+      -
+        name: smd-errors
+        type: u64
+      -
+        name: reassembly-ok
+        type: u64
+      -
+        name: rx-frag-count
+        type: u64
+      -
+        name: tx-frag-count
+        type: u64
+      -
+        name: hold-count
+        type: u64
+  -
+    name: mm
+    attributes:
+      -
+        name: header
+        value: 1
+        type: nest
+        nested-attributes: header
+      -
+        name: pmac-enabled
+        type: u8
+      -
+        name: tx-enabled
+        type: u8
+      -
+        name: tx-active
+        type: u8
+      -
+        name: tx-min-frag-size
+        type: u32
+      -
+        name: tx-min-frag-size
+        type: u32
+      -
+        name: verify-enabled
+        type: u8
+      -
+        name: verify-status
+        type: u8
+      -
+        name: verify-time
+        type: u32
+      -
+        name: max-verify-time
+        type: u32
+      -
+        name: stats
+        type: nest
+        nested-attributes: mm-stat
+
+operations:
+  enum-model: directional
+  list:
+    -
+      name: strset-get
+      doc: Get string set from the kernel.
+
+      attribute-set: strset
+
+      do: &strset-get-op
+        request:
+          value: 1
+          attributes:
+            - header
+            - stringsets
+            - counts-only
+        reply:
+          value: 1
+          attributes:
+            - header
+            - stringsets
+      dump: *strset-get-op
+
+    # TODO: fill in the requests in between
+
+    -
+      name: privflags-get
+      doc: Get device private flags.
+
+      attribute-set: privflags
+
+      do: &privflag-get-op
+        request:
+          value: 13
+          attributes:
+            - header
+        reply:
+          value: 14
+          attributes:
+            - header
+            - flags
+      dump: *privflag-get-op
+    -
+      name: privflags-set
+      doc: Set device private flags.
+
+      attribute-set: privflags
+
+      do:
+        request:
+          attributes:
+            - header
+            - flags
+    -
+      name: privflags-ntf
+      doc: Notification for change in device private flags.
+      notify: privflags-get
+
+    -
+      name: rings-get
+      doc: Get ring params.
+
+      attribute-set: rings
+
+      do: &ring-get-op
+        request:
+          attributes:
+            - header
+        reply:
+          attributes:
+            - header
+            - rx-max
+            - rx-mini-max
+            - rx-jumbo-max
+            - tx-max
+            - rx
+            - rx-mini
+            - rx-jumbo
+            - tx
+            - rx-buf-len
+            - tcp-data-split
+            - cqe-size
+            - tx-push
+      dump: *ring-get-op
+    -
+      name: rings-set
+      doc: Set ring params.
+
+      attribute-set: rings
+
+      do:
+        request:
+          attributes:
+            - header
+            - rx
+            - rx-mini
+            - rx-jumbo
+            - tx
+            - rx-buf-len
+            - tcp-data-split
+            - cqe-size
+            - tx-push
+    -
+      name: rings-ntf
+      doc: Notification for change in ring params.
+      notify: rings-get
+
+    # TODO: fill in the requests in between
+
+    -
+      name: mm-get
+      doc: Get MAC Merge configuration and state
+
+      attribute-set: mm
+
+      do: &mm-get-op
+        request:
+          value: 42
+          attributes:
+            - header
+        reply:
+          value: 42
+          attributes:
+            - header
+            - pmac-enabled
+            - tx-enabled
+            - tx-active
+            - tx-min-frag-size
+            - rx-min-frag-size
+            - verify-enabled
+            - verify-time
+            - max-verify-time
+            - stats
+      dump: *mm-get-op
+    -
+      name: mm-set
+      doc: Set MAC Merge configuration
+
+      attribute-set: mm
+
+      do:
+        request:
+          attributes:
+            - header
+            - verify-enabled
+            - verify-time
+            - tx-enabled
+            - pmac-enabled
+            - tx-min-frag-size
+    -
+      name: mm-ntf
+      doc: Notification for change in MAC Merge configuration.
+      notify: mm-get
Signature · 1 line
2.39.1
author maintainer
#13 lore raw

PATCH v2 13/14 docs: netlink: add a starting guide for working with specs

In reply to Jakub Kicinski

We have a bit of documentation about the internals of Netlink and the specs, but really the goal is for most people to not worry about those. Add a practical guide for beginners who want to poke at the specs.

---
 Documentation/userspace-api/netlink/index.rst |  1 +
 .../userspace-api/netlink/intro-specs.rst     | 80 +++++++++++++++++++
 Documentation/userspace-api/netlink/specs.rst |  3 +
 3 files changed, 84 insertions(+)
 create mode 100644 Documentation/userspace-api/netlink/intro-specs.rst
Documentation/userspace-api/netlink/index.rst +1 -0
index be250110c8f6..26f3720cb3be 100644
@@ -10,6 +10,7 @@ Netlink documentation for users.
    :maxdepth: 2
 
    intro
+   intro-specs
    specs
    c-code-gen
    genetlink-legacy
Documentation/userspace-api/netlink/intro-specs.rst new file +80 -0

80 lines not on this page · raw message

Documentation/userspace-api/netlink/specs.rst +3 -0
index 8394d74fc63a..6ffe8137cd90 100644
@@ -21,6 +21,9 @@ kernel headers directly.
 
 YAML specifications can be found under ``Documentation/netlink/specs/``
 
+This document describes details of the schema.
+See :doc:`intro-specs` for a practical starting guide.
+
 Compatibility levels
 ====================
 
Signature · 1 line
2.39.1
author maintainer
#14 lore raw

PATCH v2 14/14 tools: net: use python3 explicitly

In reply to Jakub Kicinski

The scripts require Python 3 and some distros are dropping Python 2 support.

---
 tools/net/ynl/cli.py       | 2 +-
 tools/net/ynl/ynl-gen-c.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/net/ynl/cli.py +1 -1
index e64f1478764f..db410b74d539 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # SPDX-License-Identifier: BSD-3-Clause
 
 import argparse
tools/net/ynl/ynl-gen-c.py +1 -1
index dc14da634e8e..3942f24b9163 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 import argparse
 import collections
Signature · 1 line
2.39.1
#15 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (master) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (master) by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Mon, 30 Jan 2023 18:33:40 -0800 you wrote:

I got discouraged from supporting ethtool in specs, because generating the user space C code seems a little tricky. The messages are ID'ed in a "directional" way (to and from kernel are separate ID "spaces"). There is value, however, in having the spec and being able to for example use it in Python.

[...]

Here is the summary with links:
- [net-next,v2,01/14] tools: ynl-gen: prevent do / dump reordering

    https://git.kernel.org/netdev/net-next/c/eaf317e7d2bb
  - [net-next,v2,02/14] tools: ynl: move the cli and netlink code around
    https://git.kernel.org/netdev/net-next/c/4e4480e89c47
  - [net-next,v2,03/14] tools: ynl: add an object hierarchy to represent parsed spec
    https://git.kernel.org/netdev/net-next/c/3aacf8281336
  - [net-next,v2,04/14] tools: ynl: use the common YAML loading and validation code
    https://git.kernel.org/netdev/net-next/c/30a5c6c8104f
  - [net-next,v2,05/14] tools: ynl: add support for types needed by ethtool
    https://git.kernel.org/netdev/net-next/c/19b64b48a33e
  - [net-next,v2,06/14] tools: ynl: support directional enum-model in CLI
    https://git.kernel.org/netdev/net-next/c/fd0616d34274
  - [net-next,v2,07/14] tools: ynl: support multi-attr
    https://git.kernel.org/netdev/net-next/c/90256f3f8093
  - [net-next,v2,08/14] tools: ynl: support pretty printing bad attribute names
    https://git.kernel.org/netdev/net-next/c/4cd2796f3f8d
  - [net-next,v2,09/14] tools: ynl: use operation names from spec on the CLI
    https://git.kernel.org/netdev/net-next/c/8dfec0a88868
  - [net-next,v2,10/14] tools: ynl: load jsonschema on demand
    https://git.kernel.org/netdev/net-next/c/5c6674f6eb52
  - [net-next,v2,11/14] netlink: specs: finish up operation enum-models
    https://git.kernel.org/netdev/net-next/c/8403bf044530
  - [net-next,v2,12/14] netlink: specs: add partial specification for ethtool
    https://git.kernel.org/netdev/net-next/c/b784db7ae840
  - [net-next,v2,13/14] docs: netlink: add a starting guide for working with specs
    https://git.kernel.org/netdev/net-next/c/01e47a372268
  - [net-next,v2,14/14] tools: net: use python3 explicitly
    https://git.kernel.org/netdev/net-next/c/981cbcb030d9

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html