Skip to contentSkip to conversation

ynl: rename array-nest to indexed-array

PATCH Superseded Hangbin Liu 2 participants 8 messages Apr 1, 2024 – Apr 4, 2024

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover ynl: rename array-nest to indexed-array

rename array-nest to indexed-array and add un-nest sub-type support

v3:
1. fix doc title underline too short issue (Jakub Kicinski)

v2:
1. raise exception for unsupported sub-type
2. merge all sub-type handler in _decode_array_attr
3. remove index shown in indexed-array as some implementations are non-contiguous.

Hangbin Liu (2):
ynl: rename array-nest to indexed-array
ynl: support binary/u32 sub-type for indexed-array

 Documentation/netlink/genetlink-c.yaml        |  2 +-
 Documentation/netlink/genetlink-legacy.yaml   |  2 +-
 Documentation/netlink/genetlink.yaml          |  2 +-
 Documentation/netlink/netlink-raw.yaml        |  2 +-
 Documentation/netlink/specs/nlctrl.yaml       |  6 +++--
 Documentation/netlink/specs/rt_link.yaml      |  3 ++-
 Documentation/netlink/specs/tc.yaml           | 21 ++++++++++------
 .../netlink/genetlink-legacy.rst              | 24 ++++++++++++++-----
 tools/net/ynl/lib/ynl.py                      | 18 ++++++++++----
 tools/net/ynl/ynl-gen-c.py                    | 18 +++++++++-----
 10 files changed, 67 insertions(+), 31 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v3 1/2 ynl: rename array-nest to indexed-array

Some implementations, like bonding, has nest array with same attr type. To support all kinds of entries under one nest array. As discussed[1], let's rename array-nest to indexed-array, and assuming the value is a nest by passing the type via sub-type.

[1] https://lore.kernel.org/netdev/[email protected]/

---
 Documentation/netlink/genetlink-c.yaml        |  2 +-
 Documentation/netlink/genetlink-legacy.yaml   |  2 +-
 Documentation/netlink/genetlink.yaml          |  2 +-
 Documentation/netlink/netlink-raw.yaml        |  2 +-
 Documentation/netlink/specs/nlctrl.yaml       |  6 ++++--
 Documentation/netlink/specs/rt_link.yaml      |  3 ++-
 Documentation/netlink/specs/tc.yaml           | 21 ++++++++++++-------
 .../netlink/genetlink-legacy.rst              | 12 ++++++++---
 tools/net/ynl/lib/ynl.py                      | 13 +++++++-----
 tools/net/ynl/ynl-gen-c.py                    | 18 ++++++++++------
 10 files changed, 53 insertions(+), 28 deletions(-)
Documentation/netlink/genetlink-c.yaml +1 -1
index 4dfd899a1661..4f803eaac6d8 100644
@@ -158,7 +158,7 @@ properties:
               type: &attr-type
                 enum: [ unused, pad, flag, binary,
                         uint, sint, u8, u16, u32, u64, s32, s64,
-                        string, nest, array-nest, nest-type-value ]
+                        string, nest, indexed-array, nest-type-value ]
               doc:
                 description: Documentation of the attribute.
                 type: string
Documentation/netlink/genetlink-legacy.yaml +1 -1
index b48ad3b1cc32..8db0e22fa72c 100644
@@ -201,7 +201,7 @@ properties:
                 description: The netlink attribute type
                 enum: [ unused, pad, flag, binary, bitfield32,
                         uint, sint, u8, u16, u32, u64, s32, s64,
-                        string, nest, array-nest, nest-type-value ]
+                        string, nest, indexed-array, nest-type-value ]
               doc:
                 description: Documentation of the attribute.
                 type: string
Documentation/netlink/genetlink.yaml +1 -1
index ebd6ee743fcc..b036227b46f1 100644
@@ -124,7 +124,7 @@ properties:
               type: &attr-type
                 enum: [ unused, pad, flag, binary,
                         uint, sint, u8, u16, u32, u64, s32, s64,
-                        string, nest, array-nest, nest-type-value ]
+                        string, nest, indexed-array, nest-type-value ]
               doc:
                 description: Documentation of the attribute.
                 type: string
Documentation/netlink/netlink-raw.yaml +1 -1
index a76e54cbadbc..914aa1c0a273 100644
@@ -222,7 +222,7 @@ properties:
                 description: The netlink attribute type
                 enum: [ unused, pad, flag, binary, bitfield32,
                         u8, u16, u32, u64, s8, s16, s32, s64,
-                        string, nest, array-nest, nest-type-value,
+                        string, nest, indexed-array, nest-type-value,
                         sub-message ]
               doc:
                 description: Documentation of the attribute.
Documentation/netlink/specs/nlctrl.yaml +4 -2
index b1632b95f725..a36535350bdb 100644
@@ -65,11 +65,13 @@ attribute-sets:
         type: u32
       -
         name: ops
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: op-attrs
       -
         name: mcast-groups
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: mcast-group-attrs
       -
         name: policy
Documentation/netlink/specs/rt_link.yaml +2 -1
index 81a5a3d1b04d..e5dcb2cf1724 100644
@@ -1690,7 +1690,8 @@ attribute-sets:
         type: binary
       -
         name: hw-s-info
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: hw-s-info-one
       -
         name: l3-stats
Documentation/netlink/specs/tc.yaml +14 -7
index 324fa182cd14..dbcf19e494ec 100644
@@ -1937,7 +1937,8 @@ attribute-sets:
         nested-attributes: tc-ematch-attrs
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
       -
         name: police
@@ -2077,7 +2078,8 @@ attribute-sets:
         type: u32
       -
         name: tin-stats
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-cake-tin-stats-attrs
       -
         name: deficit
@@ -2297,7 +2299,8 @@ attribute-sets:
         type: string
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
       -
         name: key-eth-dst
@@ -2798,7 +2801,8 @@ attribute-sets:
         type: string
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
       -
         name: mask
@@ -2951,7 +2955,8 @@ attribute-sets:
         type: u32
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
       -
         name: flags
@@ -3324,7 +3329,8 @@ attribute-sets:
         nested-attributes: tc-police-attrs
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
   -
     name: tc-taprio-attrs
@@ -3542,7 +3548,8 @@ attribute-sets:
         nested-attributes: tc-police-attrs
       -
         name: act
-        type: array-nest
+        type: indexed-array
+        sub-type: nest
         nested-attributes: tc-act-attrs
       -
         name: indev
Documentation/userspace-api/netlink/genetlink-legacy.rst +9 -3
index 70a77387f6c4..54e8fb25e093 100644
@@ -46,10 +46,16 @@ For reference the ``multi-attr`` array may look like this::
 
 where ``ARRAY-ATTR`` is the array entry type.
 
-array-nest
-~~~~~~~~~~
+indexed-array
+~~~~~~~~~~~~~
+
+``indexed-array`` wraps the entire array in an extra attribute (hence
+limiting its size to 64kB). The ``ENTRY`` nests are special and have the
+index of the entry as their type instead of normal attribute type.
 
-``array-nest`` creates the following structure::
+A ``sub-type`` is needed to describe what type in the ``ENTRY``. A ``nest``
+``sub-type`` means there are nest arrays in the ``ENTRY``, with the structure
+looks like::
 
   [SOME-OTHER-ATTR]
   [ARRAY-ATTR]
tools/net/ynl/lib/ynl.py +8 -5
index 82d3c98067aa..e5ad415905c7 100644
@@ -630,15 +630,18 @@ class YnlFamily(SpecFamily):
                 decoded = self._formatted_string(decoded, attr_spec.display_hint)
         return decoded
 
-    def _decode_array_nest(self, attr, attr_spec):
+    def _decode_array_attr(self, attr, attr_spec):
         decoded = []
         offset = 0
         while offset < len(attr.raw):
             item = NlAttr(attr.raw, offset)
             offset += item.full_len
 
-            subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
-            decoded.append({ item.type: subattrs })
+            if attr_spec["sub-type"] == 'nest':
+                subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
+                decoded.append({ item.type: subattrs })
+            else:
+                raise Exception(f'Unknown {attr_spec["sub-type"]} with name {attr_spec["name"]}')
         return decoded
 
     def _decode_nest_type_value(self, attr, attr_spec):
@@ -732,8 +735,8 @@ class YnlFamily(SpecFamily):
                 decoded = attr.as_scalar(attr_spec['type'], attr_spec.byte_order)
                 if 'enum' in attr_spec:
                     decoded = self._decode_enum(decoded, attr_spec)
-            elif attr_spec["type"] == 'array-nest':
-                decoded = self._decode_array_nest(attr, attr_spec)
+            elif attr_spec["type"] == 'indexed-array':
+                decoded = self._decode_array_attr(attr, attr_spec)
             elif attr_spec["type"] == 'bitfield32':
                 value, selector = struct.unpack("II", attr.raw)
                 if 'enum' in attr_spec:
tools/net/ynl/ynl-gen-c.py +12 -6
index a451cbfbd781..c0b90c104d92 100755
@@ -841,8 +841,11 @@ class AttrSet(SpecAttrSet):
             t = TypeBitfield32(self.family, self, elem, value)
         elif elem['type'] == 'nest':
             t = TypeNest(self.family, self, elem, value)
-        elif elem['type'] == 'array-nest':
-            t = TypeArrayNest(self.family, self, elem, value)
+        elif elem['type'] == 'indexed-array' and 'sub-type' in elem:
+            if elem["sub-type"] == 'nest':
+                t = TypeArrayNest(self.family, self, elem, value)
+            else:
+                raise Exception(f'new_attr: unsupported sub-type {elem["sub-type"]}')
         elif elem['type'] == 'nest-type-value':
             t = TypeNestTypeValue(self.family, self, elem, value)
         else:
@@ -1055,7 +1058,7 @@ class Family(SpecFamily):
                     if nested in self.root_sets:
                         raise Exception("Inheriting members to a space used as root not supported")
                     inherit.update(set(spec['type-value']))
-                elif spec['type'] == 'array-nest':
+                elif spec['type'] == 'indexed-array':
                     inherit.add('idx')
                 self.pure_nested_structs[nested].set_inherited(inherit)
 
@@ -1619,9 +1622,12 @@ def _multi_parse(ri, struct, init_lines, local_vars):
     multi_attrs = set()
     needs_parg = False
     for arg, aspec in struct.member_list():
-        if aspec['type'] == 'array-nest':
-            local_vars.append(f'const struct nlattr *attr_{aspec.c_name};')
-            array_nests.add(arg)
+        if aspec['type'] == 'indexed-array' and 'sub-type' in aspec:
+            if aspec["sub-type"] == 'nest':
+                local_vars.append(f'const struct nlattr *attr_{aspec.c_name};')
+                array_nests.add(arg)
+            else:
+                raise Exception(f'Not supported sub-type {aspec["sub-type"]}')
         if 'multi-attr' in aspec:
             multi_attrs.add(arg)
         needs_parg |= 'nested-attributes' in aspec
Signature · 1 line
2.43.0
#2 lore raw

PATCH v3 2/2 ynl: support binary/u32 sub-type for indexed-array

In reply to Hangbin Liu

Add binary/u32 sub-type support for indexed-array to display bond arp and ns targets. Here is what the result looks like:

 # ip link add bond0 type bond mode 1 \
   arp_ip_target 192.168.1.1,192.168.1.2 ns_ip6_target 2001::1,2001::2
 # ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/rt_link.yaml \
   --do getlink --json '{"ifname": "bond0"}' --output-json | jq '.linkinfo'
    "arp-ip-target": [
      "192.168.1.1",
      "192.168.1.2"
    ],
    [...]
    "ns-ip6-target": [
      "2001::1",
      "2001::2"
    ],
---
 .../userspace-api/netlink/genetlink-legacy.rst       | 12 +++++++++---
 tools/net/ynl/lib/ynl.py                             |  5 +++++
 2 files changed, 14 insertions(+), 3 deletions(-)
Documentation/userspace-api/netlink/genetlink-legacy.rst +9 -3
index 54e8fb25e093..6525ef6ca62f 100644
@@ -66,9 +66,15 @@ looks like::
       [MEMBER1]
       [MEMBER2]
 
-It wraps the entire array in an extra attribute (hence limiting its size
-to 64kB). The ``ENTRY`` nests are special and have the index of the entry
-as their type instead of normal attribute type.
+Other ``sub-type`` like ``u32`` means there is only one member as described
+in ``sub-type`` in the ``ENTRY``. The structure looks like::
+
+  [SOME-OTHER-ATTR]
+  [ARRAY-ATTR]
+    [ENTRY]
+      [MEMBER1]
+    [ENTRY]
+      [MEMBER1]
 
 type-value
 ~~~~~~~~~~
tools/net/ynl/lib/ynl.py +5 -0
index e5ad415905c7..aa7077cffe74 100644
@@ -640,6 +640,11 @@ class YnlFamily(SpecFamily):
             if attr_spec["sub-type"] == 'nest':
                 subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
                 decoded.append({ item.type: subattrs })
+            elif attr_spec["sub-type"] == 'binary' or attr_spec["sub-type"] == 'u32':
+                subattrs = item.as_bin()
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)
             else:
                 raise Exception(f'Unknown {attr_spec["sub-type"]} with name {attr_spec["name"]}')
         return decoded
Signature · 1 line
2.43.0
maintainer
#3 lore raw

Re: [PATCHv3 net-next 2/2] ynl: support binary/u32 sub-type for indexed-array

Documentation/userspace-api/netlink/genetlink-legacy.rst looks like:: · line 77

Quoted 9 lines · On Mon, 1 Apr 2024 11:56:51 +0800 Hangbin Liu wrote:
Documentation/userspace-api/netlink/genetlink-legacy.rst +9 -0
+Other ``sub-type`` like ``u32`` means there is only one member as described
+in ``sub-type`` in the ``ENTRY``. The structure looks like::
+
+  [SOME-OTHER-ATTR]
+  [ARRAY-ATTR]
+    [ENTRY]
+      [MEMBER1]
+    [ENTRY]
+      [MEMBER1]

I think that elsewhere in the doc we use [SOMETHING] to mean TLV of type SOMETHING, here MEMBER1/2 are presumably just payloads of each ENTRY? Maybe this is better:

[SOME-OTHER-ATTR]
[ARRAY-ATTR]

    [ENTRY u32]
    [ENTRY u32]

?

tools/net/ynl/lib/ynl.py class YnlFamily(SpecFamily): · line 644

Quoted 12 lines

type-value
~~~~~~~~~~

tools/net/ynl/lib/ynl.py +2 -0
index e5ad415905c7..aa7077cffe74 100644
@@ -640,6 +640,11 @@ class YnlFamily(SpecFamily):
             if attr_spec["sub-type"] == 'nest':
                 subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
                 decoded.append({ item.type: subattrs })
+            elif attr_spec["sub-type"] == 'binary' or attr_spec["sub-type"] == 'u32':
+                subattrs = item.as_bin()

Are you sure that as_bin() will work for all u32s? Or just when there's a hint...

Quoted 3 lines
tools/net/ynl/lib/ynl.py +3 -0
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)
#4 lore raw

Re: [PATCHv3 net-next 2/2] ynl: support binary/u32 sub-type for indexed-array

Quoted 9 lines · Jakub Kicinski

I think that elsewhere in the doc we use [SOMETHING] to mean TLV of type SOMETHING, here MEMBER1/2 are presumably just payloads of each ENTRY? Maybe this is better:

[SOME-OTHER-ATTR]
[ARRAY-ATTR]

    [ENTRY u32]
    [ENTRY u32]

Thanks, I will update the doc.

Quoted 17 lines

?

tools/net/ynl/lib/ynl.py class YnlFamily(SpecFamily): · line 644

Quoted 12 lines

type-value
~~~~~~~~~~

tools/net/ynl/lib/ynl.py +2 -0
index e5ad415905c7..aa7077cffe74 100644
@@ -640,6 +640,11 @@ class YnlFamily(SpecFamily):
             if attr_spec["sub-type"] == 'nest':
                 subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
                 decoded.append({ item.type: subattrs })
+            elif attr_spec["sub-type"] == 'binary' or attr_spec["sub-type"] == 'u32':
+                subattrs = item.as_bin()

Are you sure that as_bin() will work for all u32s? Or just when there's a hint...

I didn't check other subsystem. For bonding only, if we don't have the hint. e.g.

-

    name: arp-ip-target
    type: indexed-array
    sub-type: u32

The result will looks like:

    "arp-ip-target": [
      "c0a80101",
      "c0a80102"
    ],

Which looks good to me. Do you have other suggestion?

Thanks
Hangbin

maintainer
#5 lore raw

Re: [PATCHv3 net-next 2/2] ynl: support binary/u32 sub-type for indexed-array

Quoted 8 lines · On Wed, 3 Apr 2024 10:28:34 +0800 Hangbin Liu wrote:

I didn't check other subsystem. For bonding only, if we don't have the hint. e.g.

-

    name: arp-ip-target
    type: indexed-array
    sub-type: u32
Quoted 8 lines

The result will looks like:

    "arp-ip-target": [
      "c0a80101",
      "c0a80102"
    ],

Which looks good to me. Do you have other suggestion?

That doesn't look right, without the format hint if the type is u32 the members should be plain integers not hex strings.

#6 lore raw

Re: [PATCHv3 net-next 2/2] ynl: support binary/u32 sub-type for indexed-array

Quoted 16 lines · Jakub Kicinski
Quoted 15 lines · On Wed, 3 Apr 2024 10:28:34 +0800 Hangbin Liu wrote:

I didn't check other subsystem. For bonding only, if we don't have the hint. e.g.

-

    name: arp-ip-target
    type: indexed-array
    sub-type: u32

The result will looks like:

    "arp-ip-target": [
      "c0a80101",
      "c0a80102"
    ],
Quoted 4 lines
Quoted 1 line

Which looks good to me. Do you have other suggestion?

That doesn't look right, without the format hint if the type is u32 the members should be plain integers not hex strings.

OK, I can separate the binary and u32 dealing. How about like

tools/net/ynl/lib/ynl.py +10 -0
index e5ad415905c7..be42e4fc1037 100644
@@ -640,6 +640,16 @@ class YnlFamily(SpecFamily):
             if attr_spec["sub-type"] == 'nest':
                 subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
                 decoded.append({ item.type: subattrs })
+            elif attr_spec["sub-type"] == 'binary':
+                subattrs = item.as_bin()
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)
+            elif attr_spec["sub-type"] in NlAttr.type_formats:
+                subattrs = item.as_scalar(attr_spec['sub-type'], attr_spec.byte_order)
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)
             else:
                 raise Exception(f'Unknown {attr_spec["sub-type"]} with name {attr_spec["name"]}')
         return decoded

With only sub-type: u32 it shows like

    "arp-ip-target": [
      3232235777,
      3232235778
    ],

Thanks
Hangbin

maintainer
#7 lore raw

Re: [PATCHv3 net-next 2/2] ynl: support binary/u32 sub-type for indexed-array

Quoted 10 lines · On Wed, 3 Apr 2024 11:27:22 +0800 Hangbin Liu wrote:

OK, I can separate the binary and u32 dealing. How about like

tools/net/ynl/lib/ynl.py +0 -0
index e5ad415905c7..be42e4fc1037 100644
@@ -640,6 +640,16 @@ class YnlFamily(SpecFamily):
             if attr_spec["sub-type"] == 'nest':
                 subattrs = self._decode(NlAttrs(item.raw), attr_spec['nested-attributes'])
                 decoded.append({ item.type: subattrs })

tools/net/ynl/lib/ynl.py class YnlFamily(SpecFamily): · line 652

Quoted 10 lines
tools/net/ynl/lib/ynl.py +10 -0
+            elif attr_spec["sub-type"] == 'binary':
+                subattrs = item.as_bin()
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)
+            elif attr_spec["sub-type"] in NlAttr.type_formats:
+                subattrs = item.as_scalar(attr_spec['sub-type'], attr_spec.byte_order)
+                if attr_spec.display_hint:
+                    subattrs = self._formatted_string(subattrs, attr_spec.display_hint)
+                decoded.append(subattrs)

Aha, this look right!