Skip to contentSkip to conversation

tools: ynl: handful of forward looking updates

PATCH Applied Jakub Kicinski 3 participants 12 messages Aug 24, 2023 – Aug 25, 2023

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

cover tools: ynl: handful of forward looking updates

Small YNL improvements, mostly for work-in-progress families.

Jakub Kicinski (5):
tools: ynl: allow passing binary data
tools: ynl-gen: set length of binary fields
tools: ynl-gen: fix collecting global policy attrs
tools: ynl-gen: support empty attribute lists
netlink: specs: fix indent in fou

 Documentation/netlink/specs/fou.yaml   | 18 +++++++++---------
 tools/net/ynl/generated/ethtool-user.h |  4 ++++
 tools/net/ynl/generated/fou-user.h     |  6 ++++++
 tools/net/ynl/lib/ynl.py               |  7 ++++++-
 tools/net/ynl/ynl-gen-c.py             | 16 +++++++++++++---
 5 files changed, 38 insertions(+), 13 deletions(-)
Signature · 1 line
2.41.0
author maintainer
#1 lore raw

PATCH 1/5 tools: ynl: allow passing binary data

Recent changes made us assume that input for binary data is in hex. When using YNL as a Python library it's possible to pass in raw bytes. Bring the ability to do that back.

---
 tools/net/ynl/lib/ynl.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/net/ynl/lib/ynl.py +6 -1
index 6951bcc7efdc..fa4f1c28efc5 100644
@@ -410,7 +410,12 @@ genl_family_name_to_id = None
         elif attr["type"] == 'string':
             attr_payload = str(value).encode('ascii') + b'\x00'
         elif attr["type"] == 'binary':
-            attr_payload = bytes.fromhex(value)
+            if isinstance(value, bytes):
+                attr_payload = value
+            elif isinstance(value, str):
+                attr_payload = bytes.fromhex(value)
+            else:
+                raise Exception(f'Unknown type for binary attribute, value: {value}')
         elif attr['type'] in NlAttr.type_formats:
             format = NlAttr.get_format(attr['type'], attr.byte_order)
             attr_payload = format.pack(int(value))
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH net-next 1/5] tools: ynl: allow passing binary data

Quoted 17 lines · Jakub Kicinski

Recent changes made us assume that input for binary data is in hex. When using YNL as a Python library it's possible to pass in raw bytes. Bring the ability to do that back.

---
 tools/net/ynl/lib/ynl.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/net/ynl/lib/ynl.py +0 -0
index 6951bcc7efdc..fa4f1c28efc5 100644
@@ -410,7 +410,12 @@ genl_family_name_to_id = None
         elif attr["type"] == 'string':
             attr_payload = str(value).encode('ascii') + b'\x00'
         elif attr["type"] == 'binary':

tools/net/ynl/lib/ynl.py genl_family_name_to_id = None · line 421

Quoted 10 lines
tools/net/ynl/lib/ynl.py +6 -1
-            attr_payload = bytes.fromhex(value)
+            if isinstance(value, bytes):
+                attr_payload = value
+            elif isinstance(value, str):
+                attr_payload = bytes.fromhex(value)
+            else:
+                raise Exception(f'Unknown type for binary attribute, value: {value}')
         elif attr['type'] in NlAttr.type_formats:
             format = NlAttr.get_format(attr['type'], attr.byte_order)
             attr_payload = format.pack(int(value))
author maintainer
#3 lore raw

PATCH 2/5 tools: ynl-gen: set length of binary fields

In reply to Jakub Kicinski

Remember to set the length field in the request setters.

---
 tools/net/ynl/generated/ethtool-user.h | 4 ++++
 tools/net/ynl/generated/fou-user.h     | 6 ++++++
 tools/net/ynl/ynl-gen-c.py             | 1 +
 3 files changed, 11 insertions(+)
tools/net/ynl/generated/ethtool-user.h +4 -0
index d7d4ba855f43..ddc1a5209992 100644
@@ -1422,6 +1422,7 @@ ethtool_wol_set_req_set_sopass(struct ethtool_wol_set_req *req,
 			       const void *sopass, size_t len)
 {
 	free(req->sopass);
+	req->_present.sopass_len = len;
 	req->sopass = malloc(req->_present.sopass_len);
 	memcpy(req->sopass, sopass, req->_present.sopass_len);
 }
@@ -4071,6 +4072,7 @@ ethtool_fec_set_req_set_stats_corrected(struct ethtool_fec_set_req *req,
 					const void *corrected, size_t len)
 {
 	free(req->stats.corrected);
+	req->stats._present.corrected_len = len;
 	req->stats.corrected = malloc(req->stats._present.corrected_len);
 	memcpy(req->stats.corrected, corrected, req->stats._present.corrected_len);
 }
@@ -4079,6 +4081,7 @@ ethtool_fec_set_req_set_stats_uncorr(struct ethtool_fec_set_req *req,
 				     const void *uncorr, size_t len)
 {
 	free(req->stats.uncorr);
+	req->stats._present.uncorr_len = len;
 	req->stats.uncorr = malloc(req->stats._present.uncorr_len);
 	memcpy(req->stats.uncorr, uncorr, req->stats._present.uncorr_len);
 }
@@ -4087,6 +4090,7 @@ ethtool_fec_set_req_set_stats_corr_bits(struct ethtool_fec_set_req *req,
 					const void *corr_bits, size_t len)
 {
 	free(req->stats.corr_bits);
+	req->stats._present.corr_bits_len = len;
 	req->stats.corr_bits = malloc(req->stats._present.corr_bits_len);
 	memcpy(req->stats.corr_bits, corr_bits, req->stats._present.corr_bits_len);
 }
tools/net/ynl/generated/fou-user.h +6 -0
index d8ab50579cd1..a8f860892540 100644
@@ -91,6 +91,7 @@ fou_add_req_set_local_v6(struct fou_add_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -99,6 +100,7 @@ fou_add_req_set_peer_v6(struct fou_add_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
@@ -192,6 +194,7 @@ fou_del_req_set_local_v6(struct fou_del_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -200,6 +203,7 @@ fou_del_req_set_peer_v6(struct fou_del_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
@@ -280,6 +284,7 @@ fou_get_req_set_local_v6(struct fou_get_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -288,6 +293,7 @@ fou_get_req_set_peer_v6(struct fou_get_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
tools/net/ynl/ynl-gen-c.py +1 -0
index bdff8dfc29c9..e27deb199a70 100755
@@ -428,6 +428,7 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
     def _setter_lines(self, ri, member, presence):
         return [f"free({member});",
+                f"{presence}_len = len;",
                 f"{member} = malloc({presence}_len);",
                 f'memcpy({member}, {self.c_name}, {presence}_len);']
 
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 2/5
#4 lore raw

Re: [PATCH net-next 2/5] tools: ynl-gen: set length of binary fields

Quoted 102 lines · Jakub Kicinski

Remember to set the length field in the request setters.

---
 tools/net/ynl/generated/ethtool-user.h | 4 ++++
 tools/net/ynl/generated/fou-user.h     | 6 ++++++
 tools/net/ynl/ynl-gen-c.py             | 1 +
 3 files changed, 11 insertions(+)
tools/net/ynl/generated/ethtool-user.h +4 -0
index d7d4ba855f43..ddc1a5209992 100644
@@ -1422,6 +1422,7 @@ ethtool_wol_set_req_set_sopass(struct ethtool_wol_set_req *req,
 			       const void *sopass, size_t len)
 {
 	free(req->sopass);
+	req->_present.sopass_len = len;
 	req->sopass = malloc(req->_present.sopass_len);
 	memcpy(req->sopass, sopass, req->_present.sopass_len);
 }
@@ -4071,6 +4072,7 @@ ethtool_fec_set_req_set_stats_corrected(struct ethtool_fec_set_req *req,
 					const void *corrected, size_t len)
 {
 	free(req->stats.corrected);
+	req->stats._present.corrected_len = len;
 	req->stats.corrected = malloc(req->stats._present.corrected_len);
 	memcpy(req->stats.corrected, corrected, req->stats._present.corrected_len);
 }
@@ -4079,6 +4081,7 @@ ethtool_fec_set_req_set_stats_uncorr(struct ethtool_fec_set_req *req,
 				     const void *uncorr, size_t len)
 {
 	free(req->stats.uncorr);
+	req->stats._present.uncorr_len = len;
 	req->stats.uncorr = malloc(req->stats._present.uncorr_len);
 	memcpy(req->stats.uncorr, uncorr, req->stats._present.uncorr_len);
 }
@@ -4087,6 +4090,7 @@ ethtool_fec_set_req_set_stats_corr_bits(struct ethtool_fec_set_req *req,
 					const void *corr_bits, size_t len)
 {
 	free(req->stats.corr_bits);
+	req->stats._present.corr_bits_len = len;
 	req->stats.corr_bits = malloc(req->stats._present.corr_bits_len);
 	memcpy(req->stats.corr_bits, corr_bits, req->stats._present.corr_bits_len);
 }
tools/net/ynl/generated/fou-user.h +6 -0
index d8ab50579cd1..a8f860892540 100644
@@ -91,6 +91,7 @@ fou_add_req_set_local_v6(struct fou_add_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -99,6 +100,7 @@ fou_add_req_set_peer_v6(struct fou_add_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
@@ -192,6 +194,7 @@ fou_del_req_set_local_v6(struct fou_del_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -200,6 +203,7 @@ fou_del_req_set_peer_v6(struct fou_del_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
@@ -280,6 +284,7 @@ fou_get_req_set_local_v6(struct fou_get_req *req, const void *local_v6,
 			 size_t len)
 {
 	free(req->local_v6);
+	req->_present.local_v6_len = len;
 	req->local_v6 = malloc(req->_present.local_v6_len);
 	memcpy(req->local_v6, local_v6, req->_present.local_v6_len);
 }
@@ -288,6 +293,7 @@ fou_get_req_set_peer_v6(struct fou_get_req *req, const void *peer_v6,
 			size_t len)
 {
 	free(req->peer_v6);
+	req->_present.peer_v6_len = len;
 	req->peer_v6 = malloc(req->_present.peer_v6_len);
 	memcpy(req->peer_v6, peer_v6, req->_present.peer_v6_len);
 }
tools/net/ynl/ynl-gen-c.py +0 -0
index bdff8dfc29c9..e27deb199a70 100755

tools/net/ynl/ynl-gen-c.py from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S · line 433

Quoted 6 lines
tools/net/ynl/ynl-gen-c.py +1 -0
@@ -428,6 +428,7 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
     def _setter_lines(self, ri, member, presence):
         return [f"free({member});",
+                f"{presence}_len = len;",
                 f"{member} = malloc({presence}_len);",
                 f'memcpy({member}, {self.c_name}, {presence}_len);']
author maintainer
#5 lore raw

PATCH 3/5 tools: ynl-gen: fix collecting global policy attrs

In reply to Jakub Kicinski

We look for attributes inside do.request, but there's another layer of nesting in the spec, look inside do.request.attributes.

This bug had no effect as all global policies we generate (fou) seem to be full, anyway, and we treat full and empty the same.

Next patch will change the treatment of empty policies.

---
 tools/net/ynl/ynl-gen-c.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/net/ynl/ynl-gen-c.py +3 -1
index e27deb199a70..13d06931c045 100755
@@ -978,7 +978,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
             for op_mode in ['do', 'dump']:
                 if op_mode in op:
-                    global_set.update(op[op_mode].get('request', []))
+                    req = op[op_mode].get('request')
+                    if req:
+                        global_set.update(req.get('attributes', []))
 
         self.global_policy = []
         self.global_policy_set = attr_set_name
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 3/5
#6 lore raw

Re: [PATCH net-next 3/5] tools: ynl-gen: fix collecting global policy attrs

Quoted 18 lines · Jakub Kicinski

We look for attributes inside do.request, but there's another layer of nesting in the spec, look inside do.request.attributes.

This bug had no effect as all global policies we generate (fou) seem to be full, anyway, and we treat full and empty the same.

Next patch will change the treatment of empty policies.

---
 tools/net/ynl/ynl-gen-c.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/net/ynl/ynl-gen-c.py +0 -0
index e27deb199a70..13d06931c045 100755

tools/net/ynl/ynl-gen-c.py from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S · line 986

Quoted 10 lines
tools/net/ynl/ynl-gen-c.py +3 -1
@@ -978,7 +978,9 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
             for op_mode in ['do', 'dump']:
                 if op_mode in op:
-                    global_set.update(op[op_mode].get('request', []))
+                    req = op[op_mode].get('request')
+                    if req:
+                        global_set.update(req.get('attributes', []))
 
         self.global_policy = []
         self.global_policy_set = attr_set_name
author maintainer
#7 lore raw

PATCH 4/5 tools: ynl-gen: support empty attribute lists

In reply to Jakub Kicinski

Differentiate between empty list and None for member lists. New families may want to create request responses with no attribute. If we treat those the same as None we end up rendering
a full parsing policy in user space, instead of an empty one.

---
 tools/net/ynl/ynl-gen-c.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
tools/net/ynl/ynl-gen-c.py +9 -2
index 13d06931c045..9209bdcca9c6 100755
@@ -615,7 +615,7 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
         self.attr_list = []
         self.attrs = dict()
-        if type_list:
+        if type_list is not None:
             for t in type_list:
                 self.attr_list.append((t, self.attr_set[t]),)
         else:
@@ -1543,7 +1543,14 @@ _C_KW = {
 
     ri.cw.write_func_prot('int', f'{op_prefix(ri, "reply", deref=deref)}_parse', func_args)
 
-    _multi_parse(ri, ri.struct["reply"], init_lines, local_vars)
+    if ri.struct["reply"].member_list():
+        _multi_parse(ri, ri.struct["reply"], init_lines, local_vars)
+    else:
+        # Empty reply
+        ri.cw.block_start()
+        ri.cw.p('return MNL_CB_OK;')
+        ri.cw.block_end()
+        ri.cw.nl()
 
 
 def print_req(ri):
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 4/5
#8 lore raw

Re: [PATCH net-next 4/5] tools: ynl-gen: support empty attribute lists

Quoted 29 lines · Jakub Kicinski

Differentiate between empty list and None for member lists. New families may want to create request responses with no attribute. If we treat those the same as None we end up rendering
a full parsing policy in user space, instead of an empty one.

---
 tools/net/ynl/ynl-gen-c.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
tools/net/ynl/ynl-gen-c.py +2 -2
index 13d06931c045..9209bdcca9c6 100755
@@ -615,7 +615,7 @@ from lib import SpecFamily, SpecAttrSet, SpecAttr, SpecOperation, SpecEnumSet, S
 
         self.attr_list = []
         self.attrs = dict()
-        if type_list:
+        if type_list is not None:
             for t in type_list:
                 self.attr_list.append((t, self.attr_set[t]),)
         else:
@@ -1543,7 +1543,14 @@ _C_KW = {
 
     ri.cw.write_func_prot('int', f'{op_prefix(ri, "reply", deref=deref)}_parse', func_args)
 
-    _multi_parse(ri, ri.struct["reply"], init_lines, local_vars)
+    if ri.struct["reply"].member_list():

tools/net/ynl/ynl-gen-c.py _C_KW = { · line 1556

Quoted 10 lines
tools/net/ynl/ynl-gen-c.py +7 -0
+        _multi_parse(ri, ri.struct["reply"], init_lines, local_vars)
+    else:
+        # Empty reply
+        ri.cw.block_start()
+        ri.cw.p('return MNL_CB_OK;')
+        ri.cw.block_end()
+        ri.cw.nl()
 
 
 def print_req(ri):
author maintainer
#9 lore raw

PATCH 5/5 netlink: specs: fix indent in fou

In reply to Jakub Kicinski

Fix up the indentation. This has no functional effect, AFAICT.

---
 Documentation/netlink/specs/fou.yaml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
Documentation/netlink/specs/fou.yaml +9 -9
index 3e13826a3fdf..0af5ab842c04 100644
@@ -107,16 +107,16 @@ kernel-policy: global
       flags: [ admin-perm ]
 
       do:
-        request:  &select_attrs
+        request: &select_attrs
           attributes:
-          - af
-          - ifindex
-          - port
-          - peer_port
-          - local_v4
-          - peer_v4
-          - local_v6
-          - peer_v6
+            - af
+            - ifindex
+            - port
+            - peer_port
+            - local_v4
+            - peer_v4
+            - local_v6
+            - peer_v6
 
     -
       name: get
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 5/5
#10 lore raw

Re: [PATCH net-next 5/5] netlink: specs: fix indent in fou

Quoted 27 lines · Jakub Kicinski

Fix up the indentation. This has no functional effect, AFAICT.

---
 Documentation/netlink/specs/fou.yaml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
Documentation/netlink/specs/fou.yaml +2 -9
index 3e13826a3fdf..0af5ab842c04 100644
@@ -107,16 +107,16 @@ kernel-policy: global
       flags: [ admin-perm ]
 
       do:
-        request:  &select_attrs
+        request: &select_attrs
           attributes:
-          - af
-          - ifindex
-          - port
-          - peer_port
-          - local_v4
-          - peer_v4
-          - local_v6
-          - peer_v6
+            - af

Documentation/netlink/specs/fou.yaml kernel-policy: global · line 122

Quoted 10 lines
Documentation/netlink/specs/fou.yaml +7 -0
+            - ifindex
+            - port
+            - peer_port
+            - local_v4
+            - peer_v4
+            - local_v6
+            - peer_v6
 
     -
       name: get
#11 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Wed, 23 Aug 2023 17:30:51 -0700 you wrote:

Small YNL improvements, mostly for work-in-progress families.

Jakub Kicinski (5):
tools: ynl: allow passing binary data
tools: ynl-gen: set length of binary fields
tools: ynl-gen: fix collecting global policy attrs
tools: ynl-gen: support empty attribute lists
netlink: specs: fix indent in fou

[...]

Here is the summary with links:
- [net-next,1/5] tools: ynl: allow passing binary data

    https://git.kernel.org/netdev/net-next/c/649bde9004ac
  - [net-next,2/5] tools: ynl-gen: set length of binary fields
    https://git.kernel.org/netdev/net-next/c/a149a3a13bbc
  - [net-next,3/5] tools: ynl-gen: fix collecting global policy attrs
    https://git.kernel.org/netdev/net-next/c/dc2ef94d8926
  - [net-next,4/5] tools: ynl-gen: support empty attribute lists
    https://git.kernel.org/netdev/net-next/c/4c8c24e801e6
  - [net-next,5/5] netlink: specs: fix indent in fou
    https://git.kernel.org/netdev/net-next/c/e83d4e9b2d0f

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html