net: introduce TX shaping H/W offload API
Subsystems (5)
Paolo Abeni
maintainercover net: introduce TX shaping H/W offload API
We have a plurality of shaping-related drivers API, but none flexible enough to meet existing demand from vendors[1].
This series introduces new device APIs to configure in a flexible way TX shaping H/W offload. The new functionalities are exposed via a newly defined generic netlink interface and include introspection capabilities. Some basic self-tests are included, on top of a dummy netdevsim implementation.
The ice driver support is currently a WIP, sharing the current status earlier since some APIs details are still under discussion.
[1] https://lore.kernel.org/netdev/[email protected]/
Paolo Abeni (5):
netlink: spec: add shaper YAML spec
net: introduce HW Rate limiting driver API
netlink: spec: add shaper introspection support
net: shaper: implement introspection support
testing: net-drv: add basic shaper test
Documentation/netlink/specs/shaper.yaml | 276 +++++++ drivers/net/Kconfig | 1 + drivers/net/netdevsim/netdev.c | 29 + include/linux/netdevice.h | 16 + include/net/net_shaper.h | 208 ++++++ include/uapi/linux/net_shaper.h | 90 +++ net/Kconfig | 3 + net/Makefile | 1 + net/core/dev.c | 2 + net/core/dev.h | 6 + net/shaper/Makefile | 9 + net/shaper/shaper.c | 686 ++++++++++++++++++ net/shaper/shaper_nl_gen.c | 118 +++ net/shaper/shaper_nl_gen.h | 28 + tools/testing/selftests/drivers/net/Makefile | 1 + tools/testing/selftests/drivers/net/shaper.py | 198 +++++ .../testing/selftests/net/lib/py/__init__.py | 1 + tools/testing/selftests/net/lib/py/ynl.py | 5 + 18 files changed, 1678 insertions(+) create mode 100644 Documentation/netlink/specs/shaper.yaml create mode 100644 include/net/net_shaper.h create mode 100644 include/uapi/linux/net_shaper.h create mode 100644 net/shaper/Makefile create mode 100644 net/shaper/shaper.c create mode 100644 net/shaper/shaper_nl_gen.c create mode 100644 net/shaper/shaper_nl_gen.h create mode 100755 tools/testing/selftests/drivers/net/shaper.py
Signature · 1 line
2.45.1
Paolo Abeni
maintainerPATCH 1/5 netlink: spec: add shaper YAML spec
Define the user-space visible interface to query, configure and delete network shapers via yaml definition.
Add dummy implementations for the relevant NL callbacks.
set() and delete() operations allows touching multiple shapers with a single operation, atomically.
---
v4 -> v5:
- fixed a few typos
- set() and get() ops reply with the number of affected handles
- re-ordered bps and pps
- added 'unspec' scope
v3 -> v4:
- dropped 'major'
- renamed 'minor' as 'id'
- rename 'bw_{max,min} as 'bw-{max,min}'
v2 -> v3:
- dropped 'move' op, use parent in 'set' instead
- expand 'handle' in 'scope', 'major', 'minor'
- rename 'queue_group' scope to 'detached'
- rename 'info' attr to 'shapers'
- added pad attribute (for 64 bits' sake)
v1 -> v2:
- reset -> delete
- added 'parent' and 'burst'
--- Documentation/netlink/specs/shaper.yaml | 202 ++++++++++++++++++++++++ include/uapi/linux/net_shaper.h | 73 +++++++++ net/Kconfig | 3 + net/Makefile | 1 + net/shaper/Makefile | 9 ++ net/shaper/shaper.c | 34 ++++ net/shaper/shaper_nl_gen.c | 93 +++++++++++ net/shaper/shaper_nl_gen.h | 25 +++ 8 files changed, 440 insertions(+) create mode 100644 Documentation/netlink/specs/shaper.yaml create mode 100644 include/uapi/linux/net_shaper.h create mode 100644 net/shaper/Makefile create mode 100644 net/shaper/shaper.c create mode 100644 net/shaper/shaper_nl_gen.c create mode 100644 net/shaper/shaper_nl_gen.h
Documentation/netlink/specs/shaper.yaml new file +202 -0
@@ -0,0 +1,202 @@ +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) + +name: net_shaper + +doc: Network HW Rate Limiting offload + +definitions: + - + type: enum + name: scope + doc: the different scopes where a shaper can be attached + entries: + - name: unspec + doc: The scope is not specified + - + name: port + doc: The root shaper for the whole H/W. + - + name: netdev + doc: The main shaper for the given network device. + - + name: queue + doc: The shaper is attached to the given device queue. + - + name: detached + doc: | + The shaper can be attached to port, netdev or other + detached shapers, allowing nesting and grouping of + netdev or queues. + render-max: true + - + type: enum + name: metric + doc: different metric each shaper can support + entries: + - + name: bps + doc: Shaper operates on a bits per second basis + - + name: pps + doc: Shaper operates on a packets per second basis + +attribute-sets: + - + name: net_shaper + attributes: + - + name: ifindex + type: u32 + - + name: parent + type: nest + nested-attributes: handle + - + name: handle + type: nest + nested-attributes: handle + - + name: metric + type: u32 + enum: metric + - + name: bw-min + type: u64 + - + name: bw-max + type: u64 + - + name: burst + type: u64 + - + name: priority + type: u32 + - + name: weight + type: u32 + - + name: scope + type: u32 + enum: scope + - + name: id + type: u32 + - + name: handles + type: nest + multi-attr: true + nested-attributes: handle + - + name: shapers + type: nest + multi-attr: true + nested-attributes: ns-info + - + name: modified + type: u32 + - + name: pad + type: pad + - + name: handle + subset-of: net_shaper + attributes: + - + name: scope + - + name: id + - + name: ns-info + subset-of: net_shaper + attributes: + - + name: parent + - + name: handle + - + name: metric + - + name: bw-min + - + name: bw-max + - + name: burst + - + name: priority + - + name: weight + +operations: + list: + - + name: get + doc: | + Get / Dump information about a/all the shaper for a given device + attribute-set: net_shaper + flags: [ admin-perm ] + + do: + request: + attributes: + - ifindex + - handle + reply: + attributes: &ns-attrs + - parent + - handle + - metric + - bw-min + - bw-max + - burst + - priority + - weight + + dump: + request: + attributes: + - ifindex + reply: + attributes: *ns-attrs + - + name: set + doc: | + Create or configures the specified shapers. + The update is atomic with respect to all shaper + affected by a single command, and is allowed to + affect a subset of the specified shapers, e.g. + due to H/W resources exhaustion. In such case + the update stops at the first failure, the extack + is set accordingly. + attribute-set: net_shaper + flags: [ admin-perm ] + + do: + request: + attributes: + - ifindex + - shapers + reply: + attributes: + - modified + + - + name: delete + doc: | + Clear (remove) the specified shaper. + The update is atomic with respect to all shaper + affected by a single command, and is allowed to + affect a subset of the specified shapers, e.g. + due to H/W resources exhaustion. In such case + the update stops at the first failure, the extack + is set accordingly. + attribute-set: net_shaper + flags: [ admin-perm ] + + do: + request: + attributes: + - ifindex + - handles + reply: + attributes: + - modified
include/uapi/linux/net_shaper.h new file +73 -0
73 lines not on this page · raw message
net/Kconfig +3 -0
@@ -66,6 +66,9 @@ config SKB_DECRYPTED config SKB_EXTENSIONS bool +config NET_SHAPER + bool + menu "Networking options" source "net/packet/Kconfig"
net/Makefile +1 -0
@@ -79,3 +79,4 @@ obj-$(CONFIG_XDP_SOCKETS) += xdp/ obj-$(CONFIG_MPTCP) += mptcp/ obj-$(CONFIG_MCTP) += mctp/ obj-$(CONFIG_NET_HANDSHAKE) += handshake/ +obj-$(CONFIG_NET_SHAPER) += shaper/
net/shaper/Makefile new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Makefile for the Generic HANDSHAKE service +# +# Copyright (c) 2024, Red Hat, Inc. +# + +obj-y += shaper.o shaper_nl_gen.o +
net/shaper/shaper.c new file +34 -0
@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include <linux/kernel.h> +#include <linux/skbuff.h> + +#include "shaper_nl_gen.h" + +int net_shaper_nl_get_doit (struct sk_buff *skb,struct genl_info *info) +{ +return -EOPNOTSUPP ; +} + +int net_shaper_nl_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb) +{ +return -EOPNOTSUPP ; +} + +int net_shaper_nl_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +return -EOPNOTSUPP ; +} + +int net_shaper_nl_delete_doit (struct sk_buff *skb,struct genl_info *info) +{ +return -EOPNOTSUPP ; +} + +static int __init shaper_init (void ) +{ +return genl_register_family (&net_shaper_nl_family); +} + +subsys_initcall (shaper_init);
net/shaper/shaper_nl_gen.c new file +93 -0
93 lines not on this page · raw message
net/shaper/shaper_nl_gen.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/shaper.yaml */ +/* YNL-GEN kernel header */ + +#ifndef _LINUX_NET_SHAPER_GEN_H +#define _LINUX_NET_SHAPER_GEN_H + +#include <net/netlink.h> +#include <net/genetlink.h> + +#include <uapi/linux/net_shaper.h> + +/* Common nested types */ +extern const struct nla_policy net_shaper_handle_nl_policy[NET_SHAPER_A_ID +1 ]; +extern const struct nla_policy net_shaper_ns_info_nl_policy[NET_SHAPER_A_WEIGHT +1 ]; + +int net_shaper_nl_get_doit (struct sk_buff *skb,struct genl_info *info); +int net_shaper_nl_get_dumpit (struct sk_buff *skb,struct netlink_callback *cb); +int net_shaper_nl_set_doit (struct sk_buff *skb,struct genl_info *info); +int net_shaper_nl_delete_doit (struct sk_buff *skb,struct genl_info *info); + +extern struct genl_family net_shaper_nl_family; + +#endif /* _LINUX_NET_SHAPER_GEN_H */
Signature · 1 line
2.45.1
Paolo Abeni
maintainerPATCH 2/5 net: introduce HW Rate limiting driver API
In reply to Paolo AbeniThe network devices gain a new ops struct to directly manipulate the H/W shapers implemented by the NIC.
The shapers can be attached to a pre-defined set of 'domains' - port, device, etc. - and the overall shapers configuration pushed to the H/W is maintained by the kernel.
Each shaper is identified by an unique integer id based on the domain and additional domain-specific information - e.g. for the queue domain, the queue id.
The shaper manipulation is exposed to user-space implementing the NL operations described by the previous patch.
Note that an additional domain, not exposed to user-space, is defined here (VF - to allow implementing the existing ndo_set_vf_rate() on top of the API defined here). Such domain will allow implementing a generic ndo_set_vf_rate() on top the shapers API.
---
v4 -> v5:
- fixed a few typos
- set() operation commit changes in memory on success
- updated set() to handle positive return values for device op
- get() always fill the parent handle
v3 -> v4:
- fix build error with !NET_SHAPER
- fix skipping one item on shaper dump requiring multiple skbs
- add a bunch of comments
- clarified the return value for delete()
- rebased on yaml changes
v2 -> v3:
- completed the implementation for set/get/dump/delete NL ops
- plugged shaper cleanup at device dismatel
--- include/linux/netdevice.h | 16 ++ include/net/net_shaper.h | 195 +++++++++++++ net/core/dev.c | 2 + net/core/dev.h | 6 + net/shaper/shaper.c | 559 +++++++++++++++++++++++++++++++++++++- 5 files changed, 774 insertions(+), 4 deletions(-) create mode 100644 include/net/net_shaper.h
include/linux/netdevice.h +16 -0
@@ -80,6 +80,8 @@ struct xdp_buff;struct xdp_frame ;struct xdp_metadata_ops ;struct xdp_md ; +struct net_shaper_ops ; +struct net_shaper_data ;typedef u32 xdp_features_t ;
@@ -1597,6 +1599,13 @@ struct net_device_ops {int (*ndo_hwtstamp_set)(struct net_device *dev,struct kernel_hwtstamp_config *kernel_config,struct netlink_ext_ack *extack); + +#if IS_ENABLED (CONFIG_NET_SHAPER ) +/** @net_shaper_ops: Device shaping offload operations +* see include/net/net_shapers.h +*/ +const struct net_shaper_ops *net_shaper_ops; +#endif };/**
@@ -2405,6 +2414,13 @@ struct net_device {/** @irq_moder: dim parameters used if IS_ENABLED(CONFIG_DIMLIB). */ struct dim_irq_moder *irq_moder; + +#if IS_ENABLED (CONFIG_NET_SHAPER ) +/** @net_shaper_data: data tracking the current shaper status +* see include/net/net_shapers.h +*/ +struct net_shaper_data *net_shaper_data; +#endif };#define to_net_dev (d) container_of(d, struct net_device, dev)
include/net/net_shaper.h new file +195 -0
195 lines not on this page · raw message
net/core/dev.c +2 -0
@@ -11174,6 +11174,8 @@ void free_netdev(struct net_device *dev)/* Flush device addresses */ dev_addr_flush (dev); +dev_shaper_flush (dev); +list_for_each_entry_safe (p, n, &dev->napi_list, dev_list)netif_napi_del (p);
net/core/dev.h +6 -0
@@ -35,6 +35,12 @@ void dev_addr_flush(struct net_device *dev);int dev_addr_init (struct net_device *dev);void dev_addr_check (struct net_device *dev); +#if IS_ENABLED (CONFIG_NET_SHAPER ) +void dev_shaper_flush (struct net_device *dev); +#else +static inline void dev_shaper_flush (struct net_device *dev) {} +#endif +/* sysctls not referred to from outside net/core/ */ extern int netdev_unregister_timeout_secs;extern int weight_p;
net/shaper/shaper.c +555 -4
583 lines not on this page · raw message
Signature · 1 line
2.45.1
Paolo Abeni
maintainerPATCH 3/5 netlink: spec: add shaper introspection support
In reply to Paolo AbeniAllow the user-space to fine-grain query the shaping features supported by the NIC on each domain.
--- Documentation/netlink/specs/shaper.yaml | 74 +++++++++++++++++++++++++ include/uapi/linux/net_shaper.h | 17 ++++++ net/shaper/shaper.c | 11 ++++ net/shaper/shaper_nl_gen.c | 25 +++++++++ net/shaper/shaper_nl_gen.h | 3 + 5 files changed, 130 insertions(+)
Documentation/netlink/specs/shaper.yaml +74 -0
@@ -125,6 +125,51 @@ attribute-sets: name: priority - name: weight + - + name: capabilities + attributes: + - + name: ifindex + type: u32 + - + name: scope + type: u32 + enum: scope + - + name: support-metric-bps + doc: the device accepts 'bps' metric for bw-min, bw-max and burst + type: flag + - + name: support-metric-pps + doc: the device accepts 'pps' metric for bw-min, bw-max and burst + type: flag + - + name: support-nesting + doc: | + the device supports nesting shaper belonging to this scope + below 'detached' scoped shapers. Only 'queue', 'netdev' and + 'detached' scope and flag 'support-nesting'. + type: flag + - + name: support-bw-min + doc: the device supports a minimum guaranteed bw + type: flag + - + name: support-bw-max + doc: the device supports maximum bw shaping + type: flag + - + name: support-burst + doc: the device supports a maximum burst size + type: flag + - + name: support-priority + doc: the device supports priority scheduling + type: flag + - + name: support-weight + doc: the device supports weighted round robin scheduling + type: flag operations: list:
@@ -200,3 +245,32 @@ operations: reply: attributes: - modified + + - + name: cap-get + doc: | + Get / Dump the shaper capabilities supported by the given device + attribute-set: capabilities + + do: + request: + attributes: + - ifindex + - scope + reply: + attributes: &cap-attrs + - support-metric-bps + - support-metric-pps + - support-nesting + - support-bw-min + - support-bw-max + - support-burst + - support-priority + - support-weight + + dump: + request: + attributes: + - ifindex + reply: + attributes: *cap-attrs
include/uapi/linux/net_shaper.h +17 -0
@@ -61,10 +61,27 @@ enum {NET_SHAPER_A_MAX = (__NET_SHAPER_A_MAX -1 ) }; +enum { +NET_SHAPER_A_CAPABILITIES_IFINDEX =1 , +NET_SHAPER_A_CAPABILITIES_SCOPE , +NET_SHAPER_A_CAPABILITIES_SUPPORT_METRIC_BPS , +NET_SHAPER_A_CAPABILITIES_SUPPORT_METRIC_PPS , +NET_SHAPER_A_CAPABILITIES_SUPPORT_NESTING , +NET_SHAPER_A_CAPABILITIES_SUPPORT_BW_MIN , +NET_SHAPER_A_CAPABILITIES_SUPPORT_BW_MAX , +NET_SHAPER_A_CAPABILITIES_SUPPORT_BURST , +NET_SHAPER_A_CAPABILITIES_SUPPORT_PRIORITY , +NET_SHAPER_A_CAPABILITIES_SUPPORT_WEIGHT , + + __NET_SHAPER_A_CAPABILITIES_MAX, +NET_SHAPER_A_CAPABILITIES_MAX = (__NET_SHAPER_A_CAPABILITIES_MAX -1 ) +}; +enum {NET_SHAPER_CMD_GET =1 ,NET_SHAPER_CMD_SET ,NET_SHAPER_CMD_DELETE , +NET_SHAPER_CMD_CAP_GET , __NET_SHAPER_CMD_MAX,NET_SHAPER_CMD_MAX = (__NET_SHAPER_CMD_MAX -1 )
net/shaper/shaper.c +11 -0
@@ -558,6 +558,17 @@ int net_shaper_nl_delete_doit(struct sk_buff *skb, struct genl_info *info)return ret; } +int net_shaper_nl_cap_get_doit (struct sk_buff *skb,struct genl_info *info) +{ +return 0 ; +} + +int net_shaper_nl_cap_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb) +{ +return 0 ; +} +void dev_shaper_flush (struct net_device *dev) {struct net_shaper_info *cur;
net/shaper/shaper_nl_gen.c +25 -0
@@ -50,6 +50,17 @@ static const struct nla_policy net_shaper_delete_nl_policy[NET_SHAPER_A_HANDLES [NET_SHAPER_A_HANDLES] = NLA_POLICY_NESTED(net_shaper_handle_nl_policy), }; +/* NET_SHAPER_CMD_CAP_GET - do */ +static const struct nla_policy net_shaper_cap_get_do_nl_policy[NET_SHAPER_A_CAPABILITIES_SCOPE +1 ] = { + [NET_SHAPER_A_CAPABILITIES_IFINDEX ] = { .type =NLA_U32 , }, + [NET_SHAPER_A_CAPABILITIES_SCOPE ] =NLA_POLICY_MAX (NLA_U32 ,4 ), +}; + +/* NET_SHAPER_CMD_CAP_GET - dump */ +static const struct nla_policy net_shaper_cap_get_dump_nl_policy[NET_SHAPER_A_CAPABILITIES_IFINDEX +1 ] = { + [NET_SHAPER_A_CAPABILITIES_IFINDEX ] = { .type =NLA_U32 , }, +}; +/* Ops table for net_shaper */ static const struct genl_split_ops net_shaper_nl_ops[] = { {
@@ -80,6 +91,20 @@ static const struct genl_split_ops net_shaper_nl_ops[] = { .maxattr = NET_SHAPER_A_HANDLES, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, + { + .cmd = NET_SHAPER_CMD_CAP_GET, + .doit = net_shaper_nl_cap_get_doit, + .policy = net_shaper_cap_get_do_nl_policy, + .maxattr = NET_SHAPER_A_CAPABILITIES_SCOPE, + .flags = GENL_CMD_CAP_DO, + }, + { + .cmd = NET_SHAPER_CMD_CAP_GET, + .dumpit = net_shaper_nl_cap_get_dumpit, + .policy = net_shaper_cap_get_dump_nl_policy, + .maxattr = NET_SHAPER_A_CAPABILITIES_IFINDEX, + .flags = GENL_CMD_CAP_DUMP, + }, }; struct genl_family net_shaper_nl_family __ro_after_init = {
net/shaper/shaper_nl_gen.h +3 -0
@@ -19,6 +19,9 @@ int net_shaper_nl_get_doit(struct sk_buff *skb, struct genl_info *info);int net_shaper_nl_get_dumpit (struct sk_buff *skb,struct netlink_callback *cb);int net_shaper_nl_set_doit (struct sk_buff *skb,struct genl_info *info);int net_shaper_nl_delete_doit (struct sk_buff *skb,struct genl_info *info); +int net_shaper_nl_cap_get_doit (struct sk_buff *skb,struct genl_info *info); +int net_shaper_nl_cap_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb);extern struct genl_family net_shaper_nl_family;
Signature · 1 line
2.45.1
Paolo Abeni
maintainerPATCH 4/5 net: shaper: implement introspection support
In reply to Paolo AbeniThe netlink op is a simple wrapper around the device callback.
Extend the existing fetch_dev() helper adding an attribute argument for the requested device. Reuse such helper in the newly implemented operation.
---
v4 -> v5:
- doc fix
--- include/net/net_shaper.h | 13 +++++ net/shaper/shaper.c | 108 +++++++++++++++++++++++++++++++++++---- 2 files changed, 112 insertions(+), 9 deletions(-)
include/net/net_shaper.h +13 -0
@@ -49,6 +49,7 @@ struct net_shaper_info {* struct net_shaper_ops - Operations on device H/W shapers * @set: Modify the existing shaper. * @delete: Delete the specified shaper. +* @capabilities: Introspect the device shaper-related features * * The initial shaping configuration ad device initialization is empty/ * a no-op/does not constraint the b/w in any way.
@@ -101,6 +102,18 @@ struct net_shaper_ops {*/ int (*delete)(struct net_device *dev,int nr,const u32 *handles,struct netlink_ext_ack *extack); + +/** capabilities - get the shaper features supported by the NIC +* @dev: netdevice to operate on +* @scope: the queried scope +* @flags: bitfield of supported features for the given scope +* +* Return: +* * %0 - Success, @flags is set according to the supported features +* * %-EOPNOTSUPP - the H/W does not support the specified scope +*/ +int (*capabilities)(struct net_device *dev,enum net_shaper_scope , +unsigned long *flags); };#define NET_SHAPER_SCOPE_SHIFT 16
net/shaper/shaper.c +99 -9
158 lines not on this page · raw message
Signature · 1 line
2.45.1
Paolo Abeni
maintainerPATCH 5/5 testing: net-drv: add basic shaper test
In reply to Paolo AbeniLeverage a basic/dummy netdevsim implementation to do functional coverage for NL interface.
---
v4 -> v5:
- fixed a few typos
--- drivers/net/Kconfig | 1 + drivers/net/netdevsim/netdev.c | 29 +++ tools/testing/selftests/drivers/net/Makefile | 1 + tools/testing/selftests/drivers/net/shaper.py | 198 ++++++++++++++++++ .../testing/selftests/net/lib/py/__init__.py | 1 + tools/testing/selftests/net/lib/py/ynl.py | 5 + 6 files changed, 235 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/shaper.py
drivers/net/Kconfig +1 -0
@@ -641,6 +641,7 @@ config NETDEVSIM depends on PTP_1588_CLOCK_MOCK || PTP_1588_CLOCK_MOCK=n select NET_DEVLINK select PAGE_POOL + select NET_SHAPER help This driver is a developer testing tool and software model that can be used to test various control path networking APIs, especially
drivers/net/netdevsim/netdev.c +29 -0
@@ -22,6 +22,7 @@#include <net/netdev_queues.h> #include <net/page_pool/helpers.h> #include <net/netlink.h> +#include <net/net_shaper.h> #include <net/pkt_cls.h> #include <net/rtnetlink.h> #include <net/udp_tunnel.h>
@@ -475,6 +476,33 @@ static int nsim_stop(struct net_device *dev)return 0 ; } +static int nsim_shaper_set (struct net_device *dev,int nr, +const struct net_shaper_info *shapers, +struct netlink_ext_ack *extack) +{ +return nr; +} + +static int nsim_shaper_del (struct net_device *dev,int nr, +const u32 *handles, +struct netlink_ext_ack *extack) +{ +return nr; +} + +static int nsim_shaper_cap (struct net_device *dev,enum net_shaper_scope scope, +unsigned long *flags) +{ + *flags =ULONG_MAX ; +return 0 ; +} + +static const struct net_shaper_ops nsim_shaper_ops = { + .set = nsim_shaper_set, + .delete = nsim_shaper_del, + .capabilities = nsim_shaper_cap, +}; +static const struct net_device_ops nsim_netdev_ops = { .ndo_start_xmit = nsim_start_xmit, .ndo_set_rx_mode = nsim_set_rx_mode,
@@ -496,6 +524,7 @@ static const struct net_device_ops nsim_netdev_ops = { .ndo_bpf = nsim_bpf, .ndo_open = nsim_open, .ndo_stop = nsim_stop, + .net_shaper_ops = &nsim_shaper_ops, };static const struct net_device_ops nsim_vf_netdev_ops = {
tools/testing/selftests/drivers/net/Makefile +1 -0
@@ -6,6 +6,7 @@ TEST_PROGS := \ ping.py \ queues.py \ stats.py \ + shaper.py # end of TEST_PROGS include ../../lib.mk
tools/testing/selftests/drivers/net/shaper.py new file +198 -0
198 lines not on this page · raw message
tools/testing/selftests/net/lib/py/__init__.py +1 -0
@@ -6,3 +6,4 @@ from .netns import NetNS from .nsim import * from .utils import * from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily +from .ynl import ShaperFamily
tools/testing/selftests/net/lib/py/ynl.py +5 -0
@@ -47,3 +47,8 @@ class NetdevFamily(YnlFamily): def __init__(self): super().__init__((SPEC_PATH / Path('netdev.yaml')).as_posix(), schema='') + +class ShaperFamily(YnlFamily): + def __init__(self): + super().__init__((SPEC_PATH / Path('shaper.yaml')).as_posix(), + schema='')
Signature · 1 line
2.45.1
Jakub Kicinski
maintainerQuoted 2 lines · On Thu, 27 Jun 2024 22:17:17 +0200 Paolo Abeni wrote:
The ice driver support is currently a WIP, sharing the current status earlier since some APIs details are still under discussion.
Let's stick to RFC until it's ready? Quoting documentation:
netdevsim
~~~~~~~~~
``netdevsim`` is a test driver which can be used to exercise driver configuration APIs without requiring capable hardware. Mock-ups and tests based on ``netdevsim`` are strongly encouraged when
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
adding new APIs, but ``netdevsim`` in itself is **not** considered a use case/user. You must also implement the new APIs in a real driver. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
See: https://www.kernel.org/doc/html/next/process/maintainer-netdev.html#netdevsim
Signature · 1 line
pw-bot: rfc
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
In reply to Paolo AbeniQuoted 6 lines · On Thu, 27 Jun 2024 22:17:18 +0200 Paolo Abeni wrote:
Documentation/netlink/specs/shaper.yaml new file +6
-0+ - + name: port + doc: The root shaper for the whole H/W. + - + name: netdev + doc: The main shaper for the given network device.
Documentation/netlink/specs/shaper.yaml line 30
Quoted 10 lines
Documentation/netlink/specs/shaper.yaml new file +10
-0+ - + name: queue + doc: The shaper is attached to the given device queue. + - + name: detached + doc: | + The shaper can be attached to port, netdev or other + detached shapers, allowing nesting and grouping of + netdev or queues. + render-max: true
nit: move other properties before the list
Documentation/netlink/specs/shaper.yaml line 45
Quoted 3 lines
Documentation/netlink/specs/shaper.yaml new file +3
-0+attribute-sets: + - + name: net_shaper
s/_/-/ on all names
Quoted 9 lines
Documentation/netlink/specs/shaper.yaml new file +9
-0+ attributes: + - + name: ifindex + type: u32 + - + name: parent + type: nest + nested-attributes: handle + -
Documentation/netlink/specs/shaper.yaml line 64
Quoted 10 lines
Documentation/netlink/specs/shaper.yaml new file +10
-0+ name: handle + type: nest + nested-attributes: handle + - + name: metric + type: u32 + enum: metric + - + name: bw-min + type: u64
s/u64/uint/
Documentation/netlink/specs/shaper.yaml line 88
Quoted 5 lines
Documentation/netlink/specs/shaper.yaml new file +5
-0+ - + name: handles + type: nest + multi-attr: true + nested-attributes: handle
oh both handle and handles...
it'd be good to have more doc: for the attributes, all these things get auto-rendered on docs.kernel.org
Documentation/netlink/specs/shaper.yaml line 93
Quoted 5 lines
Documentation/netlink/specs/shaper.yaml new file +5
-0+ - + name: shapers + type: nest + multi-attr: true + nested-attributes: ns-info
How do shapers differ from shaping attrs in this scope? :S
Documentation/netlink/specs/shaper.yaml line 99
Quoted 6 lines
Documentation/netlink/specs/shaper.yaml new file +6
-0+ - + name: modified + type: u32 + - + name: pad + type: pad
after using uint this can go
Documentation/netlink/specs/shaper.yaml line 136
Quoted 8 lines
Documentation/netlink/specs/shaper.yaml new file +8
-0+operations: + list: + - + name: get + doc: | + Get / Dump information about a/all the shaper for a given device + attribute-set: net_shaper + flags: [ admin-perm ]
Any reason why get is admin-perm ?
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 9 lines · Jakub Kicinski
Documentation/netlink/specs/shaper.yaml line 93
Quoted 5 lines · On Thu, 27 Jun 2024 22:17:18 +0200 Paolo Abeni wrote:
Documentation/netlink/specs/shaper.yaml new file +5
-0+ - + name: shapers + type: nest + multi-attr: true + nested-attributes: ns-infoHow do shapers differ from shaping attrs in this scope? :S
the set() operation must configure multiple shapers with a single command - to allow the 'atomic configuration changes' need for Andrew's use-case.
Out-of-sheer ignorance on my side, the above was the most straight- forward way to provide set() with an array of shapers.
Do you mean there are better way to achieve the goal, or "just" that the documentation here is missing and _necessary_?
Quoted 10 lines
Documentation/netlink/specs/shaper.yaml line 136
Quoted 8 lines
Documentation/netlink/specs/shaper.yaml new file +8
-0+operations: + list: + - + name: get + doc: | + Get / Dump information about a/all the shaper for a given device + attribute-set: net_shaper + flags: [ admin-perm ]Any reason why get is admin-perm ?
Mostly a "better safe then sorry" approach and cargo-cult form other recent yaml changes the hard reasons. Fine to drop it, if there is agreement.
Side note: ack to everything else noted in your reply.
Thanks,
Paolo
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
In reply to Jakub KicinskiQuoted 7 lines · Jakub Kicinski
Documentation/netlink/specs/shaper.yaml line 45
Quoted 3 lines · On Thu, 27 Jun 2024 22:17:18 +0200 Paolo Abeni wrote:
Documentation/netlink/specs/shaper.yaml new file +3
-0+attribute-sets: + - + name: net_shapers/_/-/ on all names
It looks like the initial
name: net_shaper
would require some patching to ynl-gen-c.py to handle the s/_/-/ conversion in the generated file names, too.
Cheers,
Paolo
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 7 lines · Paolo Abeni
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Thu, 27 Jun 2024 22:17:18 +0200 Paolo Abeni wrote:
(quoted diff) +3
-0+attribute-sets: + - + name: net_shaper
Quoted 8 lines
Quoted 1 line
s/_/-/ on all names
It looks like the initial
name:
net_shaperwould require some patching to
ynl-gen-c.pyto handle the s/_/-/ conversion in the generated file names, too.
I mean something alike the following, to be explicit:
---
tools/net/ynl/ynl-gen-c.py +26 -26
@@ -59,9 +59,9 @@ class Type(SpecAttr): if 'nested-attributes' in attr: self.nested_attrs = attr['nested-attributes'] if self.nested_attrs == family.name: - self.nested_render_name = c_lower(f"{family.name}") + self.nested_render_name = c_lower(f"{family.ident_name}") else: - self.nested_render_name = c_lower(f"{family.name}_{self.nested_attrs}") + self.nested_render_name = c_lower(f"{family.ident_name}_{self.nested_attrs}") if self.nested_attrs in self.family.consts: self.nested_struct_type = 'struct ' + self.nested_render_name + '_'
@@ -693,9 +693,9 @@ class Struct: self.nested = type_list is None if family.name == c_lower(space_name): - self.render_name = c_lower(family.name) + self.render_name = c_lower(family.ident_name) else: - self.render_name = c_lower(family.name + '-' + space_name) + self.render_name = c_lower(family.ident_name + '-' + space_name) self.struct_name = 'struct ' + self.render_name if self.nested and space_name in family.consts: self.struct_name += '_'
@@ -761,7 +761,7 @@ class EnumEntry(SpecEnumEntry): class EnumSet(SpecEnumSet): def __init__(self, family, yaml): - self.render_name = c_lower(family.name + '-' + yaml['name']) + self.render_name = c_lower(family.ident_name + '-' + yaml['name']) if 'enum-name' in yaml: if yaml['enum-name']:
@@ -777,7 +777,7 @@ class EnumSet(SpecEnumSet): else: self.user_type = 'int' - self.value_pfx = yaml.get('name-prefix', f"{family.name}-{yaml['name']}-") + self.value_pfx = yaml.get('name-prefix', f"{family.ident_name}-{yaml['name']}-") super().__init__(family, yaml)
@@ -802,9 +802,9 @@ class AttrSet(SpecAttrSet): if 'name-prefix' in yaml: pfx = yaml['name-prefix'] elif self.name == family.name: - pfx = family.name + '-a-' + pfx = family.ident_name + '-a-' else: - pfx = f"{family.name}-a-{self.name}-" + pfx = f"{family.ident_name}-a-{self.name}-" self.name_prefix = c_upper(pfx) self.max_name = c_upper(self.yaml.get('attr-max-name', f"{self.name_prefix}max")) self.cnt_name = c_upper(self.yaml.get('attr-cnt-name', f"__{self.name_prefix}max"))
@@ -861,7 +861,7 @@ class Operation(SpecOperation): def __init__(self, family, yaml, req_value, rsp_value): super().__init__(family, yaml, req_value, rsp_value) - self.render_name = c_lower(family.name + '_' + self.name) + self.render_name = c_lower(family.ident_name + '_' + self.name) self.dual_policy = ('do' in yaml and 'request' in yaml['do']) and \ ('dump' in yaml and 'request' in yaml['dump'])
@@ -911,11 +911,11 @@ class Family(SpecFamily): if 'uapi-header' in self.yaml: self.uapi_header = self.yaml['uapi-header'] else: - self.uapi_header = f"linux/{self.name}.h" + self.uapi_header = f"linux/{self.ident_name}.h" if self.uapi_header.startswith("linux/") and self.uapi_header.endswith('.h'): self.uapi_header_name = self.uapi_header[6:-2] else: - self.uapi_header_name = self.name + self.uapi_header_name = self.ident_name def resolve(self): self.resolve_up(super())
@@ -923,7 +923,7 @@ class Family(SpecFamily): if self.yaml.get('protocol', 'genetlink') not in {'genetlink', 'genetlink-c', 'genetlink-legacy'}: raise Exception("Codegen only supported for genetlink") - self.c_name = c_lower(self.name) + self.c_name = c_lower(self.ident_name) if 'name-prefix' in self.yaml['operations']: self.op_prefix = c_upper(self.yaml['operations']['name-prefix']) else:
@@ -2173,7 +2173,7 @@ def print_kernel_op_table_fwd(family, cw, terminate): exported = not kernel_can_gen_family_struct(family) if not terminate or exported: - cw.p(f"/* Ops table for {family.name} */") + cw.p(f"/* Ops table for {family.ident_name} */") pol_to_struct = {'global': 'genl_small_ops', 'per-op': 'genl_ops',
@@ -2225,12 +2225,12 @@ def print_kernel_op_table_fwd(family, cw, terminate): continue if 'do' in op: - name = c_lower(f"{family.name}-nl-{op_name}-doit") + name = c_lower(f"{family.ident_name}-nl-{op_name}-doit") cw.write_func_prot('int', name, ['struct sk_buff *skb', 'struct genl_info *info'], suffix=';') if 'dump' in op: - name = c_lower(f"{family.name}-nl-{op_name}-dumpit") + name = c_lower(f"{family.ident_name}-nl-{op_name}-dumpit") cw.write_func_prot('int', name, ['struct sk_buff *skb', 'struct netlink_callback *cb'], suffix=';') cw.nl()
@@ -2256,13 +2256,13 @@ def print_kernel_op_table(family, cw): for x in op['dont-validate']])), ) for op_mode in ['do', 'dump']: if op_mode in op: - name = c_lower(f"{family.name}-nl-{op_name}-{op_mode}it") + name = c_lower(f"{family.ident_name}-nl-{op_name}-{op_mode}it") members.append((op_mode + 'it', name)) if family.kernel_policy == 'per-op': struct = Struct(family, op['attribute-set'], type_list=op['do']['request']['attributes']) - name = c_lower(f"{family.name}-{op_name}-nl-policy") + name = c_lower(f"{family.ident_name}-{op_name}-nl-policy") members.append(('policy', name)) members.append(('maxattr', struct.attr_max_val.enum_name)) if 'flags' in op:
@@ -2294,7 +2294,7 @@ def print_kernel_op_table(family, cw): members.append(('validate', ' | '.join([c_upper('genl-dont-validate-' + x) for x in dont_validate])), ) - name = c_lower(f"{family.name}-nl-{op_name}-{op_mode}it") + name = c_lower(f"{family.ident_name}-nl-{op_name}-{op_mode}it") if 'pre' in op[op_mode]: members.append((cb_names[op_mode]['pre'], c_lower(op[op_mode]['pre']))) members.append((op_mode + 'it', name))
@@ -2305,9 +2305,9 @@ def print_kernel_op_table(family, cw): type_list=op[op_mode]['request']['attributes']) if op.dual_policy: - name = c_lower(f"{family.name}-{op_name}-{op_mode}-nl-policy") + name = c_lower(f"{family.ident_name}-{op_name}-{op_mode}-nl-policy") else: - name = c_lower(f"{family.name}-{op_name}-nl-policy") + name = c_lower(f"{family.ident_name}-{op_name}-nl-policy") members.append(('policy', name)) members.append(('maxattr', struct.attr_max_val.enum_name)) flags = (op['flags'] if 'flags' in op else []) + ['cmd-cap-' + op_mode]
@@ -2326,7 +2326,7 @@ def print_kernel_mcgrp_hdr(family, cw): cw.block_start('enum') for grp in family.mcgrps['list']: - grp_id = c_upper(f"{family.name}-nlgrp-{grp['name']},") + grp_id = c_upper(f"{family.ident_name}-nlgrp-{grp['name']},") cw.p(grp_id) cw.block_end(';') cw.nl()
@@ -2339,7 +2339,7 @@ def print_kernel_mcgrp_src(family, cw): cw.block_start('static const struct genl_multicast_group ' + family.c_name + '_nl_mcgrps[] =') for grp in family.mcgrps['list']: name = grp['name'] - grp_id = c_upper(f"{family.name}-nlgrp-{name}") + grp_id = c_upper(f"{family.ident_name}-nlgrp-{name}") cw.p('[' + grp_id + '] = { "' + name + '", },') cw.block_end(';') cw.nl()
@@ -2361,7 +2361,7 @@ def print_kernel_family_struct_src(family, cw): if not kernel_can_gen_family_struct(family): return - cw.block_start(f"struct genl_family {family.name}_nl_family __ro_after_init =") + cw.block_start(f"struct genl_family {family.ident_name}_nl_family __ro_after_init =") cw.p('.name\t\t= ' + family.fam_key + ',') cw.p('.version\t= ' + family.ver_key + ',') cw.p('.netnsok\t= true,')
@@ -2429,7 +2429,7 @@ def render_uapi(family, cw): cw.p(' */') uapi_enum_start(family, cw, const, 'name') - name_pfx = const.get('name-prefix', f"{family.name}-{const['name']}-") + name_pfx = const.get('name-prefix', f"{family.ident_name}-{const['name']}-") for entry in enum.entries.values(): suffix = ',' if entry.value_change:
@@ -2451,7 +2451,7 @@ def render_uapi(family, cw): cw.nl() elif const['type'] == 'const': defines.append([c_upper(family.get('c-define-name', - f"{family.name}-{const['name']}")), + f"{family.ident_name}-{const['name']}")), const['value']]) if defines:
@@ -2529,7 +2529,7 @@ def render_uapi(family, cw): defines = [] for grp in family.mcgrps['list']: name = grp['name'] - defines.append([c_upper(grp.get('c-define-name', f"{family.name}-mcgrp-{name}")), + defines.append([c_upper(grp.get('c-define-name', f"{family.ident_name}-mcgrp-{name}")), f'{name}']) cw.nl() if defines:
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 4 lines · On Mon, 01 Jul 2024 17:50:17 +0200 Paolo Abeni wrote:
Quoted 2 lines
would require some patching to
ynl-gen-c.pyto handle the s/_/-/ conversion in the generated file names, too.I mean something alike the following, to be explicit:
Great, thanks! I wasn't sure we're actually ready to make the family name use dashes, since we don't have any precedent. But since you have the patch - please submit!
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
In reply to Paolo AbeniQuoted 12 lines · On Mon, 01 Jul 2024 12:14:32 +0200 Paolo Abeni wrote:
Quoted 7 lines
Documentation/netlink/specs/shaper.yaml line 93
Quoted 5 lines
Documentation/netlink/specs/shaper.yaml new file +5
-0+ - + name: shapers + type: nest + multi-attr: true + nested-attributes: ns-infoHow do shapers differ from shaping attrs in this scope? :S
the
set()operation must configure multiple shapers with a single command - to allow the 'atomic configuration changes' need for Andrew's use-case.
Quoted 5 lines
Out-of-sheer ignorance on my side, the above was the most straight- forward way to provide
set()with an array of shapers.Do you mean there are better way to achieve the goal, or "just" that the documentation here is missing and _necessary_?
I see, I had a look at patch 2 now.
But that's really "Andrew's use-case" it doesn't cover deletion, right? Sorry that I don't have a perfect suggestion either but it seems like a half-measure. It's a partial support for transactions. If we want transactions we should group ops like nftables. Have normal ops (add, delete, modify) and control ops (start, commit) which clone the entire tree, then ops change it, and commit presents new tree to the device.
Alternative would be to, instead of supporting transactions have some form of a "complex instruction set". Most transformations will take a set of inputs (+weights / prios), shaping params, and where to attach.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Documentation/netlink/specs/shaper.yaml new file +8
-0+operations: + list: + - + name: get + doc: | + Get / Dump information about a/all the shaper for a given device + attribute-set: net_shaper + flags: [ admin-perm ]
Quoted 5 lines
Quoted 1 line
Any reason why get is admin-perm ?
Mostly a "better safe then sorry" approach and cargo-cult form other recent yaml changes the hard reasons. Fine to drop it, if there is agreement.
I thought we default to GET being non-privileged.
I think that's better, monitoring shouldn't require admin perm and presumably those shapers may grow stats at some stage. But no strong feelings.
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 19 lines · Jakub Kicinski
Quoted 12 lines · On Mon, 01 Jul 2024 12:14:32 +0200 Paolo Abeni wrote:
Quoted 7 lines
Documentation/netlink/specs/shaper.yaml line 93
Quoted 5 lines
Documentation/netlink/specs/shaper.yaml new file +5
-0+ - + name: shapers + type: nest + multi-attr: true + nested-attributes: ns-infoHow do shapers differ from shaping attrs in this scope? :S
the
set()operation must configure multiple shapers with a single command - to allow the 'atomic configuration changes' need for Andrew's use-case.Quoted 5 lines
Out-of-sheer ignorance on my side, the above was the most straight- forward way to provide
set()with an array of shapers.Do you mean there are better way to achieve the goal, or "just" that the documentation here is missing and _necessary_?
Quoted 7 lines
I see, I had a look at patch 2 now.
But that's really "Andrew's use-case" it doesn't cover deletion, right? Sorry that I don't have a perfect suggestion either but it seems like a half-measure. It's a partial support for transactions. If we want transactions we should group ops like nftables. Have normal ops (add, delete, modify) and control ops (start, commit) which clone the entire tree, then ops change it, and commit presents new tree to the device.
Yes, it does not cover deletion _and_ update/add/move within the same atomic operation.
Still any configuration could be reached from default/initial state with set(<possibly many shapers>). Additionally, given any arbitrary configuration, the default/initial state could be restored with a single delete(<possibly many handlers>).
The above covers any possible limitation enforced by the H/W, not just the DSA use-case.
Do you have a strong feeling for atomic transactions from any arbitrary state towards any other? If so, I’d like to understand why?
Dealing with transactions allowing arbitrary any state <> any state atomic changes will involve some complex logic that seems better assigned to user-space.
Thanks,
Paolo
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 8 lines · On Tue, 02 Jul 2024 16:21:38 +0200 Paolo Abeni wrote:
Quoted 7 lines
I see, I had a look at patch 2 now.
But that's really "Andrew's use-case" it doesn't cover deletion, right? Sorry that I don't have a perfect suggestion either but it seems like a half-measure. It's a partial support for transactions. If we want transactions we should group ops like nftables. Have normal ops (add, delete, modify) and control ops (start, commit) which clone the entire tree, then ops change it, and commit presents new tree to the device.
Quoted 7 lines
Yes, it does not cover deletion _and_ update/add/move within the same atomic operation.
Still any configuration could be reached from default/initial state with set(<possibly many shapers>). Additionally, given any arbitrary configuration, the default/initial state could be restored with a single delete(<possibly many handlers>).
From:
q0 -. RR \
q1 / > SP
q2 -. RR /
q3 /
To:
q0 ------\
q1 -------> SP
q2 -. RR /
q3 /
You have to both delete an RR node, and set SP params on Q0 and Q1.
Quoted 5 lines
The above covers any possible limitation enforced by the H/W, not just the DSA use-case.
Do you have a strong feeling for atomic transactions from any arbitrary state towards any other? If so, I’d like to understand why?
I don't believe this is covers all cases.
Quoted 3 lines
Dealing with transactions allowing arbitrary any state <> any state atomic changes will involve some complex logic that seems better assigned to user-space.
Complex logic in which part of the code?
It's just a full clone of the xarray, then do whatever ops user is asking to do, then tree walk to render diff as a set of ops. If you mean the tree walk to convert tree diff into ops, I think we need that anyway, otherwise we may get into a situation where there's a dependency between the user space implementation and driver expectations.
Paolo Abeni
maintainerPATCH 1/5 netlink: spec: add shaper YAML spec
In reply to a message not in the archiveOops, I unintentionally dropped most recipients, re-adding them, sorry for the duplicate.
Quoted 17 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 02 Jul 2024 16:21:38 +0200 Paolo Abeni wrote:
Quoted 7 lines
I see, I had a look at patch 2 now.
But that's really "Andrew's use-case" it doesn't cover deletion, right? Sorry that I don't have a perfect suggestion either but it seems like a half-measure. It's a partial support for transactions. If we want transactions we should group ops like nftables. Have normal ops (add, delete, modify) and control ops (start, commit) which clone the entire tree, then ops change it, and commit presents new tree to the device.Quoted 7 lines
Yes, it does not cover deletion _and_ update/add/move within the same atomic operation.
Still any configuration could be reached from default/initial state with set(<possibly many shapers>). Additionally, given any arbitrary configuration, the default/initial state could be restored with a single delete(<possibly many handlers>).
Quoted 6 lines
From:
q0 -. RR \
q1 / > SP
q2 -. RR /
q3 /
Call this C1
Quoted 6 lines
To:
q0 ------\
q1 -------> SP
q2 -. RR /
q3 /
Call this C2
Quoted 1 line
You have to both delete an RR node, and set SP params on Q0 and Q1.
default -> C1:
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \
--do set --json '{ "ifindex":2, "shapers": [ \
{ "parent": { "scope": "netdev"}, "handle": {
"scope": "detached", "id": 1 }, "priority": 1 },
{ "parent": { "scope": "netdev"}, "handle": {
"scope": "detached", "id": 2 }, "priority": 2 },
{ "parent": { "scope": "detached", "id":1},
"handle": { "scope": "queue", "id": 1 }, "weight": 1 },
{ "parent": { "scope": "detached", "id":1},
"handle": { "scope": "queue", "id": 2 }, "weight": 2 },
{ "parent": { "scope": "detached" "id":2},
"handle": { "scope": "queue", "id": 3 }, "weight": 1 },
{ "parent": { "scope": "detached" "id":2},"handle": { "scope": "queue", "id": 4 }, "weight": 2 }]} C1 -> C2:
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \
--do delete --json '{ "ifindex":2, "handles": [ \
{ "scope": "queue", "id": 1 },
{ "scope": "queue", "id": 2 },
{ "scope": "queue", "id": 3 },
{ "scope": "queue", "id": 4 },
{ "scope": "detached", "id": 1 },
{ "scope": "detached", "id": 2 }]}./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \
--do set --json '{ "ifindex":2, "shapers": [ \
{ "parent": { "scope": "netdev"}, "handle": {
"scope": "detached", "id": 1 }, "priority": 1 },
{ "parent": { "scope": "netdev"}, "handle": {
"scope": "queue", "id": 1 }, "priority": 2 },
{ "parent": { "scope": "netdev"}, "handle": {
"scope": "queue", "id": 2 }, "priorirty": 3 },
{ "parent": { "scope": "detached" "id":1},
"handle": { "scope": "queue", "id": 3 }, "weight": 1 },
{ "parent": { "scope": "detached" "id":1},"handle": { "scope": "queue", "id": 4 }, "weight": 2 },
The goal is to allow the system to reach the final configuration, even with the assumption the H/W does not support any configuration except the starting one and the final one.
But the infra requires that the system _must_ support at least a 3rd configuration, the default one.
Quoted 7 lines
Quoted 5 lines
The above covers any possible limitation enforced by the H/W, not just the DSA use-case.
Do you have a strong feeling for atomic transactions from any arbitrary state towards any other? If so, I’d like to understand why?
I don't believe this is covers all cases.
Given any pair of configurations C1, C2 we can reach C2 via C1 -> default, default -> C2. respecting any H/W constraint.
Quoted 5 lines
Quoted 3 lines
Dealing with transactions allowing arbitrary any state <> any state atomic changes will involve some complex logic that seems better assigned to user-space.
Complex logic in which part of the code?
IIRC in a past iteration you pointed out that the complexity of computing the delta between 2 arbitrary configurations is significantly higher than going through the empty/default configuration.
In any case I think that the larger complexity to implement a full transactional model. nft had proven that to be very hard and bug prone. I really would avoid that option, if possible.
Thanks,
Paolo
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
In reply to a message not in the archiveQuoted 1 line · Jakub Kicinski
Not sure if dropping CCs was on purpose ?
Nope, just PEBKAC here. Re-adding them and include your reply verbatim, to avoid losing track on the ML
Quoted 77 lines · On Tue, 2 Jul 2024 21:49:09 +0200 Paolo Abeni wrote:
Quoted 68 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 02 Jul 2024 16:21:38 +0200 Paolo Abeni wrote:
Quoted 7 lines
Yes, it does not cover deletion _and_ update/add/move within the same atomic operation.
Still any configuration could be reached from default/initial state with set(<possibly many shapers>). Additionally, given any arbitrary configuration, the default/initial state could be restored with a single delete(<possibly many handlers>).
Quoted 6 lines
From:
q0 -. RR \
q1 / > SP
q2 -. RR /
q3 /Call this C1
Quoted 6 lines
To:
q0 ------\
q1 -------> SP
q2 -. RR /
q3 /Call this C2
Quoted 1 line
You have to both delete an RR node, and set SP params on Q0 and Q1.
default -> C1:
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \ --do set --json '{ "ifindex":2, "shapers": [ \ { "parent": { "scope": "netdev"}, "handle": { "scope": "detached", "id": 1 }, "priority": 1 }, { "parent": { "scope": "netdev"}, "handle": { "scope": "detached", "id": 2 }, "priority": 2 }, { "parent": { "scope": "detached", "id":1}, "handle": { "scope": "queue", "id": 1 }, "weight": 1 }, { "parent": { "scope": "detached", "id":1}, "handle": { "scope": "queue", "id": 2 }, "weight": 2 }, { "parent": { "scope": "detached" "id":2}, "handle": { "scope": "queue", "id": 3 }, "weight": 1 }, { "parent": { "scope": "detached" "id":2},"handle": { "scope": "queue", "id": 4 }, "weight": 2 }]} C1 -> C2:
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \ --do delete --json '{ "ifindex":2, "handles": [ \ { "scope": "queue", "id": 1 }, { "scope": "queue", "id": 2 }, { "scope": "queue", "id": 3 }, { "scope": "queue", "id": 4 }, { "scope": "detached", "id": 1 }, { "scope": "detached", "id": 2 }]}./tools/net/ynl/cli.py --spec Documentation/netlink/specs/shaper.yaml \ --do set --json '{ "ifindex":2, "shapers": [ \ { "parent": { "scope": "netdev"}, "handle": { "scope": "detached", "id": 1 }, "priority": 1 }, { "parent": { "scope": "netdev"}, "handle": { "scope": "queue", "id": 1 }, "priority": 2 }, { "parent": { "scope": "netdev"}, "handle": { "scope": "queue", "id": 2 }, "priorirty": 3 }, { "parent": { "scope": "detached" "id":1}, "handle": { "scope": "queue", "id": 3 }, "weight": 1 }, { "parent": { "scope": "detached" "id":1},Quoted 7 lines
"handle": { "scope": "queue", "id": 4 }, "weight": 2 },
The goal is to allow the system to reach the final configuration, even with the assumption the H/W does not support any configuration except the starting one and the final one.
But the infra requires that the system _must_ support at least a 3rd configuration, the default one.
Quoted 6 lines
This assumes there is a single daemon responsible for control of all of the shaping, which I think the endless BPF multi-program back and forth proves to be incorrect.
We'd also lose stats each time, and make reconfiguration disruptive to running workloads.
Note there is no stats support for the shapers, nor is planned, nor the H/W I know of have any support for it.
The destructive operations will be needed only when the configuration change is inherently destructive. Nothing prevents the user-space to push a direct configuration change when possible - which should be the most frequent case, in practice.
Regarding the entity responsible for control, I had in mind a single one, yes. I read the above as you are looking forward to e.g. different applications configuring their own shaping accessing directly the NL interface, am I correct? Why can’t such applications talk to that daemon, instead?
Anyway different applications must touch disjoint resources (e.g. disjoint queues sets) right? In such a case even multiple destructive configuration changes (on disjoint resources set) will not be problematic.
Still if we want to allow somewhat consistent, concurrent, destructive configuration changes on shared resource (why? It sounds a bit of overdesign at this point), we could extend the set() operation to additional support shapers deletion, e.g. adding an additional ‘delete’ flag attribute to the ‘handle’ sub-set.
Quoted 29 lines
Quoted 15 lines
Quoted 7 lines
Quoted 5 lines
The above covers any possible limitation enforced by the H/W, not just the DSA use-case.
Do you have a strong feeling for atomic transactions from any arbitrary state towards any other? If so, I’d like to understand why?
I don't believe this is covers all cases.
Given any pair of configurations C1, C2 we can reach C2 via C1 -> default, default -> C2. respecting any H/W constraint.
Quoted 4 lines
Quoted 3 lines
Dealing with transactions allowing arbitrary any state <> any state atomic changes will involve some complex logic that seems better assigned to user-space.
Quoted 6 lines
Quoted 1 line
Complex logic in which part of the code?
IIRC in a past iteration you pointed out that the complexity of computing the delta between 2 arbitrary configurations is significantly higher than going through the empty/default configuration.
For a fixed-layout scheduler where HW blocks have a fixed mapping with user's hierarchy - it's easier to program the shapers from the hierarchy directly. Since node maps to the same set of registers reprogramming will be writing to a register a value it already has
- a noop. That's different than doing a full reset and reprogram each time, as two separate calls from user space.
Quoted 4 lines
For Intel's cases OTOH, when each command is a separate FW call we can't actually enforce the atomic transitions, right? Your code seems to handle returns smaller the number of commands, from which I infer that we may execute half of the modification?
Yes, the code and the NL API allows the NIC to do the update incrementally, and AFAICS Intel ICE has no support for complex transactions.
Somewhat enforcing atomic transitions will be quite complex at best and is not need to accomplish the stated goal - allow reconfiguration even when the H/W does not support intermediate states.
Do we need to enforce atomicity? Why? NFT has proven that a transational model implementation is hard, and should be avoided if possible.
Quoted 3 lines
IOW for Andrew's HW - he'd probably prefer to look at the resulting tree, no matter what previous state we were in. For Intel we _can't_ support atomic commands, if they span multiple cycles of FW exchanges?
My understanding is that we can’t have atomic updates on Intel, from firmare perspective. As said, I don’t think it’s necessary to support them.
WRT the DSA H/W, do you mean the core should always set() the whole known tree to the driver, regardless of the specific changes asked by the user-space? If so, what about letting the driver expose some capability (or private flag) asking the core for such behavior? So that the driver will do the largish set() only with the H/W requiring that.
Anyway I'm not sure the mentioned DSA H/W would benefit from always receiving the whole configuration. e.g. changing the weight for a single queue shaper would not need the whole data-set.
Quoted 7 lines
Quoted 3 lines
In any case I think that the larger complexity to implement a full transactional model. nft had proven that to be very hard and bug prone. I really would avoid that option, if possible.
Maybe instead of discussing the user space API it'd be more beneficial to figure out a solid way of translating the existing APIs into the new model?
Could you please rephrase? I think all the arguments discussed here are related to the model - at some point that impact the user space API, too.
Thanks,
Paolo
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 8 lines · On Wed, 03 Jul 2024 16:53:38 +0200 Paolo Abeni wrote:
Note there is no stats support for the shapers, nor is planned, nor the H/W I know of have any support for it.
The destructive operations will be needed only when the configuration change is inherently destructive. Nothing prevents the user-space to push a direct configuration change when possible - which should be the most frequent case, in practice.
Quoted 5 lines
Regarding the entity responsible for control, I had in mind a single one, yes. I read the above as you are looking forward to e.g. different applications configuring their own shaping accessing directly the NL interface, am I correct? Why can’t such applications talk to that daemon, instead?
We know such daemon did not materialize in other cases. We can assume there is a daemon if we think that's a good design. We shouldn't use a mirage of a third-party daemon to wave away problems we didn't manage to solve.
Quoted 10 lines
Anyway different applications must touch disjoint resources (e.g. disjoint queues sets) right? In such a case even multiple destructive configuration changes (on disjoint resources set) will not be problematic.
Still if we want to allow somewhat consistent, concurrent, destructive configuration changes on shared resource (why? It sounds a bit of overdesign at this point), we could extend the
set()operation to additional support shapers deletion, e.g. adding an additional ‘delete’ flag attribute to the ‘handle’ sub-set.
To judge whether it's an over-design we'd need to know what the user scenarios are, those are not included in the series AFAICS.
Quoted 37 lines
Quoted 11 lines
For a fixed-layout scheduler where HW blocks have a fixed mapping with user's hierarchy - it's easier to program the shapers from the hierarchy directly. Since node maps to the same set of registers reprogramming will be writing to a register a value it already has
- a noop. That's different than doing a full reset and reprogram each time, as two separate calls from user space.For Intel's cases OTOH, when each command is a separate FW call we can't actually enforce the atomic transitions, right? Your code seems to handle returns smaller the number of commands, from which I infer that we may execute half of the modification?
Yes, the code and the NL API allows the NIC to do the update incrementally, and AFAICS Intel ICE has no support for complex transactions.
Somewhat enforcing atomic transitions will be quite complex at best and is not need to accomplish the stated goal - allow reconfiguration even when the H/W does not support intermediate states.Do we need to enforce atomicity? Why? NFT has proven that a transational model implementation is hard, and should be avoided if possible.
Quoted 3 lines
IOW for Andrew's HW - he'd probably prefer to look at the resulting tree, no matter what previous state we were in. For Intel we _can't_ support atomic commands, if they span multiple cycles of FW exchanges?
My understanding is that we can’t have atomic updates on Intel, from firmare perspective. As said, I don’t think it’s necessary to support them.
WRT the DSA H/W, do you mean the core should always
set()the whole known tree to the driver, regardless of the specific changes asked by the user-space? If so, what about letting the driver expose some capability (or private flag) asking the core for such behavior? So that the driver will do the largishset()only with the H/W requiring that.
Quoted 3 lines
Anyway I'm not sure the mentioned DSA H/W would benefit from always receiving the whole configuration. e.g. changing the weight for a single queue shaper would not need the whole data-set.
To be blunt - what I'm getting at is that the API mirrors Intel's FW API with an extra kludge to support the DSA H/W - in the end matching neither what the DSA wants nor what Intel can do.
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
In any case I think that the larger complexity to implement a full transactional model. nft had proven that to be very hard and bug prone. I really would avoid that option, if possible.
Maybe instead of discussing the user space API it'd be more beneficial to figure out a solid way of translating the existing APIs into the new model?
Could you please rephrase? I think all the arguments discussed here are related to the model - at some point that impact the user space API, too.
I was hoping that implementing the code that mirrors the existing APIs into tree operations would teach us something about the primitives that operate on the tree. The proposed primitives are really low level, which is why we need to "fuse" them into multi-change operations.
More specifically what I described a few emails up was a group+schedule+limit paradigm. Instead of describing single moves and parameter setting - transformation would describe inputs, scheduling across them, and rate limit to apply. But one transformation would always operate on at most one MUX node.
IOW I'm trying to explore whether we can find a language of transformations which will be more complex than single micro-operations on the tree, but sufficiently expressive to provide atomic transformations without transactions of micro-ops.
Paolo Abeni
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 03 Jul 2024 16:53:38 +0200 Paolo Abeni wrote:
Anyway different applications must touch disjoint resources (e.g. disjoint queues sets) right? In such a case even multiple destructive configuration changes (on disjoint resources set) will not be problematic.
Quoted 8 lines
Quoted 5 lines
Still if we want to allow somewhat consistent, concurrent, destructive configuration changes on shared resource (why? It sounds a bit of overdesign at this point), we could extend the
set()operation to additional support shapers deletion, e.g. adding an additional ‘delete’ flag attribute to the ‘handle’ sub-set.To judge whether it's an over-design we'd need to know what the user scenarios are, those are not included in the series AFAICS.
My bad, in the cover I referred to the prior discussions without explicitly quoting the contents.
The initial goal here was to allow the user-space to configure per- queue, H/W offloaded, TX shaping.
That later evolved in introducing an in-kernel H/W offload TX shaping API capable of replacing the many existing similar in-kernel APIs and supporting the foreseeable H/W capabilities.
Quoted 3 lines
To be blunt - what I'm getting at is that the API mirrors Intel's FW API with an extra kludge to support the DSA H/W - in the end matching neither what the DSA wants nor what Intel can do.
The API is similar to Intel’s FW API because to my understanding the underlying design - an arbitrary tree - is the most complete representation possible for shaping H/W. AFAICT is also similar to what other NIC vendors’ offer.
I don’t see why the APIs don’t match what Intel can do, could you please elaborate on that?
Quoted 4 lines
IOW I'm trying to explore whether we can find a language of transformations which will be more complex than single micro-operations on the tree, but sufficiently expressive to provide atomic transformations without transactions of micro-ops.
I personally find it straight-forward describing the scenario you proposed in terms of the simple operations as allowed by this series. I also think it’s easier to build arbitrarily complex scenarios in terms of simple operations instead of trying to put enough complexity in the language to describe everything. It will easily lose flexibility or increase complexity for unclear gain. Why do you think that would be a better approach?
Also the simple building blocks approach is IMHO closer to the original use-case.
Are there any other reasons for atomic operations, beyond addressing low-end H/W? Note that WRT the specific limitations we are aware of, the APIs allows atomic conf changes without resorting to transition into the default configuration.
My biggest concern is that from my perspective we are incrementally adding requisites, with increasing complexity, further and further away from the initial use-case.
Thanks,
Paolo
Jakub Kicinski
maintainerRe: [PATCH net-next 1/5] netlink: spec: add shaper YAML spec
Quoted 8 lines · On Mon, 08 Jul 2024 21:42:00 +0200 Paolo Abeni wrote:
Quoted 2 lines
To judge whether it's an over-design we'd need to know what the user scenarios are, those are not included in the series AFAICS.
My bad, in the cover I referred to the prior discussions without explicitly quoting the contents.
The initial goal here was to allow the user-space to configure per- queue, H/W offloaded, TX shaping.
Per-queue Tx shaping is already supported.
I mean "user scenarios" in the agile programming sense as in something that gets closer to production use cases. "Set rate limit on a queue" is a suggested solution not a statement of a problem.
Quoted 8 lines
That later evolved in introducing an in-kernel H/W offload TX shaping API capable of replacing the many existing similar in-kernel APIs and supporting the foreseeable H/W capabilities.
Quoted 3 lines
To be blunt - what I'm getting at is that the API mirrors Intel's FW API with an extra kludge to support the DSA H/W - in the end matching neither what the DSA wants nor what Intel can do.
Quoted 7 lines
The API is similar to Intel’s FW API because to my understanding the underlying design - an arbitrary tree - is the most complete representation possible for shaping H/W. AFAICT is also similar to what other NIC vendors’ offer.
I don’t see why the APIs don’t match what Intel can do, could you please elaborate on that?
That's not the main point I was making, I was complaining about how the "extension" to support DSA HW was bolted onto this API.
Undeniably the implementation must be stored as a tree (with some max depth). That doesn't imply that, for example, arbitrary re-parenting of non-leaf nodes is an operation that makes sense for all devices. From memory devlink rate doesn't allow mixing some node types after one parent, too.
Quoted 5 lines
Quoted 4 lines
IOW I'm trying to explore whether we can find a language of transformations which will be more complex than single micro-operations on the tree, but sufficiently expressive to provide atomic transformations without transactions of micro-ops.
Quoted 7 lines
I personally find it straight-forward describing the scenario you proposed in terms of the simple operations as allowed by this series. I also think it’s easier to build arbitrarily complex scenarios in terms of simple operations instead of trying to put enough complexity in the language to describe everything. It will easily lose flexibility or increase complexity for unclear gain. Why do you think that would be a better approach?
I strongly dislike the operation grouping/batching as it stands in v1. Do you think that part of the design is clean?
I also disagree with the assertion that having a language of transformations more advanced that "add/move/delete" increases complexity. Language gives you properties you can reason about. That's why rbtree, B-tree and other algos define a language of transformations. Naive ops make arbitrary trees easy but I put it to you that allowing arbitrary trees without any enforced invariants will breed far more complexity and bugs than a properly designed language :)
The primary operation of interest is, in fact: given a set of resources (queues or netdevs) which currently feed one mux node - build a sub-hierarchy.
Use case 1 - container b/w sharing. Container manager will want to group a set of queues, and feed a higher layer RR node (so that number of queues doesn't impact load sharing).
Say we have two containers (c1, c2 represent queues assigned to them); before container 3 starts:
c1 - \
c1 - >RR
c1 - / > RR(root)
c2 - \ RR
c2 - /
allocate 2 queues:
c1 - \
c1 - ->RR
c1 - / > RR(root)
c2 - \ RR
c2 - / '
qX /
qY /
hierarchize ("group([qX, qY], type="rr")):
c1 - \
c1 - >RR
c1 - / \
c2 - \ RR -> RR(root)
c2 - / /
c3 - \ RR
c3 - /
The container manager just wants so say "take the new queues (X, Y), put them under an RR node". If the language is build around creating mux nodes - that's a single call.
Note that the RR(root) node is implicit (in your API it's not visible but it is implied).
Users case 2 - delegation - the neat thing about using such construct is that as you can see we never referred to output, i.e. RR(root). The output can be implied by whatever node the queues already output to. So say container 1 (c1) wants to set a b/w limit on two of it's queues (let's call its queues A B C):
rr_id = group([B C], type="rr")
rate_limit(rr_id, 1Gbps)
and end up with:
cA ----- \
cB - \RR*/ RR
cC - / \
c2 - \ RR -> RR(root)
c2 - / /
c3 - \ RR
c3 - /* new node, also has rate limit set
You don't have to worry about parentage permissions. Container can only add nodes (which is always safe) or delete nodes (and we can trivially enforce it only deletes node it has created itself).
Quoted 5 lines
Also the simple building blocks approach is IMHO closer to the original use-case.
Are there any other reasons for atomic operations, beyond addressing low-end H/W?
I think atomic changes are convenient to match what the user wants to do. And the second use case is that I do believe there's a real need to allow uncoordinated agents to modify sections on the hierarchy.
Neither of those are hard requirements, but I think any application driven requirement should come before "that's the FW API for vendor X" I hope this we can agree on.
Thinking about it (for longer than I care to admit), one concern I have about the "mux creation" API I described above is that it forces existence of leaf and non-leaf nodes at the same parent, at least transiently.
Can we go back to an API with an explicit create/modify/delete? All we need for Andrew's use case, I believe, is to be able to "somewhat atomically" move leaf nodes.