devlink and mlx5: Support cross-function rate scheduling
Subsystems (8)
Tariq Toukan
maintainerv5 cover devlink and mlx5: Support cross-function rate scheduling
Hi,
This series by Cosmin and Jiri adds support for cross-function rate scheduling in devlink and mlx5.
This is V5, find V4 here:
https://lore.kernel.org/all/[email protected]/
Regards,
Tariq
[1]
devlink objects support rate management for TX scheduling, which involves maintaining a tree of rate nodes that corresponds to TX schedulers in hardware. 'man devlink-rate' has the full details.
The tree of rate nodes is maintained per devlink object, protected by the devlink lock.
There exists hardware capable of instantiating TX scheduling trees spanning multiple functions of the same physical device (and thus devlink objects) and therefore the current API and locking scheme is insufficient.
This patch series changes the devlink rate implementation and API to allow supporting such hardware and managing TX scheduling trees across multiple functions of a physical device.
Modeling this requires having devlink rate nodes with parents in other devlink objects. A naive approach that relies on the current one-lock-per-devlink model is impossible, as it would require in some cases acquiring multiple devlink locks in the correct order.
The solution proposed in this patch series consists of two parts:
1. Representing the underlying physical NIC as a shared devlink object on the faux bus and nesting all its PF devlink instances in it.
2. Changing the devlink rate implementation to store rates in this shared devlink object, if it exists, and use its lock to protect against concurrent changes of the scheduling tree.
With these in place, cross-esw scheduling support is added to mlx5. The neat part about this approach is that it works for SFs as well, which are already nested in their parent PF instances. So with this series, complex scheduling trees spanning multiple SFs across multiple PFs of the same NIC can now be supported.
V1 of this patch series was sent a long time ago [2], using a different approach of storing rates in a shared rate domain with special locking rules. This new approach uses standard devlink instances and nesting.
Patches:
devlink rate changes for cross-device TX scheduling:
devlink: Reverse locking order for nested instances
documentation: networking: add shared devlink documentation
devlink: Add helpers to lock nested-in instances
devlink: Refactor devlink_rate_nodes_check
devlink: Decouple rate storage from associated devlink object
devlink: Add parent dev to devlink API
devlink: Allow parent dev for rate-set and rate-new
devlink: Allow rate node parents from other devlinks
devlink: introduce shared devlink instance for PFs on same chip
mlx5 support for cross-device TX scheduling:
net/mlx5: Add a shared devlink instance for PFs on same chip net/mlx5: Expose a function to clear a vport's parent net/mlx5: Store QoS sched nodes in the sh_devlink net/mlx5: qos: Support cross-esw tx scheduling net/mlx5: qos: Enable cross-device scheduling net/mlx5: Document devlink rates and cross-esw scheduling
[2] https://lore.kernel.org/netdev/[email protected]/
V5:
- Made parts of shd generic devlink infra (Jakub).
- Stopped using __free (Krzysztof Kozlowski).
- Moved some generated netlink in the correct patch (Simon Horman).
- Addressed cleanup bug (Jakub).
- Clarified uses of shared devlink in documentation.
V4:
- Fix typo in documentation (Randy Dunlap).
V3:
- Remove mistakenly repeated devlink interface in docs (Jakub).
- Add Jiri's review tags on ML.
V2:
- Rebase.
- Add Jiri's review tags on ML.
Cosmin Ratiu (12):
devlink: Reverse locking order for nested instances
devlink: Add helpers to lock nested-in instances
devlink: Refactor devlink_rate_nodes_check
devlink: Decouple rate storage from associated devlink object
devlink: Add parent dev to devlink API
devlink: Allow parent dev for rate-set and rate-new
devlink: Allow rate node parents from other devlinks net/mlx5: Expose a function to clear a vport's parent net/mlx5: Store QoS sched nodes in the sh_devlink net/mlx5: qos: Support cross-device tx scheduling net/mlx5: qos: Enable cross-device scheduling net/mlx5: Document devlink rates
Jiri Pirko (3):
documentation: networking: add shared devlink documentation
devlink: introduce shared devlink instance for PFs on same chip net/mlx5: Add a shared devlink instance for PFs on same chip
Documentation/netlink/specs/devlink.yaml | 22 +- .../networking/devlink/devlink-port.rst | 2 + .../networking/devlink/devlink-shared.rst | 95 +++++ Documentation/networking/devlink/index.rst | 1 + Documentation/networking/devlink/mlx5.rst | 33 ++ .../net/ethernet/mellanox/mlx5/core/Makefile | 5 +- .../net/ethernet/mellanox/mlx5/core/devlink.c | 1 + .../mellanox/mlx5/core/esw/devlink_port.c | 2 +- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 332 ++++++++---------- .../net/ethernet/mellanox/mlx5/core/esw/qos.h | 3 - .../net/ethernet/mellanox/mlx5/core/eswitch.c | 9 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 14 +- .../net/ethernet/mellanox/mlx5/core/main.c | 17 + .../ethernet/mellanox/mlx5/core/sh_devlink.c | 91 +++++ .../ethernet/mellanox/mlx5/core/sh_devlink.h | 14 + include/linux/mlx5/driver.h | 1 + include/net/devlink.h | 13 + include/uapi/linux/devlink.h | 2 + net/devlink/Makefile | 2 +- net/devlink/core.c | 48 ++- net/devlink/dev.c | 7 +- net/devlink/devl_internal.h | 11 +- net/devlink/netlink.c | 67 +++- net/devlink/netlink_gen.c | 23 +- net/devlink/netlink_gen.h | 8 + net/devlink/rate.c | 290 +++++++++++---- net/devlink/sh_dev.c | 163 +++++++++ 27 files changed, 978 insertions(+), 298 deletions(-) create mode 100644 Documentation/networking/devlink/devlink-shared.rst create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.h create mode 100644 net/devlink/sh_dev.c
base-commit: c5e7b1d1cc8a6cb8b709eef34c93a9458427ab2e
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 1/15 documentation: networking: add shared devlink documentation
From: Jiri Pirko <[email protected]>
Document shared devlink instances for multiple PFs on the same chip.
--- .../networking/devlink/devlink-shared.rst | 95 +++++++++++++++++++ Documentation/networking/devlink/index.rst | 1 + 2 files changed, 96 insertions(+) create mode 100644 Documentation/networking/devlink/devlink-shared.rst
Documentation/networking/devlink/devlink-shared.rst new file +95 -0
@@ -0,0 +1,95 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Devlink Shared Instances +============================ + +Overview +======== + +Shared devlink instances allow multiple physical functions (PFs) on the same +chip to share an additional devlink instance for chip-wide operations. This +is implemented within individual drivers alongside the individual PF devlink +instances, not replacing them. + +Multiple PFs may reside on the same physical chip, running a single firmware. +Some of the resources and configurations may be shared among these PFs. The +shared devlink instance provides an object to pin configuration knobs on. + +The shared devlink instance is backed by a faux device and provides a common +interface for operations that affect the entire chip rather than individual PFs. +A faux device is used as a backing device for the 'entire chip' since there's no +additional real device instantiated by hardware besides the PF devices. + +Implementation +============== + +Architecture +------------ + +The implementation uses: + +* **Faux device**: Virtual device backing the shared devlink instance +* **Chip identification**: PFs are grouped by chip using a driver-specific identifier +* **Shared instance management**: Global list of shared instances with reference counting + +API Functions +------------- + +The following functions are provided for managing shared devlink instances: + +* ``devlink_shd_get()``: Get or create a shared devlink instance identified by a string ID +* ``devlink_shd_put()``: Release a reference on a shared devlink instance +* ``devlink_shd_get_priv()``: Get private data from shared devlink instance + +Initialization Flow +------------------- + +1. **PF calls shared devlink init** during driver probe +2. **Chip identification** using driver-specific method to determine device identity +3. **Get or create shared instance** using ``devlink_shd_get()``: + + * The function looks up existing instance by identifier + * If none exists, creates new instance: + - Creates faux device with chip identifier as name + - Allocates and registers devlink instance + - Adds to global shared instances list + - Increments reference count + +4. **Set nested devlink instance** for the PF devlink instance using + ``devl_nested_devlink_set()`` before registering the PF devlink instance + +Cleanup Flow +------------ + +1. **Cleanup** when PF is removed +2. **Call** ``devlink_shd_put()`` to release reference (decrements reference count) +3. **Shared instance is automatically destroyed** when the last PF removes (device list becomes empty) + +Chip Identification +------------------- + +PFs belonging to the same chip are identified using a driver-specific method. +The driver is free to choose any identifier that is suitable for determining +whether two PFs are part of the same device. Examples include: + +* **PCI VPD serial numbers**: Extract from PCI VPD +* **Device tree properties**: Read chip identifier from device tree +* **Other hardware-specific identifiers**: Any unique identifier that groups PFs by chip + +Locking +------- + +A global mutex (``shd_mutex``) protects the shared instances list during +registration/deregistration. + +Similarly to other nested devlink instance relationships, devlink lock of +the shared instance should be always taken after the devlink lock of PF. + +Reference Counting +------------------ + +Each shared devlink instance maintains a reference count (``refcount_t refcount``). +The reference count is incremented when ``devlink_shd_get()`` is called and +decremented when ``devlink_shd_put()`` is called. When the reference count +reaches zero, the shared instance is automatically destroyed.
Documentation/networking/devlink/index.rst +1 -0
@@ -68,6 +68,7 @@ general. devlink-resource devlink-selftests devlink-trap + devlink-shared Driver-specific documentation -----------------------------
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 2/15 devlink: introduce shared devlink instance for PFs on same chip
In reply to Tariq ToukanFrom: Jiri Pirko <[email protected]>
Multiple PFs may reside on the same physical chip, running a single firmware. Some of the resources and configurations may be shared among these PFs. Currently, there is no good object to pin the configuration knobs on.
Introduce a shared devlink instance, instantiated upon probe of the first PF and removed during remove of the last PF. The shared devlink instance is backed by a faux device, as there is no PCI device related to it. The implementation uses reference counting to manage the
lifecycle: each PF that probes calls devlink_shd_get() to get or create the shared instance, and calls devlink_shd_put() when it removes. The shared instance is automatically destroyed when the last PF removes.
Example:
$ devlink dev
pci/0000:08:00.0:
nested_devlink:
auxiliary/mlx5_core.eth.0
faux/mlx5_core_83013c12b77faa1a30000c82a1045c91:
nested_devlink:
pci/0000:08:00.0
pci/0000:08:00.1
auxiliary/mlx5_core.eth.0
pci/0000:08:00.1:
nested_devlink:
auxiliary/mlx5_core.eth.1
auxiliary/mlx5_core.eth.1--- include/net/devlink.h | 6 ++ net/devlink/Makefile | 2 +- net/devlink/sh_dev.c | 163 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 170 insertions(+), 1 deletion(-) create mode 100644 net/devlink/sh_dev.c
include/net/devlink.h +6 -0
@@ -1644,6 +1644,12 @@ void devlink_register(struct devlink *devlink);void devlink_unregister (struct devlink *devlink);void devlink_free (struct devlink *devlink); +struct devlink *devlink_shd_get (const char *id, +const struct devlink_ops *ops, +size_t priv_size); +void devlink_shd_put (struct devlink *devlink); +void *devlink_shd_get_priv (struct devlink *devlink); +/** * struct devlink_port_ops - Port operations * @port_split: Callback used to split the port into multiple ones.
net/devlink/Makefile +1 -1
@@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-y := core.o netlink.o netlink_gen.o dev.o port.o sb.o dpipe.o \ - resource.o param.o region.o health.o trap.o rate.o linecard.o + resource.o param.o region.o health.o trap.o rate.o linecard.o sh_dev.o
net/devlink/sh_dev.c new file +163 -0
163 lines not on this page · raw message
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 3/15 devlink: Reverse locking order for nested instances
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Commit [1] defined the locking expectations for nested devlink
instances: the nested-in devlink instance lock needs to be acquired before the nested devlink instance lock. The code handling devlink rels was architected with that assumption in mind.
There are no actual users of double locking yet but that is about to change in the upcoming patches in the series.
Code operating on nested devlink instances will require also obtaining the nested-in instance lock, but such code may already be called from a variety of places with the nested devlink instance lock. Then, there's no way to acquire the nested-in lock other than making sure that all callers acquire it first.
Reversing the nested lock order allows incrementally acquiring the nested-in instance lock when needed (perhaps even a chain of locks up to the root) without affecting any caller.
The only affected use of nesting is devlink_nl_nested_fill(), which iterates over nested devlink instances with the RCU lock, without locking them, so there's no possibility of deadlock.
So this commit just updates a comment regarding the nested locks.
[1] commit c137743bce02b ("devlink: introduce object and nested devlink relationship infra")
--- net/devlink/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
net/devlink/core.c +1 -3
@@ -178,9 +178,7 @@ int devlink_rel_nested_in_add(u32 *rel_index, u32 devlink_index,* a notification of a change of this object should be sent * over netlink. The parent devlink instance lock needs to be * taken during the notification preparation. -* However, since the devlink lock of nested instance is held here, -* we would end with wrong devlink instance lock ordering and -* deadlock. Therefore the work is utilized to avoid that. +* Since the parent may or may not be locked, 'work' is utilized. */ void devlink_rel_nested_in_notify (struct devlink *devlink) {
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 4/15 devlink: Add helpers to lock nested-in instances
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Upcoming code will need to obtain a reference to locked nested-in devlink instances. Add helpers to lock, obtain an already locked reference and unlock/unref the nested-in instance.
--- net/devlink/core.c | 42 +++++++++++++++++++++++++++++++++++++ net/devlink/devl_internal.h | 3 +++ 2 files changed, 45 insertions(+)
net/devlink/core.c +42 -0
@@ -67,6 +67,48 @@ static void __devlink_rel_put(struct devlink_rel *rel)devlink_rel_free (rel); } +struct devlink *devlink_nested_in_get_lock (struct devlink_rel *rel) +{ +struct devlink *devlink; + +if (!rel) +return NULL ; + devlink =devlinks_xa_get (rel->nested_in.devlink_index); +if (!devlink) +return NULL ; +devl_lock (devlink); +if (devl_is_registered (devlink)) +return devlink; +devl_unlock (devlink); +devlink_put (devlink); +return NULL ; +} + +/* Returns the nested in devlink object and validates its lock is held. */ +struct devlink *devlink_nested_in_get_locked (struct devlink_rel *rel) +{ +struct devlink *devlink; +unsigned long index; + +if (!rel) +return NULL ; + index = rel->nested_in.devlink_index; + devlink =xa_find (&devlinks, &index, index,DEVLINK_REGISTERED ); +if (devlink) +devl_assert_locked (devlink); +return devlink; +} + +void devlink_nested_in_put_unlock (struct devlink_rel *rel) +{ +struct devlink *devlink =devlink_nested_in_get_locked (rel); + +if (devlink) { +devl_unlock (devlink); +devlink_put (devlink); + } +} +static void devlink_rel_nested_in_notify_work (struct work_struct *work) { struct devlink_rel *rel = container_of(work, struct devlink_rel,
net/devlink/devl_internal.h +3 -0
@@ -120,6 +120,9 @@ typedef void devlink_rel_notify_cb_t(struct devlink *devlink, u32 obj_index);typedef void devlink_rel_cleanup_cb_t (struct devlink *devlink,u32 obj_index,u32 rel_index); +struct devlink *devlink_nested_in_get_lock (struct devlink_rel *rel); +struct devlink *devlink_nested_in_get_locked (struct devlink_rel *rel); +void devlink_nested_in_put_unlock (struct devlink_rel *rel);void devlink_rel_nested_in_clear (u32 rel_index); int devlink_rel_nested_in_add(u32 *rel_index, u32 devlink_index, u32 obj_index, devlink_rel_notify_cb_t *notify_cb,
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 5/15 devlink: Refactor devlink_rate_nodes_check
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
devlink_rate_nodes_check() was used to verify there are no devlink rate nodes created when switching the esw mode.
Rate management code is about to become more complex, so refactor this function:
- remove unused param 'mode'.
- add a new 'rate_filter' param.
- rename to devlink_rates_check().
- expose devlink_rate_is_node() to be used as a rate filter.
This makes it more usable from multiple places, so use it from those places as well.
--- net/devlink/core.c | 2 +- net/devlink/dev.c | 7 ++++--- net/devlink/devl_internal.h | 6 ++++-- net/devlink/rate.c | 13 +++++++------ 4 files changed, 16 insertions(+), 12 deletions(-)
net/devlink/core.c +1 -1
@@ -517,7 +517,7 @@ void devlink_free(struct devlink *devlink)WARN_ON (!list_empty (&devlink->resource_list));WARN_ON (!list_empty (&devlink->dpipe_table_list));WARN_ON (!list_empty (&devlink->sb_list)); -WARN_ON (!list_empty (&devlink->rate_list)); +WARN_ON (devlink_rates_check (devlink,NULL ,NULL ));WARN_ON (!list_empty (&devlink->linecard_list));WARN_ON (!xa_empty (&devlink->ports));
net/devlink/dev.c +4 -3
@@ -434,7 +434,7 @@ static void devlink_reload_reinit_sanity_check(struct devlink *devlink)WARN_ON (!list_empty (&devlink->trap_list));WARN_ON (!list_empty (&devlink->dpipe_table_list));WARN_ON (!list_empty (&devlink->sb_list)); -WARN_ON (!list_empty (&devlink->rate_list)); +WARN_ON (devlink_rates_check (devlink,NULL ,NULL ));WARN_ON (!list_empty (&devlink->linecard_list));WARN_ON (!xa_empty (&devlink->ports)); }
@@ -713,10 +713,11 @@ int devlink_nl_eswitch_set_doit(struct sk_buff *skb, struct genl_info *info)if (info->attrs[DEVLINK_ATTR_ESWITCH_MODE ]) {if (!ops->eswitch_mode_set)return -EOPNOTSUPP ; - mode =nla_get_u16 (info->attrs[DEVLINK_ATTR_ESWITCH_MODE ]); - err =devlink_rate_nodes_check (devlink, mode, info->extack); + err =devlink_rates_check (devlink, devlink_rate_is_node, + info->extack);if (err)return err; + mode =nla_get_u16 (info->attrs[DEVLINK_ATTR_ESWITCH_MODE ]); err = ops->eswitch_mode_set (devlink, mode, info->extack);if (err)return err;
net/devlink/devl_internal.h +4 -2
@@ -300,8 +300,10 @@ int devlink_resources_validate(struct devlink *devlink, struct genl_info *info);/* Rates */ -int devlink_rate_nodes_check (struct devlink *devlink,u16 mode, -struct netlink_ext_ack *extack); +bool devlink_rate_is_node (const struct devlink_rate *devlink_rate); +int devlink_rates_check (struct devlink *devlink, +bool (*rate_filter)(const struct devlink_rate *), +struct netlink_ext_ack *extack);/* Linecards */ unsigned int devlink_linecard_index (struct devlink_linecard *linecard);
net/devlink/rate.c +7 -6
@@ -12,8 +12,7 @@ devlink_rate_is_leaf(struct devlink_rate *devlink_rate)return devlink_rate->type ==DEVLINK_RATE_TYPE_LEAF ; } -static inline bool -devlink_rate_is_node (struct devlink_rate *devlink_rate) +bool devlink_rate_is_node (const struct devlink_rate *devlink_rate) {return devlink_rate->type ==DEVLINK_RATE_TYPE_NODE ; }
@@ -688,14 +687,16 @@ int devlink_nl_rate_del_doit(struct sk_buff *skb, struct genl_info *info)return err; } -int devlink_rate_nodes_check (struct devlink *devlink,u16 mode, -struct netlink_ext_ack *extack) +int devlink_rates_check (struct devlink *devlink, +bool (*rate_filter)(const struct devlink_rate *), +struct netlink_ext_ack *extack) {struct devlink_rate *devlink_rate;list_for_each_entry (devlink_rate, &devlink->rate_list, list) -if (devlink_rate_is_node (devlink_rate)) { -NL_SET_ERR_MSG (extack,"Rate node(s) exists." ); +if (!rate_filter ||rate_filter (devlink_rate)) { +if (extack) +NL_SET_ERR_MSG (extack,"Rate node(s) exists." );return -EBUSY ; }return 0 ;
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 6/15 devlink: Decouple rate storage from associated devlink object
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Devlink rate leafs and nodes were stored in their respective devlink objects pointed to by devlink_rate->devlink.
This patch removes that association by introducing the concept of 'rate node devlink', which is where all rates that could link to each other are stored. For now this is the same as devlink_rate->devlink.
After this patch, the devlink rates stored in this devlink instance could potentially be from multiple other devlink instances. So all rate node manipulation code was updated to:
- correctly compare the actual devlink object during iteration.
- maybe acquire additional locks (noop for now).
--- include/net/devlink.h | 2 + net/devlink/rate.c | 192 +++++++++++++++++++++++++++++++----------- 2 files changed, 144 insertions(+), 50 deletions(-)
include/net/devlink.h +2 -0
@@ -1820,6 +1820,8 @@ void devlink_port_attrs_pci_sf_set(struct devlink_port *devlink_port, bool external);int devl_port_fn_devlink_set (struct devlink_port *devlink_port,struct devlink *fn_devlink); +struct devlink *devl_rate_lock (struct devlink *devlink); +void devl_rate_unlock (struct devlink *devlink);struct devlink_rate *devl_rate_node_create (struct devlink *devlink,void *priv,char *node_name,struct devlink_rate *parent);
net/devlink/rate.c +142 -50
401 lines not on this page · raw message
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [net-next,V5,06/15] devlink: Decouple rate storage from associated devlink object
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
For local reproduction steps see https://netdev-ai.bots.linux.dev/ai-local.html
---
devlink: Decouple rate storage from associated devlink object
This patch introduces the concept of a "rate node devlink" to allow rates from multiple devlink instances to be stored together. It updates all rate node manipulation code to filter by devlink during iteration and adds lock/unlock helpers (currently no-ops) for future locking support.
Quoted 4 lines
net/devlink/rate.c +0
-0
[ ... ]
Quoted 15 lines
net/devlink/rate.c +3
-1@@ -209,10 +235,12 @@ devlink_nl_rate_get_dump_one(struct sk_buff *msg, struct devlink *devlink, {struct devlink_nl_dump_state *state =devlink_dump_state (cb);struct devlink_rate *devlink_rate; +struct devlink *rate_devlink;int idx =0 ;int err =0 ; -list_for_each_entry (devlink_rate, &devlink->rate_list, list) { + rate_devlink =devl_rate_lock (devlink); +list_for_each_entry (devlink_rate, &rate_devlink->rate_list, list) {enum devlink_command cmd =DEVLINK_CMD_RATE_NEW ;u32 id =NETLINK_CB (cb->skb).portid;
net/devlink/rate.c devlink_nl_rate_get_dump_one(struct sk_buff *msg, struct devlink *devlink, · line 252
Quoted 5 lines
net/devlink/rate.c +2
-0@@ -220,6 +248,9 @@ devlink_nl_rate_get_dump_one(struct sk_buff *msg, struct devlink *devlink, idx++;continue ; } +if (devlink_rate->devlink != devlink) +continue ;
When this filter skips a non-matching entry, idx is not incremented. However, state->idx is used to track position across netlink dump resumptions and expects idx to reflect the actual position in the list.
Once cross-device rate nodes are enabled (later in this series), the list can contain entries from multiple devlinks. If the dump requires multiple netlink messages and non-matching entries exist between matching ones, the idx tracking becomes inconsistent with the list position. This can cause entries to be reported multiple times on dump resumption.
Should this continue statement also increment idx to maintain correct position tracking?
Quoted 9 lines
net/devlink/rate.c +2
-0+ err =devlink_nl_rate_fill (msg, devlink_rate, cmd, id, cb->nlh->nlmsg_seq, flags,NULL );if (err) {@@ -228,6 +259,7 @@ devlink_nl_rate_get_dump_one(struct sk_buff *msg, struct devlink *devlink, } idx++; } +devl_rate_unlock (devlink);
[ ... ]
Signature · 1 line
pw-bot: cr
Re: [net-next,V5,06/15] devlink: Decouple rate storage from associated devlink object
Quoted 23 lines · Jakub Kicinski
net/devlink/rate.c devlink_nl_rate_get_dump_one(struct sk_buff · line 253
Quoted 7 lines
net/devlink/rate.c +2
-0@@ -220,6 +248,9 @@ devlink_nl_rate_get_dump_one(struct sk_buff *msg, struct devlink *devlink, idx++; continue; } +if (devlink_rate->devlink != devlink) +continue ;When this filter skips a non-matching entry, idx is not incremented. However,
state->idxis used to track position across netlink dump resumptions and expects idx to reflect the actual position in the list.Once cross-device rate nodes are enabled (later in this series), the list
can contain entries from multiple devlinks. If the dump requires multiple
netlink messages and non-matching entries exist between matching ones,
the idx tracking becomes inconsistent with the list position. This can
cause entries to be reported multiple times on dump resumption.
Quoted 2 lines
Should this continue statement also increment idx to maintain correct position tracking?
Another useful finding. I suspect the code might work as-is, since state->idx tracks "the next pending item for 'devlink'" from the list, but it's quite hard to reason about that and it's probably better to do as the tool suggests, and track the list position as now. Will do so in the next version.
Cosmin.
Tariq Toukan
maintainerPATCH v5 7/15 devlink: Add parent dev to devlink API
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Upcoming changes to the rate commands need the parent devlink specified. This change adds a nested 'parent-dev' attribute to the API and helpers to obtain and put a reference to the parent devlink instance in info->user_ptr[1].
To avoid deadlocks, the parent devlink is unlocked before obtaining the main devlink instance that is the target of the request.
A reference to the parent is kept until the end of the request to avoid it suddenly disappearing.
This means that this reference is of limited use without additional protection.
--- Documentation/netlink/specs/devlink.yaml | 12 +++++ include/uapi/linux/devlink.h | 2 + net/devlink/devl_internal.h | 2 + net/devlink/netlink.c | 67 ++++++++++++++++++++++-- net/devlink/netlink_gen.c | 5 ++ net/devlink/netlink_gen.h | 8 +++ 6 files changed, 91 insertions(+), 5 deletions(-)
Documentation/netlink/specs/devlink.yaml +12 -0
@@ -867,6 +867,10 @@ attribute-sets: type: flag doc: Request restoring parameter to its default value. value: 183 + + - name: parent-dev + type: nest + nested-attributes: dl-parent-dev - name: dl-dev-stats subset-of: devlink
@@ -1289,6 +1293,14 @@ attribute-sets: Specifies the bandwidth share assigned to the Traffic Class. The bandwidth for the traffic class is determined in proportion to the sum of the shares of all configured classes. + - + name: dl-parent-dev + subset-of: devlink + attributes: + - + name: bus-name + - + name: dev-name operations: enum-model: directional
include/uapi/linux/devlink.h +2 -0
@@ -642,6 +642,8 @@ enum devlink_attr {DEVLINK_ATTR_PARAM_VALUE_DEFAULT ,/* dynamic */ DEVLINK_ATTR_PARAM_RESET_DEFAULT ,/* flag */ +DEVLINK_ATTR_PARENT_DEV ,/* nested */ +/* Add new attributes above here, update the spec in * Documentation/netlink/specs/devlink.yaml and re-generate * net/devlink/netlink_gen.c.
net/devlink/devl_internal.h +2 -0
@@ -162,6 +162,8 @@ typedef int devlink_nl_dump_one_func_t(struct sk_buff *msg,struct devlink *devlink_get_from_attrs_lock (struct net *net,struct nlattr **attrs,bool dev_lock); +struct devlink * +devlink_get_parent_from_attrs_lock (struct net *net,struct nlattr **attrs);int devlink_nl_dumpit (struct sk_buff *msg,struct netlink_callback *cb,devlink_nl_dump_one_func_t *dump_one);
net/devlink/netlink.c +62 -5
111 lines not on this page · raw message
net/devlink/netlink_gen.c +5 -0
@@ -39,6 +39,11 @@ devlink_attr_param_type_validate(const struct nlattr *attr, }/* Common nested types */ +const struct nla_policy devlink_dl_parent_dev_nl_policy[DEVLINK_ATTR_DEV_NAME +1 ] = { + [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, + [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, +}; +const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ] = { [DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ] = { .type =NLA_BINARY , }, [DEVLINK_PORT_FN_ATTR_STATE ] =NLA_POLICY_MAX (NLA_U8 ,1 ),
net/devlink/netlink_gen.h +8 -0
@@ -13,6 +13,7 @@#include <uapi/linux/devlink.h> /* Common nested types */ +extern const struct nla_policy devlink_dl_parent_dev_nl_policy[DEVLINK_ATTR_DEV_NAME +1 ];extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ];extern const struct nla_policy devlink_dl_rate_tc_bws_nl_policy[DEVLINK_RATE_TC_ATTR_BW +1 ];extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ];
@@ -29,12 +30,19 @@ int devlink_nl_pre_doit_dev_lock(const struct genl_split_ops *ops,int devlink_nl_pre_doit_port_optional (const struct genl_split_ops *ops,struct sk_buff *skb,struct genl_info *info); +int devlink_nl_pre_doit_parent_dev_optional (const struct genl_split_ops *ops, +struct sk_buff *skb, +struct genl_info *info);void devlink_nl_post_doit (const struct genl_split_ops *ops,struct sk_buff *skb,struct genl_info *info);void devlink_nl_post_doit_dev_lock (const struct genl_split_ops *ops,struct sk_buff *skb,struct genl_info *info); +void +devlink_nl_post_doit_parent_dev_optional (const struct genl_split_ops *ops, +struct sk_buff *skb, +struct genl_info *info);int devlink_nl_get_doit (struct sk_buff *skb,struct genl_info *info);int devlink_nl_get_dumpit (struct sk_buff *skb,struct netlink_callback *cb);
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 8/15 devlink: Allow parent dev for rate-set and rate-new
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Currently, a devlink rate's parent device is assumed to be the same as the one where the devlink rate is created.
This patch changes that to allow rate commands to accept an additional argument that specifies the parent dev. This will allow devlink rate groups with leafs from other devices.
Example of the new usage with ynl:
Creating a group on pci/0000:08:00.1 with a parent to an already existing pci/0000:08:00.1/group1:./tools/net/ynl/pyynl/cli.py --spec \
Documentation/netlink/specs/devlink.yaml --do rate-new --json '{
"bus-name": "pci",
"dev-name": "0000:08:00.1",
"rate-node-name": "group2",
"rate-parent-node-name": "group1",
"parent-dev": {
"bus-name": "pci",
"dev-name": "0000:08:00.1"
}}'
Setting the parent of leaf node pci/0000:08:00.1/65537 to pci/0000:08:00.0/group1:./tools/net/ynl/pyynl/cli.py --spec \
Documentation/netlink/specs/devlink.yaml --do rate-set --json '{
"bus-name": "pci",
"dev-name": "0000:08:00.1",
"port-index": 65537,
"parent-dev": {
"bus-name": "pci",
"dev-name": "0000:08:00.0"
},
"rate-parent-node-name": "group1"}'
--- Documentation/netlink/specs/devlink.yaml | 10 ++++++---- net/devlink/netlink_gen.c | 18 ++++++++++-------- 2 files changed, 16 insertions(+), 12 deletions(-)
Documentation/netlink/specs/devlink.yaml +6 -4
@@ -2218,8 +2218,8 @@ operations: dont-validate: [strict] flags: [admin-perm] do: - pre: devlink-nl-pre-doit - post: devlink-nl-post-doit + pre: devlink-nl-pre-doit-parent-dev-optional + post: devlink-nl-post-doit-parent-dev-optional request: attributes: - bus-name
@@ -2231,6 +2231,7 @@ operations: - rate-tx-weight - rate-parent-node-name - rate-tc-bws + - parent-dev - name: rate-new
@@ -2239,8 +2240,8 @@ operations: dont-validate: [strict] flags: [admin-perm] do: - pre: devlink-nl-pre-doit - post: devlink-nl-post-doit + pre: devlink-nl-pre-doit-parent-dev-optional + post: devlink-nl-post-doit-parent-dev-optional request: attributes: - bus-name
@@ -2252,6 +2253,7 @@ operations: - rate-tx-weight - rate-parent-node-name - rate-tc-bws + - parent-dev - name: rate-del
net/devlink/netlink_gen.c +10 -8
@@ -536,7 +536,7 @@ static const struct nla_policy devlink_rate_get_dump_nl_policy[DEVLINK_ATTR_DEV_ };/* DEVLINK_CMD_RATE_SET - do */ -static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TC_BWS +1 ] = { +static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_PARENT_DEV +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , },
@@ -546,10 +546,11 @@ static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TC_B [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), + [DEVLINK_ATTR_PARENT_DEV] = NLA_POLICY_NESTED(devlink_dl_parent_dev_nl_policy), };/* DEVLINK_CMD_RATE_NEW - do */ -static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TC_BWS +1 ] = { +static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_PARENT_DEV +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , },
@@ -559,6 +560,7 @@ static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TC_B [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), + [DEVLINK_ATTR_PARENT_DEV] = NLA_POLICY_NESTED(devlink_dl_parent_dev_nl_policy), };/* DEVLINK_CMD_RATE_DEL - do */
@@ -1202,21 +1204,21 @@ const struct genl_split_ops devlink_nl_ops[74] = { { .cmd = DEVLINK_CMD_RATE_SET, .validate = GENL_DONT_VALIDATE_STRICT, - .pre_doit = devlink_nl_pre_doit, + .pre_doit = devlink_nl_pre_doit_parent_dev_optional, .doit = devlink_nl_rate_set_doit, - .post_doit = devlink_nl_post_doit, + .post_doit = devlink_nl_post_doit_parent_dev_optional, .policy = devlink_rate_set_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TC_BWS, + .maxattr = DEVLINK_ATTR_PARENT_DEV, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, { .cmd = DEVLINK_CMD_RATE_NEW, .validate = GENL_DONT_VALIDATE_STRICT, - .pre_doit = devlink_nl_pre_doit, + .pre_doit = devlink_nl_pre_doit_parent_dev_optional, .doit = devlink_nl_rate_new_doit, - .post_doit = devlink_nl_post_doit, + .post_doit = devlink_nl_post_doit_parent_dev_optional, .policy = devlink_rate_new_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TC_BWS, + .maxattr = DEVLINK_ATTR_PARENT_DEV, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, {
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 9/15 devlink: Allow rate node parents from other devlinks
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
This commit makes use of the building blocks previously added to implement cross-device rate nodes.
A new 'supported_cross_device_rate_nodes' bool is added to devlink_ops which lets drivers advertise support for cross-device rate objects. If enabled and if there is a common shared devlink instance, then:
- all rate objects will be stored in the top-most common nested instance and
- rate objects can have parents from other devices sharing the same common instance.
The parent devlink from info->user_ptr[1] is not locked, so none of its mutable fields can be used. But parent setting only requires comparing devlink pointer comparisons. Additionally, since the shared devlink is locked, other rate operations cannot concurrently happen.
The rate lock/unlock functions are now exported, so that drivers implementing this can protect against concurrent modifications on any shared device structures.
--- .../networking/devlink/devlink-port.rst | 2 + include/net/devlink.h | 5 + net/devlink/rate.c | 91 +++++++++++++++++-- 3 files changed, 90 insertions(+), 8 deletions(-)
Documentation/networking/devlink/devlink-port.rst +2 -0
@@ -417,6 +417,8 @@ API allows to configure following rate object's parameters: Parent node name. Parent node rate limits are considered as additional limits to all node children limits. ``tx_max`` is an upper limit for children. ``tx_share`` is a total bandwidth distributed among children. + If the device supports cross-function scheduling, the parent can be from a + different function of the same underlying device. ``tc_bw`` Allow users to set the bandwidth allocation per traffic class on rate
include/net/devlink.h +5 -0
@@ -1585,6 +1585,11 @@ struct devlink_ops {struct devlink_rate *parent, void *priv_child, void *priv_parent, struct netlink_ext_ack *extack); +/* Indicates if cross-device rate nodes are supported. +* This also requires a shared common ancestor object all devices that +* could share rate nodes are nested in. +*/ +bool supported_cross_device_rate_nodes;/** * selftests_check() - queries if selftest is supported * @devlink: devlink instance
net/devlink/rate.c +83 -8
151 lines not on this page · raw message
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 10/15 net/mlx5: Add a shared devlink instance for PFs on same chip
In reply to Tariq ToukanFrom: Jiri Pirko <[email protected]>
Use the previously introduced shared devlink infrastructure to create a shared devlink instance for mlx5 PFs that reside on the same physical chip. The shared instance is identified by the chip's serial number extracted from PCI VPD (V3 keyword, with fallback to serial number for older devices).
Each PF that probes calls mlx5_shd_init() which extracts the chip serial number and uses devlink_shd_get() to get or create the shared instance. When a PF removes, mlx5_shd_uninit() calls devlink_shd_put() to release the reference. The shared instance is automatically destroyed when the last PF removes.
Make the PF devlink instances nested in this shared devlink instance, allowing userspace to identify which PFs belong to the same physical chip.
Example:
$ devlink dev
pci/0000:08:00.0:
nested_devlink:
auxiliary/mlx5_core.eth.0
faux/mlx5_core_83013c12b77faa1a30000c82a1045c91:
nested_devlink:
pci/0000:08:00.0
pci/0000:08:00.1
auxiliary/mlx5_core.eth.0
pci/0000:08:00.1:
nested_devlink:
auxiliary/mlx5_core.eth.1
auxiliary/mlx5_core.eth.1--- .../net/ethernet/mellanox/mlx5/core/Makefile | 5 +- .../net/ethernet/mellanox/mlx5/core/main.c | 17 +++++ .../ethernet/mellanox/mlx5/core/sh_devlink.c | 63 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sh_devlink.h | 12 ++++ include/linux/mlx5/driver.h | 1 + 5 files changed, 96 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +3 -2
@@ -16,8 +16,9 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \ fs_counters.o fs_ft_pool.o rl.o lag/debugfs.o lag/lag.o dev.o events.o wq.o lib/gid.o \ lib/devcom.o lib/pci_vsc.o lib/dm.o lib/fs_ttc.o diag/fs_tracepoint.o \ - diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o diag/reporter_vnic.o \ - fw_reset.o qos.o lib/tout.o lib/aso.o wc.o fs_pool.o lib/nv_param.o + diag/fw_tracer.o diag/crdump.o devlink.o sh_devlink.o diag/rsc_dump.o \ + diag/reporter_vnic.o fw_reset.o qos.o lib/tout.o lib/aso.o wc.o fs_pool.o \ + lib/nv_param.o # # Netdev basic
drivers/net/ethernet/mellanox/mlx5/core/main.c +17 -0
@@ -74,6 +74,7 @@#include "mlx5_irq.h" #include "hwmon.h" #include "lag/lag.h" +#include "sh_devlink.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -1520,10 +1521,16 @@ int mlx5_init_one(struct mlx5_core_dev *dev)int err;devl_lock (devlink); +if (dev->shd) { + err =devl_nested_devlink_set (dev->shd, devlink); +if (err) + goto unlock; + }devl_register (devlink); err =mlx5_init_one_devl_locked (dev);if (err)devl_unregister (devlink); +unlock:devl_unlock (devlink);return err; }
@@ -2015,6 +2022,13 @@ static int probe_one(struct pci_dev *pdev, const struct pci_device_id *id) goto pci_init_err; } + err =mlx5_shd_init (dev); +if (err) { +mlx5_core_err (dev,"mlx5_shd_init failed with error code %d\n" , + err); + goto shd_init_err; + } + err =mlx5_init_one (dev);if (err) { mlx5_core_err(dev, "mlx5_init_one failed with error code %d\n",
@@ -2026,6 +2040,8 @@ static int probe_one(struct pci_dev *pdev, const struct pci_device_id *id)return 0 ; err_init_one: +mlx5_shd_uninit (dev); +shd_init_err:mlx5_pci_close (dev); pci_init_err:mlx5_mdev_uninit (dev);
@@ -2047,6 +2063,7 @@ static void remove_one(struct pci_dev *pdev)mlx5_drain_health_wq (dev);mlx5_sriov_disable (pdev, false);mlx5_uninit_one (dev); +mlx5_shd_uninit (dev);mlx5_pci_close (dev);mlx5_mdev_uninit (dev);mlx5_adev_idx_free (dev->priv.adev_idx);
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +63 -0
63 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5_SH_DEVLINK_H__ +#define __MLX5_SH_DEVLINK_H__ + +#include <linux/mlx5/driver.h> + +int mlx5_shd_init (struct mlx5_core_dev *dev); +void mlx5_shd_uninit (struct mlx5_core_dev *dev); + +#endif /* __MLX5_SH_DEVLINK_H__ */
include/linux/mlx5/driver.h +1 -0
@@ -798,6 +798,7 @@ struct mlx5_core_dev {enum mlx5_wc_state wc_state;/* sync write combining state */ struct mutex wc_state_lock; +struct devlink *shd; };struct mlx5_db {
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [net-next,V5,10/15] net/mlx5: Add a shared devlink instance for PFs on same chip
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
For local reproduction steps see https://netdev-ai.bots.linux.dev/ai-local.html
---
net/mlx5: Add a shared devlink instance for PFs on same chip
Creates shared devlink infrastructure for mlx5 PFs on the same physical chip, identified by chip serial number from PCI VPD.
Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +0
-0
[ ... ]
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c line 14
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +3
-0+int mlx5_shd_init (struct mlx5_core_dev *dev) +{ + u8 *vpd_data __free(kfree) = NULL;
The __free(kfree) annotation here combined with the early return below looks problematic.
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +6
-0+struct pci_dev *pdev = dev->pdev; +unsigned int vpd_size, kw_len; +struct devlink *devlink; +const char *sn; +char *end; +int start;
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c line 30
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +10
-0+int err; + +if (!mlx5_core_is_pf (dev)) +return 0 ; + + vpd_data =pci_vpd_alloc (pdev, &vpd_size); +if (IS_ERR (vpd_data)) { + err =PTR_ERR (vpd_data); +return err == -ENODEV ?0 : err; + }
When pci_vpd_alloc() returns an ERR_PTR (e.g., -EIO or -ENOMEM) and the function returns here, does the __free(kfree) cleanup call kfree() on an ERR_PTR value? The kfree() implementation only handles NULL and ZERO_SIZE_PTR safely via the ZERO_OR_NULL_PTR() check - ERR_PTR values are large unsigned values that would pass this check and proceed to virt_to_page() on an invalid address.
This same issue was raised by Krzysztof Kozlowski in review of V4:
https://lore.kernel.org/netdev/[email protected]/
To fix this, either set vpd_data = NULL before the early return, or remove the __free(kfree) annotation and handle cleanup manually.
[ ... ]
Jakub Kicinski
maintainerRe: [net-next,V5,10/15] net/mlx5: Add a shared devlink instance for PFs on same chip
Quoted 6 lines · On Wed, 21 Jan 2026 19:39:59 -0800 Jakub Kicinski wrote:
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c line 14
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +3
-0+int mlx5_shd_init (struct mlx5_core_dev *dev) +{ + u8 *vpd_data __free(kfree) = NULL;The
__free(kfree)annotation here combined with the early return below looks problematic.
__free() should be considered banned for netdev.
Please.
It clearly adds more bugs than it fixes.
Re: [PATCH net-next V5 10/15] net/mlx5: Add a shared devlink instance for PFs on same chip
In reply to Tariq Toukandrivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c line 14
Quoted 7 lines · Tariq Toukan
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +7
-0+ +static const struct devlink_ops mlx5_shd_ops = { +}; + +int mlx5_shd_init (struct mlx5_core_dev *dev) +{ + u8 *vpd_data __free(kfree) = NULL;
So you just ignore people's comments. What is the point of review then?
Please respond to review and do not send the same buggy or wrong code.
Best regards,
Krzysztof
Re: [PATCH net-next V5 10/15] net/mlx5: Add a shared devlink instance for PFs on same chip
Quoted 12 lines · Krzysztof Kozlowski
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c line 14
Quoted 7 lines · Tariq Toukan
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c new file +7
-0+ +static const struct devlink_ops mlx5_shd_ops = { +}; + +int mlx5_shd_init (struct mlx5_core_dev *dev) +{ + u8 *vpd_data __free(kfree) = NULL;So you just ignore people's comments. What is the point of review then?
Quoted 2 lines
Please respond to review and do not send the same buggy or wrong code.
Apologies, it was an oversight. I thought Jiri fixed this part and didn't think to look at this particular bit. I even added a (false) statement in the cover letter about this.
Will __really__ fix in the next version.
Cosmin.
Tariq Toukan
maintainerPATCH v5 11/15 net/mlx5: Expose a function to clear a vport's parent
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Currently, clearing a vport's parent happens with a call that looks like this:
mlx5_esw_qos_vport_update_parent(vport, NULL, NULL);
Change that to something nicer that looks like this:
mlx5_esw_qos_vport_clear_parent(vport);
--- .../ethernet/mellanox/mlx5/core/esw/devlink_port.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c | 11 +++++++++-- drivers/net/ethernet/mellanox/mlx5/core/eswitch.h | 3 +-- 3 files changed, 11 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +1 -1
@@ -202,7 +202,7 @@ void mlx5_esw_offloads_devlink_port_unregister(struct mlx5_vport *vport)return ; dl_port = vport->dl_port; -mlx5_esw_qos_vport_update_parent (vport,NULL ,NULL ); +mlx5_esw_qos_vport_clear_parent (vport);devl_rate_leaf_destroy (&dl_port->dl_port);devl_port_unregister (&dl_port->dl_port);
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +9 -2
@@ -1896,8 +1896,10 @@ int mlx5_esw_devlink_rate_node_del(struct devlink_rate *rate_node, void *priv,return 0 ; } -int mlx5_esw_qos_vport_update_parent (struct mlx5_vport *vport,struct mlx5_esw_sched_node *parent, -struct netlink_ext_ack *extack) +static int +mlx5_esw_qos_vport_update_parent (struct mlx5_vport *vport, +struct mlx5_esw_sched_node *parent, +struct netlink_ext_ack *extack) {struct mlx5_eswitch *esw = vport->dev->priv.eswitch;int err =0 ;
@@ -1922,6 +1924,11 @@ int mlx5_esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw_sreturn err; } +void mlx5_esw_qos_vport_clear_parent (struct mlx5_vport *vport) +{ +mlx5_esw_qos_vport_update_parent (vport,NULL ,NULL ); +} + int mlx5_esw_devlink_rate_leaf_parent_set(struct devlink_rate *devlink_rate, struct devlink_rate *parent, void *priv, void *parent_priv,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1 -2
@@ -452,8 +452,7 @@ int mlx5_eswitch_set_vport_trust(struct mlx5_eswitch *esw, u16 vport_num, bool setting);int mlx5_eswitch_set_vport_rate (struct mlx5_eswitch *esw,u16 vport,u32 max_rate,u32 min_rate); -int mlx5_esw_qos_vport_update_parent (struct mlx5_vport *vport,struct mlx5_esw_sched_node *node, -struct netlink_ext_ack *extack); +void mlx5_esw_qos_vport_clear_parent (struct mlx5_vport *vport);int mlx5_eswitch_set_vepa (struct mlx5_eswitch *esw,u8 setting);int mlx5_eswitch_get_vepa (struct mlx5_eswitch *esw,u8 *setting); int mlx5_eswitch_get_vport_config(struct mlx5_eswitch *esw,
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [net-next,V5,11/15] net/mlx5: Expose a function to clear a vport's parent
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
For local reproduction steps see https://netdev-ai.bots.linux.dev/ai-local.html
---
net/mlx5e: Skip ESN replay window setup for IPsec crypto offload
This patch fixes a WARN_ON being triggered for crypto offload by adding a check to skip the replay window setup when the offload type is not XFRM_DEV_OFFLOAD_PACKET.
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c void mlx5e_ipsec_build_accel_xfrm_attrs(struct mlx5e_ipsec_sa_entry *sa_entry, · line 435
Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c +2
-1@@ -430,7 +430,8 @@ void mlx5e_ipsec_build_accel_xfrm_attrs(struct mlx5e_ipsec_sa_entry *sa_entry, attrs->replay_esn.esn = sa_entry->esn_state.esn; attrs->replay_esn.esn_msb = sa_entry->esn_state.esn_msb; attrs->replay_esn.overlap = sa_entry->esn_state.overlap; -if (attrs->dir ==XFRM_DEV_OFFLOAD_OUT ) +if (attrs->dir ==XFRM_DEV_OFFLOAD_OUT || + attrs->type !=XFRM_DEV_OFFLOAD_PACKET ) goto skip_replay_window;
At this point in mlx5e_ipsec_build_accel_xfrm_attrs(), attrs->type has not been initialized yet. The function starts with memset(attrs, 0, ...) which sets attrs->type to 0 (XFRM_DEV_OFFLOAD_UNSPECIFIED).
The assignment attrs->type = x->xso.type happens later after the skip_replay_window label. This means the condition attrs->type != XFRM_DEV_OFFLOAD_PACKET evaluates as 0 != 2 which is always true.
As a result, the replay window setup will be skipped for all inbound configurations, not just crypto offload. Should this check use x->xso.type instead of attrs->type, or should attrs->type be assigned earlier in the function before this check?
Jakub Kicinski
maintainerRe: [net-next,V5,11/15] net/mlx5: Expose a function to clear a vport's parent
Quoted 28 lines · On Wed, 21 Jan 2026 19:40:01 -0800 Jakub Kicinski wrote:
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
For local reproduction steps see https://netdev-ai.bots.linux.dev/ai-local.html
---
net/mlx5e: Skip ESN replay window setup for IPsec crypto offloadThis patch fixes a
WARN_ONbeing triggered for crypto offload by adding a check to skip the replay window setup when the offload type is notXFRM_DEV_OFFLOAD_PACKET.drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c void mlx5e_ipsec_build_accel_xfrm_attrs(struct mlx5e_ipsec_sa_entry *sa_entry, · line 435
Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c +2
-1@@ -430,7 +430,8 @@ void mlx5e_ipsec_build_accel_xfrm_attrs(struct mlx5e_ipsec_sa_entry *sa_entry, attrs->replay_esn.esn = sa_entry->esn_state.esn; attrs->replay_esn.esn_msb = sa_entry->esn_state.esn_msb; attrs->replay_esn.overlap = sa_entry->esn_state.overlap; -if (attrs->dir ==XFRM_DEV_OFFLOAD_OUT ) +if (attrs->dir ==XFRM_DEV_OFFLOAD_OUT || + attrs->type !=XFRM_DEV_OFFLOAD_PACKET ) goto skip_replay_window;At this point in
mlx5e_ipsec_build_accel_xfrm_attrs(),attrs->typehas not been initialized yet. The function starts withmemset(attrs, 0, ...)which setsattrs->typeto 0 (XFRM_DEV_OFFLOAD_UNSPECIFIED).
Quoted 8 lines
The assignment
attrs->type=x->xso.typehappens later after theskip_replay_windowlabel. This means the conditionattrs->type!=XFRM_DEV_OFFLOAD_PACKETevaluates as 0 != 2 which is always true.As a result, the replay window setup will be skipped for all inbound configurations, not just crypto offload. Should this check use
x->xso.typeinstead ofattrs->type, or shouldattrs->typebe assigned earlier in the function before this check?
Herm, something misfired here, this is a review for the series of fixes posted on the same day, not for the rate scheduling.
Tariq Toukan
maintainerPATCH v5 12/15 net/mlx5: Store QoS sched nodes in the sh_devlink
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
In order to support cross-esw scheduling, the nodes for all devices need to be accessible from the same place so that normalization can work. Store them in the shared devlink instance and provide an accessor for them.
Protecting against concurrent QoS modifications is now done with the shd lock. All calls originating from devlink rate already have the shd lock held so only the additional entry points into QoS need to acquire it.
As part of this change, the E-Switch qos domain concept was removed. E-Switch QoS domains were added with the intention of eventually implementing shared qos domains to support cross-esw scheduling in the previous approach ([1]), but they are no longer necessary in the new approach.
[1] https://lore.kernel.org/netdev/[email protected]/
--- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 201 +++++------------- .../net/ethernet/mellanox/mlx5/core/esw/qos.h | 3 - .../net/ethernet/mellanox/mlx5/core/eswitch.c | 9 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 11 +- .../ethernet/mellanox/mlx5/core/sh_devlink.c | 30 ++- .../ethernet/mellanox/mlx5/core/sh_devlink.h | 2 + 6 files changed, 87 insertions(+), 169 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +49 -152
@@ -5,60 +5,16 @@#include "lib/mlx5.h" #include "esw/qos.h" #include "en/port.h" +#include "sh_devlink.h" #define CREATE_TRACE_POINTS #include "diag/qos_tracepoint.h" /* Minimum supported BW share value by the HW is 1 Mbit/sec */ #define MLX5_MIN_BW_SHARE 1 -/* Holds rate nodes associated with an E-Switch. */ -struct mlx5_qos_domain { -/* Serializes access to all qos changes in the qos domain. */ -struct mutex lock; -/* List of all mlx5_esw_sched_nodes. */ -struct list_head nodes; -}; - -static void esw_qos_lock (struct mlx5_eswitch *esw) -{ -mutex_lock (&esw->qos.domain->lock); -} - -static void esw_qos_unlock (struct mlx5_eswitch *esw) -{ -mutex_unlock (&esw->qos.domain->lock); -} -static void esw_assert_qos_lock_held (struct mlx5_eswitch *esw) { -lockdep_assert_held (&esw->qos.domain->lock); -} - -static struct mlx5_qos_domain *esw_qos_domain_alloc (void ) -{ -struct mlx5_qos_domain *qos_domain; - - qos_domain =kzalloc (sizeof (*qos_domain),GFP_KERNEL ); -if (!qos_domain) -return NULL ; - -mutex_init (&qos_domain->lock); -INIT_LIST_HEAD (&qos_domain->nodes); - -return qos_domain; -} - -static int esw_qos_domain_init (struct mlx5_eswitch *esw) -{ - esw->qos.domain =esw_qos_domain_alloc (); - -return esw->qos.domain ?0 : -ENOMEM ; -} - -static void esw_qos_domain_release (struct mlx5_eswitch *esw) -{ -kfree (esw->qos.domain); - esw->qos.domain =NULL ; +devl_assert_locked (esw->dev->shd); }enum sched_node_type {
@@ -111,7 +67,8 @@ static void esw_qos_node_attach_to_parent(struct mlx5_esw_sched_node *node)if (!node->parent) {/* Root children are assigned a depth level of 2. */ node->level =2 ; -list_add_tail (&node->entry, &node->esw->qos.domain->nodes); +list_add_tail (&node->entry, +mlx5_shd_get_qos_nodes (node->esw->dev)); }else { node->level = node->parent->level +1 ;list_add_tail (&node->entry, &node->parent->children);
@@ -324,14 +281,15 @@ static int esw_qos_create_rate_limit_element(struct mlx5_esw_sched_node *node,static u32 esw_qos_calculate_min_rate_divider (struct mlx5_eswitch *esw,struct mlx5_esw_sched_node *parent) { -struct list_head *nodes = parent ? &parent->children : &esw->qos.domain->nodes;u32 fw_max_bw_share =MLX5_CAP_QOS (esw->dev, max_tsar_bw_share);struct mlx5_esw_sched_node *node; +struct list_head *nodes;u32 max_guarantee =0 ;/* Find max min_rate across all nodes. * This will correspond to fw_max_bw_share in the final bw_share calculation. */ + nodes = parent ? &parent->children :mlx5_shd_get_qos_nodes (esw->dev);list_for_each_entry (node, nodes, entry) {if (node->esw == esw && node->ix != esw->qos.root_tsar_ix && node->min_rate > max_guarantee)
@@ -372,10 +330,11 @@ static void esw_qos_normalize_min_rate(struct mlx5_eswitch *esw,struct mlx5_esw_sched_node *parent, struct netlink_ext_ack *extack) { - struct list_head *nodes = parent ? &parent->children : &esw->qos.domain->nodes; u32 divider = esw_qos_calculate_min_rate_divider(esw, parent);struct mlx5_esw_sched_node *node; +struct list_head *nodes; + nodes = parent ? &parent->children :mlx5_shd_get_qos_nodes (esw->dev);list_for_each_entry (node, nodes, entry) {if (node->esw != esw || node->ix == esw->qos.root_tsar_ix)continue ;
@@ -715,7 +674,7 @@ __esw_qos_create_vports_sched_node(struct mlx5_eswitch *esw, struct mlx5_esw_sch goto err_alloc_node; } -list_add_tail (&node->entry, &esw->qos.domain->nodes); +list_add_tail (&node->entry,mlx5_shd_get_qos_nodes (esw->dev));esw_qos_normalize_min_rate (esw,NULL , extack);trace_mlx5_esw_node_qos_create (esw->dev, node, node->ix);
@@ -1108,7 +1067,8 @@ static int mlx5_esw_qos_vport_enable(struct mlx5_vport *vport, enum sched_node_treturn -ENOMEM ; }if (!parent) -list_add_tail (&sched_node->entry, &esw->qos.domain->nodes); +list_add_tail (&sched_node->entry, +mlx5_shd_get_qos_nodes (esw->dev)); sched_node->max_rate = max_rate; sched_node->min_rate = min_rate;
@@ -1143,7 +1103,7 @@ void mlx5_esw_qos_vport_disable(struct mlx5_vport *vport)struct mlx5_esw_sched_node *parent;lockdep_assert_held (&esw->state_lock); -esw_qos_lock (esw); +devl_lock (esw->dev->shd);if (!vport->qos.sched_node) goto unlock;
@@ -1152,7 +1112,7 @@ void mlx5_esw_qos_vport_disable(struct mlx5_vport *vport)mlx5_esw_qos_vport_disable_locked (vport); unlock: -esw_qos_unlock (esw); +devl_unlock (esw->dev->shd); } static int mlx5_esw_qos_set_vport_max_rate(struct mlx5_vport *vport, u32 max_rate,
@@ -1191,26 +1151,25 @@ int mlx5_esw_qos_set_vport_rate(struct mlx5_vport *vport, u32 max_rate, u32 min_struct mlx5_eswitch *esw = vport->dev->priv.eswitch;int err; -esw_qos_lock (esw); +devl_lock (esw->dev->shd); err =mlx5_esw_qos_set_vport_min_rate (vport, min_rate,NULL );if (!err) err =mlx5_esw_qos_set_vport_max_rate (vport, max_rate,NULL ); -esw_qos_unlock (esw); +devl_unlock (esw->dev->shd);return err; }bool mlx5_esw_qos_get_vport_rate (struct mlx5_vport *vport,u32 *max_rate,u32 *min_rate) { -struct mlx5_eswitch *esw = vport->dev->priv.eswitch;bool enabled; -esw_qos_lock (esw); +devl_lock (vport->dev->shd); enabled = !!vport->qos.sched_node;if (enabled) { *max_rate = vport->qos.sched_node->max_rate; *min_rate = vport->qos.sched_node->min_rate; } -esw_qos_unlock (esw); +devl_unlock (vport->dev->shd);return enabled; }
@@ -1576,9 +1535,9 @@ int mlx5_esw_qos_modify_vport_rate(struct mlx5_eswitch *esw, u16 vport_num, u32return err; } -esw_qos_lock (esw); +devl_lock (esw->dev->shd); err =mlx5_esw_qos_set_vport_max_rate (vport, rate_mbps,NULL ); -esw_qos_unlock (esw); +devl_unlock (esw->dev->shd);return err; }
@@ -1667,44 +1626,24 @@ static void esw_vport_qos_prune_empty(struct mlx5_vport *vport)mlx5_esw_qos_vport_disable_locked (vport); } -int mlx5_esw_qos_init (struct mlx5_eswitch *esw) -{ -if (esw->qos.domain) -return 0 ;/* Nothing to change. */ - -return esw_qos_domain_init (esw); -} - -void mlx5_esw_qos_cleanup (struct mlx5_eswitch *esw) -{ -if (esw->qos.domain) -esw_qos_domain_release (esw); -} -/* Eswitch devlink rate API */ int mlx5_esw_devlink_rate_leaf_tx_share_set (struct devlink_rate *rate_leaf,void *priv,u64 tx_share,struct netlink_ext_ack *extack) {struct mlx5_vport *vport = priv; -struct mlx5_eswitch *esw;int err; - esw = vport->dev->priv.eswitch; -if (!mlx5_esw_allowed (esw)) +if (!mlx5_esw_allowed (vport->dev->priv.eswitch))return -EPERM ; err =esw_qos_devlink_rate_to_mbps (vport->dev,"tx_share" , &tx_share, extack);if (err)return err; -esw_qos_lock (esw); err =mlx5_esw_qos_set_vport_min_rate (vport, tx_share, extack); -if (err) - goto out; -esw_vport_qos_prune_empty (vport); -out: -esw_qos_unlock (esw); +if (!err) +esw_vport_qos_prune_empty (vport);return err; }
@@ -1712,24 +1651,18 @@ int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void * u64 tx_max, struct netlink_ext_ack *extack) { struct mlx5_vport *vport = priv; -struct mlx5_eswitch *esw;int err; - esw = vport->dev->priv.eswitch; -if (!mlx5_esw_allowed (esw)) +if (!mlx5_esw_allowed (vport->dev->priv.eswitch))return -EPERM ; err =esw_qos_devlink_rate_to_mbps (vport->dev,"tx_max" , &tx_max, extack);if (err)return err; -esw_qos_lock (esw); err =mlx5_esw_qos_set_vport_max_rate (vport, tx_max, extack); -if (err) - goto out; -esw_vport_qos_prune_empty (vport); -out: -esw_qos_unlock (esw); +if (!err) +esw_vport_qos_prune_empty (vport);return err; }
@@ -1740,34 +1673,30 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, {struct mlx5_esw_sched_node *vport_node;struct mlx5_vport *vport = priv; -struct mlx5_eswitch *esw;bool disable;int err =0 ; - esw = vport->dev->priv.eswitch; -if (!mlx5_esw_allowed (esw)) +if (!mlx5_esw_allowed (vport->dev->priv.eswitch))return -EPERM ; disable =esw_qos_tc_bw_disabled (tc_bw); -esw_qos_lock (esw);if (!esw_qos_vport_validate_unsupported_tc_bw (vport, tc_bw)) {NL_SET_ERR_MSG_MOD (extack,"E-Switch traffic classes number is not supported" ); - err = -EOPNOTSUPP ; - goto unlock; +return -EOPNOTSUPP ; } vport_node = vport->qos.sched_node;if (disable && !vport_node) - goto unlock; +return 0 ;if (disable) {if (vport_node->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) err =esw_qos_vport_update (vport,SCHED_NODE_TYPE_VPORT , vport_node->parent, extack);esw_vport_qos_prune_empty (vport); - goto unlock; +return err; }if (!vport_node) {
@@ -1782,8 +1711,6 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, }if (!err)esw_qos_set_tc_arbiter_bw_shares (vport_node, tc_bw, extack); -unlock: -esw_qos_unlock (esw);return err; }
@@ -1793,28 +1720,22 @@ int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, struct netlink_ext_ack *extack) {struct mlx5_esw_sched_node *node = priv; -struct mlx5_eswitch *esw = node->esw;bool disable;int err; -if (!esw_qos_validate_unsupported_tc_bw (esw, tc_bw)) { +if (!esw_qos_validate_unsupported_tc_bw (node->esw, tc_bw)) {NL_SET_ERR_MSG_MOD (extack,"E-Switch traffic classes number is not supported" );return -EOPNOTSUPP ; } disable =esw_qos_tc_bw_disabled (tc_bw); -esw_qos_lock (esw); -if (disable) { - err =esw_qos_node_disable_tc_arbitration (node, extack); - goto unlock; - } +if (disable) +return esw_qos_node_disable_tc_arbitration (node, extack); err =esw_qos_node_enable_tc_arbitration (node, extack);if (!err)esw_qos_set_tc_arbiter_bw_shares (node, tc_bw, extack); -unlock: -esw_qos_unlock (esw);return err; }
@@ -1822,17 +1743,14 @@ int mlx5_esw_devlink_rate_node_tx_share_set(struct devlink_rate *rate_node, void u64 tx_share, struct netlink_ext_ack *extack) { struct mlx5_esw_sched_node *node = priv; -struct mlx5_eswitch *esw = node->esw;int err; - err =esw_qos_devlink_rate_to_mbps (esw->dev,"tx_share" , &tx_share, extack); + err =esw_qos_devlink_rate_to_mbps (node->esw->dev,"tx_share" , + &tx_share, extack);if (err)return err; -esw_qos_lock (esw); - err =esw_qos_set_node_min_rate (node, tx_share, extack); -esw_qos_unlock (esw); -return err; +return esw_qos_set_node_min_rate (node, tx_share, extack); } int mlx5_esw_devlink_rate_node_tx_max_set(struct devlink_rate *rate_node, void *priv,
@@ -1846,10 +1764,7 @@ int mlx5_esw_devlink_rate_node_tx_max_set(struct devlink_rate *rate_node, void *if (err)return err; -esw_qos_lock (esw); - err =esw_qos_sched_elem_config (node, tx_max, node->bw_share, extack); -esw_qos_unlock (esw); -return err; +return esw_qos_sched_elem_config (node, tx_max, node->bw_share, extack); } int mlx5_esw_devlink_rate_node_new(struct devlink_rate *rate_node, void **priv,
@@ -1857,30 +1772,23 @@ int mlx5_esw_devlink_rate_node_new(struct devlink_rate *rate_node, void **priv, {struct mlx5_esw_sched_node *node;struct mlx5_eswitch *esw; -int err =0 ; esw =mlx5_devlink_eswitch_get (rate_node->devlink);if (IS_ERR (esw))return PTR_ERR (esw); -esw_qos_lock (esw);if (esw->mode !=MLX5_ESWITCH_OFFLOADS ) {NL_SET_ERR_MSG_MOD (extack,"Rate node creation supported only in switchdev mode" ); - err = -EOPNOTSUPP ; - goto unlock; +return -EOPNOTSUPP ; } node =esw_qos_create_vports_sched_node (esw, extack); -if (IS_ERR (node)) { - err =PTR_ERR (node); - goto unlock; - } +if (IS_ERR (node)) +return PTR_ERR (node); *priv = node; -unlock: -esw_qos_unlock (esw); -return err; +return 0 ; } int mlx5_esw_devlink_rate_node_del(struct devlink_rate *rate_node, void *priv,
@@ -1889,10 +1797,9 @@ int mlx5_esw_devlink_rate_node_del(struct devlink_rate *rate_node, void *priv,struct mlx5_esw_sched_node *node = priv;struct mlx5_eswitch *esw = node->esw; -esw_qos_lock (esw);__esw_qos_destroy_node (node, extack);esw_qos_put (esw); -esw_qos_unlock (esw); +return 0 ; }
@@ -1909,7 +1816,6 @@ mlx5_esw_qos_vport_update_parent(struct mlx5_vport *vport,return -EOPNOTSUPP ; } -esw_qos_lock (esw);if (!vport->qos.sched_node && parent) {enum sched_node_type type;
@@ -1920,13 +1826,15 @@ mlx5_esw_qos_vport_update_parent(struct mlx5_vport *vport, }else if (vport->qos.sched_node) { err =esw_qos_vport_update_parent (vport, parent, extack); } -esw_qos_unlock (esw); +return err; }void mlx5_esw_qos_vport_clear_parent (struct mlx5_vport *vport) { +devl_lock (vport->dev->shd);mlx5_esw_qos_vport_update_parent (vport,NULL ,NULL ); +devl_unlock (vport->dev->shd); } int mlx5_esw_devlink_rate_leaf_parent_set(struct devlink_rate *devlink_rate,
@@ -1939,13 +1847,8 @@ int mlx5_esw_devlink_rate_leaf_parent_set(struct devlink_rate *devlink_rate,int err; err =mlx5_esw_qos_vport_update_parent (vport, node, extack); -if (!err) { -struct mlx5_eswitch *esw = vport->dev->priv.eswitch; - -esw_qos_lock (esw); +if (!err)esw_vport_qos_prune_empty (vport); -esw_qos_unlock (esw); - }return err; }
@@ -2071,14 +1974,12 @@ static int mlx5_esw_qos_node_update_parent(struct mlx5_esw_sched_node *node, struct netlink_ext_ack *extack) {struct mlx5_esw_sched_node *curr_parent; -struct mlx5_eswitch *esw = node->esw;int err; err =mlx5_esw_qos_node_validate_set_parent (node, parent, extack);if (err)return err; -esw_qos_lock (esw); curr_parent = node->parent;if (node->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) { err = esw_qos_tc_arbiter_node_update_parent(node, parent,
@@ -2088,15 +1989,11 @@ static int mlx5_esw_qos_node_update_parent(struct mlx5_esw_sched_node *node, }if (err) - goto out; - -esw_qos_normalize_min_rate (esw, curr_parent, extack); -esw_qos_normalize_min_rate (esw, parent, extack); - -out: -esw_qos_unlock (esw); +return err; -return err; +esw_qos_normalize_min_rate (node->esw, curr_parent, extack); +esw_qos_normalize_min_rate (node->esw, parent, extack); +return 0 ; } int mlx5_esw_devlink_rate_node_parent_set(struct devlink_rate *devlink_rate,
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h +0 -3
@@ -6,9 +6,6 @@#ifdef CONFIG_MLX5_ESWITCH -int mlx5_esw_qos_init (struct mlx5_eswitch *esw); -void mlx5_esw_qos_cleanup (struct mlx5_eswitch *esw); -int mlx5_esw_qos_set_vport_rate (struct mlx5_vport *evport,u32 max_rate,u32 min_rate);bool mlx5_esw_qos_get_vport_rate (struct mlx5_vport *vport,u32 *max_rate,u32 *min_rate);void mlx5_esw_qos_vport_disable (struct mlx5_vport *vport);
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +1 -8
@@ -49,6 +49,7 @@#include "ecpf.h" #include "en/mod_hdr.h" #include "en_accel/ipsec.h" +#include "sh_devlink.h" enum {MLX5_ACTION_NONE =0 ,
@@ -1618,10 +1619,6 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int num_vfs)MLX5_NB_INIT (&esw->nb, eswitch_vport_event,NIC_VPORT_CHANGE );mlx5_eq_notifier_register (esw->dev, &esw->nb); - err =mlx5_esw_qos_init (esw); -if (err) - goto err_esw_init; -if (esw->mode ==MLX5_ESWITCH_LEGACY ) { err =esw_legacy_enable (esw); }else {
@@ -2028,9 +2025,6 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) goto reps_err; esw->mode =MLX5_ESWITCH_LEGACY ; - err =mlx5_esw_qos_init (esw); -if (err) - goto reps_err;mutex_init (&esw->offloads.encap_tbl_lock);hash_init (esw->offloads.encap_tbl);
@@ -2080,7 +2074,6 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw)esw_info (esw->dev,"cleanup\n" ); -mlx5_esw_qos_cleanup (esw);destroy_workqueue (esw->work_queue);WARN_ON (refcount_read (&esw->qos.refcnt));mutex_destroy (&esw->state_lock);
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +6 -5
@@ -224,8 +224,9 @@ struct mlx5_vport {struct mlx5_vport_info info; -/* Protected with the E-Switch qos domain lock. The Vport QoS can -* either be disabled (sched_node is NULL) or in one of three states: +/* Protected by mlx5_shd_lock(). +* The Vport QoS can either be disabled (sched_node is NULL) or in one +* of three states: * 1. Regular QoS (sched_node is a vport node). * 2. TC QoS enabled on the vport (sched_node is a TC arbiter). * 3. TC QoS enabled on the vport's parent node
@@ -356,7 +357,6 @@ enum { };struct dentry ; -struct mlx5_qos_domain ;struct mlx5_eswitch {struct mlx5_core_dev *dev;
@@ -383,12 +383,13 @@ struct mlx5_eswitch {struct rw_semaphore mode_lock;atomic64_t user_count; -/* Protected with the E-Switch qos domain lock. */ +/* The QoS tree is stored in mlx5_shd. +* QoS changes are serialized with mlx5_shd_lock(). +*/ struct {/* Initially 0, meaning no QoS users and QoS is disabled. */ refcount_t refcnt;u32 root_tsar_ix; -struct mlx5_qos_domain *domain; } qos;struct mlx5_esw_bridge_offloads *br_offloads;
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.c +29 -1
@@ -9,6 +9,16 @@static const struct devlink_ops mlx5_shd_ops = { }; +struct mlx5_shd_priv { +struct list_head qos_nodes; +}; + +static int mlx5_shd_priv_init (struct mlx5_shd_priv *shd_priv) +{ +INIT_LIST_HEAD (&shd_priv->qos_nodes); +return 0 ; +} +int mlx5_shd_init (struct mlx5_core_dev *dev) { u8 *vpd_data __free(kfree) = NULL;
@@ -45,11 +55,18 @@ int mlx5_shd_init(struct mlx5_core_dev *dev) *end ='\0' ;/* Get or create shared devlink instance */ - devlink =devlink_shd_get (sn, &mlx5_shd_ops,0 ); + devlink =devlink_shd_get (sn, &mlx5_shd_ops, +sizeof (struct mlx5_shd_priv ));kfree (sn);if (!devlink)return -ENOMEM ; + err =mlx5_shd_priv_init (devlink_shd_get_priv (devlink)); +if (err <0 ) { +devlink_shd_put (devlink); +return err; + } + dev->shd = devlink;return 0 ; }
@@ -61,3 +78,14 @@ void mlx5_shd_uninit(struct mlx5_core_dev *dev)devlink_shd_put (dev->shd); } + +struct list_head *mlx5_shd_get_qos_nodes (struct mlx5_core_dev *dev) +{ +struct mlx5_shd_priv *shd_priv; + +if (!dev->shd) +return NULL ; +devl_assert_locked (dev->shd); + shd_priv =devlink_shd_get_priv (dev->shd); +return &shd_priv->qos_nodes; +}
drivers/net/ethernet/mellanox/mlx5/core/sh_devlink.h +2 -0
@@ -9,4 +9,6 @@int mlx5_shd_init (struct mlx5_core_dev *dev);void mlx5_shd_uninit (struct mlx5_core_dev *dev); +struct list_head *mlx5_shd_get_qos_nodes (struct mlx5_core_dev *dev); +#endif /* __MLX5_SH_DEVLINK_H__ */
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [net-next,V5,12/15] net/mlx5: Store QoS sched nodes in the sh_devlink
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
For local reproduction steps see https://netdev-ai.bots.linux.dev/ai-local.html
---
net/mlx5: Store QoS sched nodes in the sh_devlink
This commit moves the QoS scheduling nodes from per-eswitch storage to the shared devlink instance. It replaces per-eswitch locking with the shared devlink lock and removes explicit locking from devlink rate callbacks with the expectation that devlink holds the shd lock.
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +0
-0
[ ... ]
Quoted 5 lines
static void
esw_assert_qos_lock_held(struct mlx5_eswitch *esw){drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +1
-1-lockdep_assert_held (&esw->qos.domain->lock); +devl_assert_locked (esw->dev->shd);}
[ ... ]
Quoted 17 lines
int mlx5_esw_devlink_rate_leaf_tx_share_set(struct
devlink_rate*rate_leaf, void *priv,
u64tx_share, structnetlink_ext_ack*extack){ struct mlx5_vport *vport = priv;drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +0
-1-struct mlx5_eswitch *esw;int err;drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +1
-2- esw = vport->dev->priv.eswitch; -if (!mlx5_esw_allowed (esw)) +if (!mlx5_esw_allowed (vport->dev->priv.eswitch))return -EPERM; err = esw_qos_devlink_rate_to_mbps(vport->dev, "tx_share", &tx_share, extack); if (err) return err;drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +0
-1-esw_qos_lock (esw);
Quoted 1 line
err = mlx5_esw_qos_set_vport_min_rate(vport, tx_share, extack);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
At this commit, the esw_qos_lock/unlock calls are removed with the expectation that "all calls originating from devlink rate already have the shd lock held" (per the commit message). However, the devlink rate infrastructure only locks dev->shd when supported_cross_device_rate_nodes is set in the devlink_ops, which does not happen until commit ef6db4a7f381 ("net/mlx5: qos: Enable cross-device scheduling").
The call chain is:
devlink_nl_rate_set_doit() -> devl_rate_lock() -> mlx5_esw_devlink_rate_leaf_tx_share_set() -> mlx5_esw_qos_set_vport_min_rate() -> esw_assert_qos_lock_held() -> devl_assert_locked(esw->dev->shd)
Since devl_rate_lock() only locks the nested-in devlink when supported_cross_device_rate_nodes is true, and that flag is not set at this commit, the assertion will fail with lockdep enabled, and QoS data structures will be accessed without proper locking.
This creates a bisect window between this commit and ef6db4a7f381 where the QoS locking is broken. Could the setting of supported_cross_device_rate_nodes be moved into this patch to avoid the bisect issue?
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +2
-5-if (err) - goto out; -esw_vport_qos_prune_empty (vport); -out: -esw_qos_unlock (esw); +if (!err) +esw_vport_qos_prune_empty (vport);return err; }
[ ... ]
Re: [net-next,V5,12/15] net/mlx5: Store QoS sched nodes in the sh_devlink
Quoted 8 lines · Jakub Kicinski
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
This creates a bisect window between this commit and
ef6db4a7f381where
the QoS locking is broken. Could the setting ofsupported_cross_device_rate_nodes
be moved into this patch to avoid the bisect issue?
This was a useful insight, and a clear 2-patch bisect window where warnings would be generated.
I will merge the setting of supported_cross_device_rate_nodes with this patch, since the mlx5 code is protected against cross-esw parents until the next one in the series ("Support cross-device tx scheduling").
Cosmin.
Tariq Toukan
maintainerPATCH v5 13/15 net/mlx5: qos: Support cross-device tx scheduling
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Up to now, rate groups could only contain vports from the same E-Switch. This patch relaxes that restriction if the device supports it (HCA_CAP.esw_cross_esw_sched == true) and the right conditions are met:
- Link Aggregation (LAG) is enabled.
- The E-Switches are from the same shared devlink device.
This patch does not yet enable cross-esw scheduling, it's just the last preparatory patch.
--- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 122 ++++++++++++------ 1 file changed, 86 insertions(+), 36 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +86 -36
@@ -50,7 +50,9 @@ struct mlx5_esw_sched_node {enum sched_node_type type;/* The eswitch this node belongs to. */ struct mlx5_eswitch *esw; -/* The children nodes of this node, empty list for leaf nodes. */ +/* The children nodes of this node, empty list for leaf nodes. +* Can be from multiple E-Switches. +*/ struct list_head children;/* Valid only if this node is associated with a vport. */ struct mlx5_vport *vport;
@@ -419,6 +421,7 @@ esw_qos_vport_create_sched_element(struct mlx5_esw_sched_node *vport_node,struct mlx5_esw_sched_node *parent = vport_node->parent;u32 sched_ctx[MLX5_ST_SZ_DW (scheduling_context)] = {};struct mlx5_core_dev *dev = vport_node->esw->dev; +struct mlx5_vport *vport = vport_node->vport;void *attr; if (!mlx5_qos_element_type_supported(
@@ -430,11 +433,18 @@ esw_qos_vport_create_sched_element(struct mlx5_esw_sched_node *vport_node,MLX5_SET (scheduling_context, sched_ctx, element_type,SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT ); attr =MLX5_ADDR_OF (scheduling_context, sched_ctx, element_attributes); -MLX5_SET (vport_element, attr, vport_number, vport_node->vport->vport); +MLX5_SET (vport_element, attr, vport_number, vport->vport);MLX5_SET (scheduling_context, sched_ctx, parent_element_id, parent ? parent->ix : vport_node->esw->qos.root_tsar_ix);MLX5_SET (scheduling_context, sched_ctx, max_average_bw, vport_node->max_rate); +if (vport->dev != dev) { +/* The port is assigned to a node on another eswitch. */ +MLX5_SET (vport_element, attr, eswitch_owner_vhca_id_valid, + true); +MLX5_SET (vport_element, attr, eswitch_owner_vhca_id, +MLX5_CAP_GEN (vport->dev, vhca_id)); + }return esw_qos_node_create_sched_element (vport_node, sched_ctx, extack); }
@@ -446,6 +456,7 @@ esw_qos_vport_tc_create_sched_element(struct mlx5_esw_sched_node *vport_tc_node, { u32 sched_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; struct mlx5_core_dev *dev = vport_tc_node->esw->dev; + struct mlx5_vport *vport = vport_tc_node->vport; void *attr; if (!mlx5_qos_element_type_supported(
@@ -457,8 +468,7 @@ esw_qos_vport_tc_create_sched_element(struct mlx5_esw_sched_node *vport_tc_node,MLX5_SET (scheduling_context, sched_ctx, element_type,SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC ); attr =MLX5_ADDR_OF (scheduling_context, sched_ctx, element_attributes); -MLX5_SET (vport_tc_element, attr, vport_number, - vport_tc_node->vport->vport); +MLX5_SET (vport_tc_element, attr, vport_number, vport->vport);MLX5_SET (vport_tc_element, attr, traffic_class, vport_tc_node->tc);MLX5_SET (scheduling_context, sched_ctx, max_bw_obj_id, rate_limit_elem_ix);
@@ -466,6 +476,13 @@ esw_qos_vport_tc_create_sched_element(struct mlx5_esw_sched_node *vport_tc_node, vport_tc_node->parent->ix);MLX5_SET (scheduling_context, sched_ctx, bw_share, vport_tc_node->bw_share); +if (vport->dev != dev) { +/* The port is assigned to a node on another eswitch. */ +MLX5_SET (vport_tc_element, attr, eswitch_owner_vhca_id_valid, + true); +MLX5_SET (vport_tc_element, attr, eswitch_owner_vhca_id, +MLX5_CAP_GEN (vport->dev, vhca_id)); + }return esw_qos_node_create_sched_element (vport_tc_node, sched_ctx, extack);
@@ -1194,6 +1211,29 @@ static int esw_qos_vport_tc_check_type(enum sched_node_type curr_type,return 0 ; } +static bool esw_qos_validate_unsupported_tc_bw (struct mlx5_eswitch *esw, +u32 *tc_bw) +{ +int i, num_tcs =esw_qos_num_tcs (esw->dev); + +for (i = num_tcs; i <DEVLINK_RATE_TCS_MAX ; i++) +if (tc_bw[i]) +return false; + +return true; +} + +static bool esw_qos_vport_validate_unsupported_tc_bw (struct mlx5_vport *vport, +u32 *tc_bw) +{ +struct mlx5_esw_sched_node *node = vport->qos.sched_node; +struct mlx5_eswitch *esw = vport->dev->priv.eswitch; + + esw = (node && node->parent) ? node->parent->esw : esw; + +return esw_qos_validate_unsupported_tc_bw (esw, tc_bw); +} + static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type type, struct mlx5_esw_sched_node *parent,
@@ -1213,8 +1253,17 @@ static int esw_qos_vport_update(struct mlx5_vport *vport,if (err)return err; -if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) +if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) { +struct mlx5_eswitch *esw = parent ? + parent->esw : vport->dev->priv.eswitch; +esw_qos_tc_arbiter_get_bw_shares (vport_node, curr_tc_bw); +if (!esw_qos_validate_unsupported_tc_bw (esw, curr_tc_bw)) { +NL_SET_ERR_MSG_MOD (extack, +"Unsupported traffic classes on the new device" ); +return -EOPNOTSUPP ; + } + }esw_qos_vport_disable (vport, extack);
@@ -1224,10 +1273,9 @@ static int esw_qos_vport_update(struct mlx5_vport *vport, extack =NULL ; } -if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) { +if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type)esw_qos_set_tc_arbiter_bw_shares (vport_node, curr_tc_bw, extack); - }return err; }
@@ -1575,30 +1623,6 @@ static int esw_qos_devlink_rate_to_mbps(struct mlx5_core_dev *mdev, const char *return 0 ; } -static bool esw_qos_validate_unsupported_tc_bw (struct mlx5_eswitch *esw, -u32 *tc_bw) -{ -int i, num_tcs =esw_qos_num_tcs (esw->dev); - -for (i = num_tcs; i <DEVLINK_RATE_TCS_MAX ; i++) { -if (tc_bw[i]) -return false; - } - -return true; -} - -static bool esw_qos_vport_validate_unsupported_tc_bw (struct mlx5_vport *vport, -u32 *tc_bw) -{ -struct mlx5_esw_sched_node *node = vport->qos.sched_node; -struct mlx5_eswitch *esw = vport->dev->priv.eswitch; - - esw = (node && node->parent) ? node->parent->esw : esw; - -return esw_qos_validate_unsupported_tc_bw (esw, tc_bw); -} -static bool esw_qos_tc_bw_disabled (u32 *tc_bw) {int i;
@@ -1803,18 +1827,44 @@ int mlx5_esw_devlink_rate_node_del(struct devlink_rate *rate_node, void *priv,return 0 ; } +static int +mlx5_esw_validate_cross_esw_scheduling (struct mlx5_eswitch *esw, +struct mlx5_esw_sched_node *parent, +struct netlink_ext_ack *extack) +{ +if (!parent || esw == parent->esw) +return 0 ; + +if (!MLX5_CAP_QOS (esw->dev, esw_cross_esw_sched)) { +NL_SET_ERR_MSG_MOD (extack, +"Cross E-Switch scheduling is not supported" ); +return -EOPNOTSUPP ; + } +if (esw->dev->shd != parent->esw->dev->shd) { +NL_SET_ERR_MSG_MOD (extack, +"Cannot add vport to a parent belonging to a different device" ); +return -EOPNOTSUPP ; + } +if (!mlx5_lag_is_active (esw->dev)) { +NL_SET_ERR_MSG_MOD (extack, +"Cross E-Switch scheduling requires LAG to be activated" ); +return -EOPNOTSUPP ; + } + +return 0 ; +} +static int mlx5_esw_qos_vport_update_parent (struct mlx5_vport *vport,struct mlx5_esw_sched_node *parent,struct netlink_ext_ack *extack) {struct mlx5_eswitch *esw = vport->dev->priv.eswitch; -int err =0 ; +int err; -if (parent && parent->esw != esw) { -NL_SET_ERR_MSG_MOD (extack,"Cross E-Switch scheduling is not supported" ); -return -EOPNOTSUPP ; - } + err =mlx5_esw_validate_cross_esw_scheduling (esw, parent, extack); +if (err) +return err;if (!vport->qos.sched_node && parent) {enum sched_node_type type;
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 14/15 net/mlx5: qos: Enable cross-device scheduling
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
Enable the new devlink feature.
All rate nodes will be stored in the shared device, when available. The shared instance lock will additionally be acquired by all rate manipulation code to protect against concurrent execution across different functions of the same physical device.
--- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +1 -0
@@ -385,6 +385,7 @@ static const struct devlink_ops mlx5_devlink_ops = { .rate_node_del = mlx5_esw_devlink_rate_node_del, .rate_leaf_parent_set = mlx5_esw_devlink_rate_leaf_parent_set, .rate_node_parent_set = mlx5_esw_devlink_rate_node_parent_set, + .supported_cross_device_rate_nodes = true,#endif #ifdef CONFIG_MLX5_SF_MANAGER .port_new = mlx5_devlink_sf_port_new,
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v5 15/15 net/mlx5: Document devlink rates
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
It seems rates were not documented in the mlx5-specific file, so add examples on how to limit VFs and groups and also provide an example of the intended way to achieve cross-esw scheduling.
--- Documentation/networking/devlink/mlx5.rst | 33 +++++++++++++++++++++++ 1 file changed, 33 insertions(+)
Documentation/networking/devlink/mlx5.rst +33 -0
@@ -419,3 +419,36 @@ User commands examples: .. note:: This command can run over all interfaces such as PF/VF and representor ports. + +Rates +===== + +mlx5 devices can limit transmission of individual VFs or a group of them via +the devlink-rate API in switchdev mode. + +User commands examples: + +- Print the existing rates:: + + $ devlink port function rate show + +- Set a max tx limit on traffic from VF0:: + + $ devlink port function rate set pci/0000:82:00.0/1 tx_max 10Gbit + +- Create a rate group with a max tx limit and adding two VFs to it:: + + $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit + $ devlink port function rate set pci/0000:82:00.0/1 parent group1 + $ devlink port function rate set pci/0000:82:00.0/2 parent group1 + +- Same scenario, with a min guarantee of 20% of the bandwidth for the first VFs:: + + $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit + $ devlink port function rate set pci/0000:82:00.0/1 parent group1 tx_share 2Gbit + $ devlink port function rate set pci/0000:82:00.0/2 parent group1 + +- Cross-device scheduling:: + + $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit + $ devlink port function rate set pci/0000:82:00.1/32769 parent pci/0000:82:00.0/group1
Signature · 1 line
2.44.0