devlink: Add boot-time eswitch mode defaults
Subsystems (9)
Mark Bloch
maintainerv3 cover devlink: Add boot-time eswitch mode defaults
This series adds a devlink_eswitch_mode= kernel command line parameter for applying a default devlink eswitch mode during device initialization.
Following the discussion with Jakub[1] and the feedback on the RFC postings, this version keeps the scope limited to a boot-time devlink eswitch mode default only.
The option selects either all devlink handles or an explicit comma separated handle list:
devlink_eswitch_mode=*=switchdev devlink_eswitch_mode=pci/0000:08:00.0,pci/0000:09:00.1=switchdev_inactive
The supported modes are legacy, switchdev and switchdev_inactive. The selected mode is applied through the existing eswitch_mode_set() devlink operation, the same operation used by the devlink eswitch mode command.
The preparatory patches move registration points that expose the devlink instance before the driver is ready for a registration-time eswitch mode change. Where registration is moved later, the matching unregister path is moved earlier so unregister notifications are sent from devl_unregister() before object teardown. The final patch adds the parser and applies the default from devlink core when a matching instance is registered and after a successful devlink reload that performed DRIVER_REINIT.
Patch 1 skips devlink health recovery notifications while a devlink instance is not registered. Health state and counters are still updated, but there is no registered instance for userspace to observe or receive notifications from yet. This lets drivers move registration later without hitting health notification registration assertions during early initialization.
Patch 2 moves netdevsim devlink registration after device initialization, so registration-time defaults can call eswitch_mode_set() after simulator state is ready. It also unregisters devlink before netdevsim tears down the objects that were registered before devlink became visible.
Patch 3 clears the mlx5 FW reset-in-progress bit before reloading after a firmware reset.
Patch 4 moves mlx5 devlink registration after device initialization, including the lightweight init path, and moves unregister before the matching teardown.
Patch 5 moves octeontx2 AF devlink registration after SR-IOV setup and switch lock initialization.
Patch 6 moves octeontx2 PF devlink registration after PF SR-IOV state setup.
Patch 7 adds the devlink_eswitch_mode= parser, documentation, registration-time default application and successful reload default application.
Changelog:
v2 -> v3:
- Change the devlink_eswitch_mode= API syntax to use <selector>=<mode> instead of [<selector>]:<mode>, following a comment from Randy Dunlap.
v1 -> v2:
- Move default eswitch mode application into devlink core. The default is now applied during devlink registration and after a successful devlink reload that performed DRIVER_REINIT.
- Remove the exported devl_apply_default_esw_mode() driver API and the mlx5 driver-side call to it.
- Skip devlink health recovery notifications while the devlink instance is not registered, so drivers can move registration later without early health work hitting registration assertions.
- Move mlx5 devlink registration after device initialization, including the lightweight init path, so the core can apply the default through the normal registration flow.
- Move the matching netdevsim and mlx5 unregister paths before object teardown, so unregister notifications come from devl_unregister() and the later object teardown paths run while the devlink instance is no longer registered.
- Add registration-ordering preparation patches for netdevsim and octeontx2 AF/PF, so their eswitch state is ready before registration-time defaults may call eswitch_mode_set().
[1] https://lore.kernel.org/all/[email protected]/
RFC V1 : https://lore.kernel.org/all/[email protected]/
RFC V2 : https://lore.kernel.org/all/[email protected]/
v1 : https://lore.kernel.org/all/[email protected]/
v2 : https://lore.kernel.org/all/[email protected]/
Mark Bloch (7):
devlink: Skip health recover notifications before register
netdevsim: Register devlink after device init net/mlx5: Clear FW reset-in-progress bit before reload net/mlx5: Register devlink after device init
octeontx2-af: Register devlink after SR-IOV init
octeontx2-pf: Register devlink after SR-IOV state init
devlink: Add eswitch mode boot defaults
.../admin-guide/kernel-parameters.txt | 25 ++ .../networking/devlink/devlink-defaults.rst | 78 +++++ Documentation/networking/devlink/index.rst | 1 + .../net/ethernet/marvell/octeontx2/af/rvu.c | 24 +- .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 17 +- .../ethernet/mellanox/mlx5/core/fw_reset.c | 28 +- .../net/ethernet/mellanox/mlx5/core/main.c | 34 ++- drivers/net/netdevsim/dev.c | 15 +- net/devlink/core.c | 271 ++++++++++++++++++ net/devlink/dev.c | 3 + net/devlink/devl_internal.h | 1 + net/devlink/health.c | 3 +- 12 files changed, 451 insertions(+), 49 deletions(-) create mode 100644 Documentation/networking/devlink/devlink-defaults.rst
base-commit: bfa3d89cc15c09f7d1581c834a5ed725189ec19f
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 1/7 devlink: Skip health recover notifications before register
devlink health reports can be generated before the devlink instance is registered. This can happen during driver initialization when a driver creates health reporters early and its health polling detects an error before devlink_register() is reached.
devlink health still records the report state and counters in that case, but userspace cannot observe the devlink instance yet and there is no registered handle to notify through. Skip the netlink notification while the devlink instance is not registered instead of asserting registration.
This keeps later userspace queries useful after registration while avoiding a warning from early health reports.
--- net/devlink/health.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
net/devlink/health.c +1 -2
@@ -513,9 +513,8 @@ static void devlink_recover_notify(struct devlink_health_reporter *reporter,int err;WARN_ON (cmd !=DEVLINK_CMD_HEALTH_REPORTER_RECOVER ); -ASSERT_DEVLINK_REGISTERED (devlink); -if (!devlink_nl_notify_need (devlink)) +if (!__devl_is_registered (devlink) || !devlink_nl_notify_need (devlink))return ; msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL );
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 2/7 netdevsim: Register devlink after device init
In reply to Mark Blochdevl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.
Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.
--- drivers/net/netdevsim/dev.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
drivers/net/netdevsim/dev.c +7 -8
@@ -1680,13 +1680,9 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) goto err_devlink_unlock; } - err =devl_register (devlink); -if (err) - goto err_vfc_free; - err =nsim_dev_resources_register (devlink);if (err) - goto err_dl_unregister; + goto err_vfc_free; err =devl_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));
@@ -1733,9 +1729,14 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) goto err_hwstats_exit; nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ; + err =devl_register (devlink); +if (err) + goto err_port_del_all;devl_unlock (devlink);return 0 ; +err_port_del_all: +nsim_dev_port_del_all (nsim_dev); err_hwstats_exit:nsim_dev_hwstats_exit (nsim_dev); err_psample_exit:
@@ -1757,8 +1758,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) ARRAY_SIZE(nsim_devlink_params)); err_resource_unregister:devl_resources_unregister (devlink); -err_dl_unregister: -devl_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs); err_devlink_unlock:
@@ -1797,6 +1796,7 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)struct devlink *devlink =priv_to_devlink (nsim_dev);devl_lock (devlink); +devl_unregister (devlink);nsim_dev_reload_destroy (nsim_dev);nsim_bpf_dev_exit (nsim_dev);
@@ -1804,7 +1804,6 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)devl_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));devl_resources_unregister (devlink); -devl_unregister (devlink);kfree (nsim_dev->vfconfigs);kfree (nsim_dev->fa_cookie);mutex_destroy (&nsim_dev->progs_list_lock);
Signature · 1 line
2.34.1
Jakub Kicinski
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:
devl_register()makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may calleswitch_mode_set()to apply a boot-time default eswitch mode.Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Quoted 5 lines
Move
devl_unregister()to the beginning ofnsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.
This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.
Mark Bloch
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:
devl_register()makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may calleswitch_mode_set()to apply a boot-time default eswitch mode.Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Quoted 5 lines
Move
devl_unregister()to the beginning ofnsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.
Quoted 8 lines
This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases.devl_lock()is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks holddevl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.
Let me clarify.
No locking is changed here, and I don't want to make register/unregister ordering a substitute for devl_lock().
The only requirement I have for this series is that devl_register() is called only once the driver is ready for devlink core to call eswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode from devl_register() instead of adding an explicit driver call.
So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.
For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.
Mark
Jakub Kicinski
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:
devl_register()makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may calleswitch_mode_set()to apply a boot-time default eswitch mode.Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Quoted 5 lines
Move
devl_unregister()to the beginning ofnsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.Quoted 8 lines
This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases.devl_lock()is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks holddevl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.Let me clarify.
Quoted 7 lines
No locking is changed here, and I don't want to make register/unregister ordering a substitute for
devl_lock().The only requirement I have for this series is that
devl_register()is called only once the driver is ready for devlink core to calleswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode fromdevl_register()instead of adding an explicit driver call.
This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.
If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once devl_lock() is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.
Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...
Quoted 7 lines
So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.
For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.
Mark Bloch
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 36 lines · Jakub Kicinski
Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:
devl_register()makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may calleswitch_mode_set()to apply a boot-time default eswitch mode.Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Quoted 5 lines
Move
devl_unregister()to the beginning ofnsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.Quoted 8 lines
This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases.devl_lock()is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks holddevl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.Let me clarify.
Quoted 7 lines
No locking is changed here, and I don't want to make register/unregister ordering a substitute for
devl_lock().The only requirement I have for this series is that
devl_register()is called only once the driver is ready for devlink core to calleswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode fromdevl_register()instead of adding an explicit driver call.
Quoted 4 lines
This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.
Right, the reason I moved in this direction is that in v1 I had the explicit driver call, and Jiri asked to make this transparent from devlink core instead.
Quoted 7 lines
If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once
devl_lock()is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.
The workqueue fallback is possible, but I think it makes the semantics more complicated.
We would need to track devlink instances which still need the default applied, and the worker would have to skip/remove them once handled.
More importantly, the worker can race with userspace setting the eswitch mode, so we would also need some state to tell whether the user already changed the mode. That feels more fragile than an explicit driver call.
Quoted 4 lines
Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...
That's why I started with the explicit call in the first place.
I can switch back to this model: drivers which support boot time eswitch defaults will opt in and call the helper once they are ready. This keeps the support explicit per driver and avoids making it depend on where devl_register() happens in the init path.
With that, devlink can tell at register time whether the instance supports boot time eswitch defaults. If the user configured a default for an instance whose driver did not opt in, devlink can write to dmesg from devl_register().
Not perfect, but at least the user gets a visible failure instead of the config being silently ignored.
Mark
Quoted 8 lines
Quoted 8 lines
So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.
For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.
Mark Bloch
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 84 lines · Mark Bloch
Quoted 36 lines · Jakub Kicinski
Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:
devl_register()makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may calleswitch_mode_set()to apply a boot-time default eswitch mode.Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.
Quoted 5 lines
Move
devl_unregister()to the beginning ofnsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.Quoted 8 lines
This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases.devl_lock()is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks holddevl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.Let me clarify.
Quoted 7 lines
No locking is changed here, and I don't want to make register/unregister ordering a substitute for
devl_lock().The only requirement I have for this series is that
devl_register()is called only once the driver is ready for devlink core to calleswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode fromdevl_register()instead of adding an explicit driver call.Quoted 4 lines
This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.
Right, the reason I moved in this direction is that in v1 I had the explicit driver call, and Jiri asked to make this transparent from devlink core instead.
Quoted 7 lines
If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once
devl_lock()is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.The workqueue fallback is possible, but I think it makes the semantics more complicated.
We would need to track devlink instances which still need the default applied, and the worker would have to skip/remove them once handled.
More importantly, the worker can race with userspace setting the eswitch mode, so we would also need some state to tell whether the user already changed the mode. That feels more fragile than an explicit driver call.
Quoted 4 lines
Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...That's why I started with the explicit call in the first place.
I can switch back to this model: drivers which support boot time eswitch defaults will opt in and call the helper once they are ready. This keeps the support explicit per driver and avoids making it depend on where
devl_register()happens in the init path.With that, devlink can tell at register time whether the instance supports boot time eswitch defaults. If the user configured a default for an instance whose driver did not opt in, devlink can write to dmesg from
devl_register().
Quoted 4 lines
Not perfect, but at least the user gets a visible failure instead of the config being silently ignored.
Mark
Jakub, Jiri, any thoughts?
I think the explicit helper is the cleanest option here, without any workqueue fallback inside devlink. It avoids depending on devl_register() ordering, and makes the support explicit per driver.
Does that sound like an acceptable direction?
Mark
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.
For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.
Jakub Kicinski
maintainerRe: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init
Quoted 5 lines · On Tue, 16 Jun 2026 20:29:25 +0300 Mark Bloch wrote:
I think the explicit helper is the cleanest option here, without any workqueue fallback inside devlink. It avoids depending on
devl_register()ordering, and makes the support explicit per driver.Does that sound like an acceptable direction?
I'd much rather have the workqueue with the purely theoretical race with user space than a bunch of drivers that don't act on the cmdline params.
Mark Bloch
maintainerPATCH v3 3/7 net/mlx5: Clear FW reset-in-progress bit before reload
In reply to Mark Blochmlx5 sets MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS when acknowledging a sync reset request. This bit blocks devlink reload and other devlink operations while the firmware reset is running, but it was kept set until after the driver reload finished.
Clear the reset-in-progress bit once the reset unload flow is done and PCI access is back, before reloading the device. For a reset initiated through devlink, clear it before completing the reload waiter. For a reset reported through an asynchronous firmware event, keep the unload flow outside devl_lock, then take devl_lock before clearing the bit and reloading through the devl-locked load helper.
--- .../ethernet/mellanox/mlx5/core/fw_reset.c | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +17 -11
@@ -238,24 +238,30 @@ static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev) {struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;struct devlink *devlink =priv_to_devlink (dev); +int err;/* if this is the driver that initiated the fw reset, devlink completed the reload */ if (test_bit (MLX5_FW_RESET_FLAGS_PENDING_COMP , &fw_reset->reset_flags)) { +clear_bit (MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS , + &fw_reset->reset_flags);complete (&fw_reset->done); - }else { -mlx5_sync_reset_unload_flow (dev, false); -if (mlx5_health_wait_pci_up (dev)) -mlx5_core_err (dev,"reset reload flow aborted, PCI reads still not working\n" ); -else -mlx5_load_one (dev, true); -devl_lock (devlink); -devlink_remote_reload_actions_performed (devlink,0 , -BIT (DEVLINK_RELOAD_ACTION_DRIVER_REINIT ) | -BIT (DEVLINK_RELOAD_ACTION_FW_ACTIVATE )); -devl_unlock (devlink); +return ; } +mlx5_sync_reset_unload_flow (dev, false); + err =mlx5_health_wait_pci_up (dev); + +devl_lock (devlink);clear_bit (MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS , &fw_reset->reset_flags); +if (err) +mlx5_core_err (dev,"reset reload flow aborted, PCI reads still not working\n" ); +else +mlx5_load_one_devl_locked (dev, true); + +devlink_remote_reload_actions_performed (devlink,0 , +BIT (DEVLINK_RELOAD_ACTION_DRIVER_REINIT ) | +BIT (DEVLINK_RELOAD_ACTION_FW_ACTIVATE )); +devl_unlock (devlink); }static void mlx5_stop_sync_reset_poll (struct mlx5_core_dev *dev)
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 4/7 net/mlx5: Register devlink after device init
In reply to Mark Blochdevl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.
Move mlx5 devlink registration after mlx5 device initialization completes, including the lightweight init path, so registration-time devlink operations see initialized driver state.
Move devl_unregister() before the matching teardown paths, so unregister notifications are emitted from devl_unregister() before mlx5 removes the devlink objects.
Add a devl-locked uninit helper so failed nested devlink setup can unwind the initialized device before the instance is registered.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 34 ++++++++++++++----- 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +25 -9
@@ -1454,31 +1454,40 @@ int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev)return err; } +static void mlx5_uninit_one_devl_locked (struct mlx5_core_dev *dev); +int mlx5_init_one (struct mlx5_core_dev *dev) {struct devlink *devlink =priv_to_devlink (dev);int err;devl_lock (devlink); + err =mlx5_init_one_devl_locked (dev); +if (err) + goto unlock; +if (dev->shd) { err =devl_nested_devlink_set (dev->shd, devlink);if (err) - goto unlock; + goto err_uninit; } +devl_register (devlink); - err =mlx5_init_one_devl_locked (dev); -if (err) -devl_unregister (devlink); +devl_unlock (devlink); +return 0 ; + +err_uninit: +mlx5_uninit_one_devl_locked (dev); unlock:devl_unlock (devlink);return err; } -void mlx5_uninit_one (struct mlx5_core_dev *dev) +static void mlx5_uninit_one_devl_locked (struct mlx5_core_dev *dev) {struct devlink *devlink =priv_to_devlink (dev); -devl_lock (devlink); +devl_assert_locked (devlink);mutex_lock (&dev->intf_state_mutex);mlx5_hwmon_dev_unregister (dev);
@@ -1501,7 +1510,15 @@ void mlx5_uninit_one(struct mlx5_core_dev *dev)mlx5_function_teardown (dev, true); out:mutex_unlock (&dev->intf_state_mutex); +} + +void mlx5_uninit_one (struct mlx5_core_dev *dev) +{ +struct devlink *devlink =priv_to_devlink (dev); + +devl_lock (devlink);devl_unregister (devlink); +mlx5_uninit_one_devl_locked (dev);devl_unlock (devlink); }
@@ -1636,7 +1653,6 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)int err;devl_lock (devlink); -devl_register (devlink); dev->state =MLX5_DEVICE_STATE_UP ; err =mlx5_function_enable (dev, true,mlx5_tout_ms (dev,FW_PRE_INIT_TIMEOUT ));if (err) {
@@ -1656,6 +1672,7 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev) goto query_hca_caps_err; } +devl_register (devlink);devl_unlock (devlink);return 0 ;
@@ -1663,7 +1680,6 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)mlx5_function_disable (dev, true); out: dev->state =MLX5_DEVICE_STATE_INTERNAL_ERROR ; -devl_unregister (devlink);devl_unlock (devlink);return err; }
@@ -1673,8 +1689,8 @@ void mlx5_uninit_one_light(struct mlx5_core_dev *dev)struct devlink *devlink =priv_to_devlink (dev);devl_lock (devlink); -mlx5_devlink_params_unregister (priv_to_devlink (dev));devl_unregister (devlink); +mlx5_devlink_params_unregister (priv_to_devlink (dev));devl_unlock (devlink);if (dev->state !=MLX5_DEVICE_STATE_UP )return ;
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 5/7 octeontx2-af: Register devlink after SR-IOV init
In reply to Mark BlochA later patch makes devlink registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.
Move octeontx2 AF devlink registration after SR-IOV is enabled and the representor switch lock is initialized, so the AF eswitch mode set path sees the state it depends on.
If devlink registration fails after SR-IOV setup, unregister interrupts before disabling SR-IOV. This keeps the AF-VF mailbox IRQ handlers synchronized before the AF-VF mailbox workqueue is destroyed.
--- .../net/ethernet/marvell/octeontx2/af/rvu.c | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/net/ethernet/marvell/octeontx2/af/rvu.c +13 -11
@@ -3545,6 +3545,7 @@ static void rvu_update_module_params(struct rvu *rvu)static int rvu_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct device *dev = &pdev->dev; +bool sriov_done = false;struct rvu *rvu;int err;
@@ -3634,26 +3635,27 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id) goto err_flr; } - err =rvu_register_dl (rvu); -if (err) { -dev_err (dev,"%s: Failed to register devlink\n" , __func__); - goto err_irq; - } -rvu_setup_rvum_blk_revid (rvu);/* Enable AF's VFs (if any) */ err =rvu_enable_sriov (rvu);if (err) {dev_err (dev,"%s: Failed to enable sriov\n" , __func__); - goto err_dl; + goto err_irq; + } + sriov_done = true; + +mutex_init (&rvu->rswitch.switch_lock); + + err =rvu_register_dl (rvu); +if (err) { +dev_err (dev,"%s: Failed to register devlink\n" , __func__); + goto err_irq; }/* Initialize debugfs */ rvu_dbg_init (rvu); -mutex_init (&rvu->rswitch.switch_lock); -if (rvu->fwdata)ptp_start (rvu, rvu->fwdata->sclk, rvu->fwdata->ptp_ext_clk_rate, rvu->fwdata->ptp_ext_tstamp);
@@ -3662,10 +3664,10 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)rvu_alloc_cint_qint_mem (rvu, &rvu->pf[RVU_AFPF ],BLKADDR_NIX0 , (rvu->hw->block[BLKADDR_NIX0 ].lf.max));return 0 ; -err_dl: -rvu_unregister_dl (rvu); err_irq:rvu_unregister_interrupts (rvu); +if (sriov_done) +rvu_disable_sriov (rvu); err_flr:rvu_flr_wq_destroy (rvu); err_mbox:
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 6/7 octeontx2-pf: Register devlink after SR-IOV state init
In reply to Mark BlochA later patch makes devlink registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.
Move octeontx2 PF devlink registration after PF SR-IOV configuration state is initialized, so representor creation has the state it needs.
Add a separate unwind label so failures after devlink registration unregister devlink before cleaning up SR-IOV state.
--- .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +9 -8
@@ -3278,14 +3278,14 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (err) goto err_mcam_flow_del; - err =otx2_register_dl (pf); -if (err) - goto err_mcam_flow_del; -/* Initialize SR-IOV resources */ err =otx2_sriov_vfcfg_init (pf);if (err) - goto err_pf_sriov_init; + goto err_shutdown_tc; + + err =otx2_register_dl (pf); +if (err) + goto err_sriov_cleannup;/* Enable link notifications */ otx2_cgx_config_linkevents (pf, true);
@@ -3293,7 +3293,7 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id) pf->af_xdp_zc_qidx =bitmap_zalloc (qcount,GFP_KERNEL );if (!pf->af_xdp_zc_qidx) { err = -ENOMEM ; - goto err_sriov_cleannup; + goto err_dl_unregister; }#ifdef CONFIG_DCB
@@ -3310,10 +3310,11 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id) err_free_zc_bmap:bitmap_free (pf->af_xdp_zc_qidx);#endif +err_dl_unregister: +otx2_unregister_dl (pf); err_sriov_cleannup:otx2_sriov_vfcfg_cleanup (pf); -err_pf_sriov_init: -otx2_unregister_dl (pf); +err_shutdown_tc:otx2_shutdown_tc (pf); err_mcam_flow_del:otx2_mcam_flow_del (pf);
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 7/7 devlink: Add eswitch mode boot defaults
In reply to Mark BlochAdd devlink_eswitch_mode= command line support for setting a default eswitch mode during device initialization.
The supported syntax selects either all devlink handles or one explicit comma-separated handle list:
devlink_eswitch_mode=*=<mode> devlink_eswitch_mode=<handle>[,<handle>...]=<mode>
where <mode> is one of legacy, switchdev or switchdev_inactive. All selected handles receive the same mode. Assigning different modes to different handle lists in the same parameter value is not supported.
The default is applied through the existing eswitch_mode_set() devlink operation, matching the userspace devlink eswitch mode command. devlink core applies it when a matching devlink instance is registered and after a successful devlink reload that performed DRIVER_REINIT, so rebuilt device state returns to the requested boot default.
Document the devlink_eswitch_mode= syntax and duplicate handle handling.
--- .../admin-guide/kernel-parameters.txt | 25 ++ .../networking/devlink/devlink-defaults.rst | 78 +++++ Documentation/networking/devlink/index.rst | 1 + net/devlink/core.c | 271 ++++++++++++++++++ net/devlink/dev.c | 3 + net/devlink/devl_internal.h | 1 + 6 files changed, 379 insertions(+) create mode 100644 Documentation/networking/devlink/devlink-defaults.rst
Documentation/admin-guide/kernel-parameters.txt +25 -0
@@ -1246,6 +1246,31 @@ Kernel parameters dell_smm_hwmon.fan_max= [HW] Maximum configurable fan speed. + devlink_eswitch_mode= + [NET] + Format: + <selector>=<mode> + + <selector>: + * | <handle>[,<handle>...] + + <handle>: + <bus-name>/<dev-name> + + Configure default devlink eswitch mode for matching + devlink instances during device initialization. + + <mode>: + legacy | switchdev | switchdev_inactive + + Examples: + devlink_eswitch_mode=*=switchdev + devlink_eswitch_mode=pci/0000:08:00.0=switchdev + devlink_eswitch_mode=pci/0000:08:00.0,pci/0000:09:00.1=switchdev_inactive + + See Documentation/networking/devlink/devlink-defaults.rst + for the full syntax. + dfltcc= [HW,S390] Format: { on | off | def_only | inf_only | always } on: s390 zlib hardware support for compression on
Documentation/networking/devlink/devlink-defaults.rst new file +78 -0
78 lines not on this page · raw message
Documentation/networking/devlink/index.rst +1 -0
@@ -56,6 +56,7 @@ general. :maxdepth: 1 devlink-dpipe + devlink-defaults devlink-eswitch-attr devlink-flash devlink-health
net/devlink/core.c +271 -0
308 lines not on this page · raw message
net/devlink/dev.c +3 -0
@@ -478,6 +478,9 @@ int devlink_reload(struct devlink *devlink, struct net *dest_net,return err;WARN_ON (!(*actions_performed &BIT (action))); +if (*actions_performed &BIT (DEVLINK_RELOAD_ACTION_DRIVER_REINIT )) +devlink_apply_default_esw_mode (devlink); +/* Catch driver on updating the remote action within devlink reload */ WARN_ON (memcmp (remote_reload_stats, devlink->stats.remote_reload_stats,sizeof (remote_reload_stats)));
net/devlink/devl_internal.h +1 -0
@@ -71,6 +71,7 @@ extern struct genl_family devlink_nl_family;struct devlink *__devlink_alloc (const struct devlink_ops *ops,size_t priv_size,struct net *net,struct device *dev,const struct device_driver *dev_driver); +void devlink_apply_default_esw_mode (struct devlink *devlink); #define devl_warn(devlink, format, args...) \ do { \
Signature · 1 line
2.34.1
Quoted 2 lines · Mark Bloch
This series adds a devlink_eswitch_mode= kernel command line parameter for applying a default devlink eswitch mode during device initialization.
Please trim your Cc list. There's nothing I can do about networking. You're CCing too many people unnecessarily.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette