Skip to contentSkip to conversation

devlink: Add boot-time eswitch mode defaults

PATCH Superseded Mark Bloch 3 participants 15 messages Jun 5 – Jun 16

Subsystems (9)

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Mark Bloch

author maintainer
#0 lore raw

v3 cover devlink: Add boot-time eswitch mode defaults

This series adds a devlink_eswitch_mode= kernel command line parameter for applying a default devlink eswitch mode during device initialization.

Following the discussion with Jakub[1] and the feedback on the RFC postings, this version keeps the scope limited to a boot-time devlink eswitch mode default only.

The option selects either all devlink handles or an explicit comma separated handle list:

devlink_eswitch_mode=*=switchdev devlink_eswitch_mode=pci/0000:08:00.0,pci/0000:09:00.1=switchdev_inactive

The supported modes are legacy, switchdev and switchdev_inactive. The selected mode is applied through the existing eswitch_mode_set() devlink operation, the same operation used by the devlink eswitch mode command.

The preparatory patches move registration points that expose the devlink instance before the driver is ready for a registration-time eswitch mode change. Where registration is moved later, the matching unregister path is moved earlier so unregister notifications are sent from devl_unregister() before object teardown. The final patch adds the parser and applies the default from devlink core when a matching instance is registered and after a successful devlink reload that performed DRIVER_REINIT.

Patch 1 skips devlink health recovery notifications while a devlink instance is not registered. Health state and counters are still updated, but there is no registered instance for userspace to observe or receive notifications from yet. This lets drivers move registration later without hitting health notification registration assertions during early initialization.

Patch 2 moves netdevsim devlink registration after device initialization, so registration-time defaults can call eswitch_mode_set() after simulator state is ready. It also unregisters devlink before netdevsim tears down the objects that were registered before devlink became visible.

Patch 3 clears the mlx5 FW reset-in-progress bit before reloading after a firmware reset.

Patch 4 moves mlx5 devlink registration after device initialization, including the lightweight init path, and moves unregister before the matching teardown.

Patch 5 moves octeontx2 AF devlink registration after SR-IOV setup and switch lock initialization.

Patch 6 moves octeontx2 PF devlink registration after PF SR-IOV state setup.

Patch 7 adds the devlink_eswitch_mode= parser, documentation, registration-time default application and successful reload default application.

Changelog:

v2 -> v3:

- Change the devlink_eswitch_mode= API syntax to use <selector>=<mode> instead of [<selector>]:<mode>, following a comment from Randy Dunlap.

v1 -> v2:

- Move default eswitch mode application into devlink core. The default is now applied during devlink registration and after a successful devlink reload that performed DRIVER_REINIT.

- Remove the exported devl_apply_default_esw_mode() driver API and the mlx5 driver-side call to it.

- Skip devlink health recovery notifications while the devlink instance is not registered, so drivers can move registration later without early health work hitting registration assertions.

- Move mlx5 devlink registration after device initialization, including the lightweight init path, so the core can apply the default through the normal registration flow.

- Move the matching netdevsim and mlx5 unregister paths before object teardown, so unregister notifications come from devl_unregister() and the later object teardown paths run while the devlink instance is no longer registered.

- Add registration-ordering preparation patches for netdevsim and octeontx2 AF/PF, so their eswitch state is ready before registration-time defaults may call eswitch_mode_set().

[1] https://lore.kernel.org/all/[email protected]/
RFC V1 : https://lore.kernel.org/all/[email protected]/
RFC V2 : https://lore.kernel.org/all/[email protected]/
v1 : https://lore.kernel.org/all/[email protected]/
v2 : https://lore.kernel.org/all/[email protected]/

Mark Bloch (7):
devlink: Skip health recover notifications before register
netdevsim: Register devlink after device init net/mlx5: Clear FW reset-in-progress bit before reload net/mlx5: Register devlink after device init
octeontx2-af: Register devlink after SR-IOV init
octeontx2-pf: Register devlink after SR-IOV state init
devlink: Add eswitch mode boot defaults

 .../admin-guide/kernel-parameters.txt         |  25 ++
 .../networking/devlink/devlink-defaults.rst   |  78 +++++
 Documentation/networking/devlink/index.rst    |   1 +
 .../net/ethernet/marvell/octeontx2/af/rvu.c   |  24 +-
 .../ethernet/marvell/octeontx2/nic/otx2_pf.c  |  17 +-
 .../ethernet/mellanox/mlx5/core/fw_reset.c    |  28 +-
 .../net/ethernet/mellanox/mlx5/core/main.c    |  34 ++-
 drivers/net/netdevsim/dev.c                   |  15 +-
 net/devlink/core.c                            | 271 ++++++++++++++++++
 net/devlink/dev.c                             |   3 +
 net/devlink/devl_internal.h                   |   1 +
 net/devlink/health.c                          |   3 +-
 12 files changed, 451 insertions(+), 49 deletions(-)
 create mode 100644 Documentation/networking/devlink/devlink-defaults.rst

base-commit: bfa3d89cc15c09f7d1581c834a5ed725189ec19f

Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v3 1/7 devlink: Skip health recover notifications before register

devlink health reports can be generated before the devlink instance is registered. This can happen during driver initialization when a driver creates health reporters early and its health polling detects an error before devlink_register() is reached.

devlink health still records the report state and counters in that case, but userspace cannot observe the devlink instance yet and there is no registered handle to notify through. Skip the netlink notification while the devlink instance is not registered instead of asserting registration.

This keeps later userspace queries useful after registration while avoiding a warning from early health reports.

---
 net/devlink/health.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
net/devlink/health.c +1 -2
index ea7a334e939b..376e79497771 100644
@@ -513,9 +513,8 @@ static void devlink_recover_notify(struct devlink_health_reporter *reporter,
 	int err;
 
 	WARN_ON(cmd != DEVLINK_CMD_HEALTH_REPORTER_RECOVER);
-	ASSERT_DEVLINK_REGISTERED(devlink);
 
-	if (!devlink_nl_notify_need(devlink))
+	if (!__devl_is_registered(devlink) || !devlink_nl_notify_need(devlink))
 		return;
 
 	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH v3 2/7 netdevsim: Register devlink after device init

In reply to Mark Bloch

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

---
 drivers/net/netdevsim/dev.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
drivers/net/netdevsim/dev.c +7 -8
index aed9ad5f1b43..7cf4102b049e 100644
@@ -1680,13 +1680,9 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 		goto err_devlink_unlock;
 	}
 
-	err = devl_register(devlink);
-	if (err)
-		goto err_vfc_free;
-
 	err = nsim_dev_resources_register(devlink);
 	if (err)
-		goto err_dl_unregister;
+		goto err_vfc_free;
 
 	err = devl_params_register(devlink, nsim_devlink_params,
 				   ARRAY_SIZE(nsim_devlink_params));
@@ -1733,9 +1729,14 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 		goto err_hwstats_exit;
 
 	nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_LEGACY;
+	err = devl_register(devlink);
+	if (err)
+		goto err_port_del_all;
 	devl_unlock(devlink);
 	return 0;
 
+err_port_del_all:
+	nsim_dev_port_del_all(nsim_dev);
 err_hwstats_exit:
 	nsim_dev_hwstats_exit(nsim_dev);
 err_psample_exit:
@@ -1757,8 +1758,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 			       ARRAY_SIZE(nsim_devlink_params));
 err_resource_unregister:
 	devl_resources_unregister(devlink);
-err_dl_unregister:
-	devl_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
 err_devlink_unlock:
@@ -1797,6 +1796,7 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 	struct devlink *devlink = priv_to_devlink(nsim_dev);
 
 	devl_lock(devlink);
+	devl_unregister(devlink);
 	nsim_dev_reload_destroy(nsim_dev);
 
 	nsim_bpf_dev_exit(nsim_dev);
@@ -1804,7 +1804,6 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 	devl_params_unregister(devlink, nsim_devlink_params,
 			       ARRAY_SIZE(nsim_devlink_params));
 	devl_resources_unregister(devlink);
-	devl_unregister(devlink);
 	kfree(nsim_dev->vfconfigs);
 	kfree(nsim_dev->fa_cookie);
 	mutex_destroy(&nsim_dev->progs_list_lock);
Signature · 1 line
2.34.1
maintainer
#3 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Quoted 5 lines

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.

author maintainer
#4 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Quoted 5 lines

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

Quoted 8 lines

This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.

Let me clarify.

No locking is changed here, and I don't want to make register/unregister ordering a substitute for devl_lock().

The only requirement I have for this series is that devl_register() is called only once the driver is ready for devlink core to call eswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode from devl_register() instead of adding an explicit driver call.

So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.

For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.

Mark

maintainer
#5 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Quoted 5 lines

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

Quoted 8 lines

This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.

Let me clarify.

Quoted 7 lines

No locking is changed here, and I don't want to make register/unregister ordering a substitute for devl_lock().

The only requirement I have for this series is that devl_register() is called only once the driver is ready for devlink core to call eswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode from devl_register() instead of adding an explicit driver call.

This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.

If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once devl_lock() is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.

Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...

Quoted 7 lines

So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.

For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.

author maintainer
#6 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 36 lines · Jakub Kicinski
Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Quoted 5 lines

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

Quoted 8 lines

This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.

Let me clarify.

Quoted 7 lines

No locking is changed here, and I don't want to make register/unregister ordering a substitute for devl_lock().

The only requirement I have for this series is that devl_register() is called only once the driver is ready for devlink core to call eswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode from devl_register() instead of adding an explicit driver call.

Quoted 4 lines

This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.

Right, the reason I moved in this direction is that in v1 I had the explicit driver call, and Jiri asked to make this transparent from devlink core instead.

Quoted 7 lines

If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once devl_lock() is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.

The workqueue fallback is possible, but I think it makes the semantics more complicated.

We would need to track devlink instances which still need the default applied, and the worker would have to skip/remove them once handled.

More importantly, the worker can race with userspace setting the eswitch mode, so we would also need some state to tell whether the user already changed the mode. That feels more fragile than an explicit driver call.

Quoted 4 lines

Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...

That's why I started with the explicit call in the first place.

I can switch back to this model: drivers which support boot time eswitch defaults will opt in and call the helper once they are ready. This keeps the support explicit per driver and avoids making it depend on where devl_register() happens in the init path.

With that, devlink can tell at register time whether the instance supports boot time eswitch defaults. If the user configured a default for an instance whose driver did not opt in, devlink can write to dmesg from devl_register().

Not perfect, but at least the user gets a visible failure instead of the config being silently ignored.

Mark

Quoted 8 lines
Quoted 8 lines

So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.

For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.

author maintainer
#7 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 84 lines · Mark Bloch
Quoted 36 lines · Jakub Kicinski
Quoted 27 lines · On Thu, 11 Jun 2026 09:02:03 +0300 Mark Bloch wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 5 Jun 2026 21:10:25 +0300 Mark Bloch wrote:

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move netdevsim registration after all objects (resources, params, regions, traps, debugfs etc) are initialized, and after the initial eswitch mode is set to legacy.

Quoted 5 lines

Move devl_unregister() to the beginning of nsim_drv_remove(), before those devlink objects are torn down. This keeps devlink register/unregister as the notification barrier and makes the later object teardown paths run after devlink is no longer registered, so they do not emit their own netlink DEL notifications.

Quoted 8 lines

This is going backwards. At some point someone from nVidia thought that we can order our way out of locking, so mlx5 is likely ordered this way, but this must not be required, or in any way normalized.
We (syzbot) quickly discovered that it doesn't cover all corner cases. devl_lock() is exposed specifically to allow the driver to finish whatever init it needs without letting user space invoke callbacks, yet. Almost (?) all driver callbacks hold devl_lock(), so maybe the devlink instance is "visible" to user space but that should not matter.

Let me clarify.

Quoted 7 lines

No locking is changed here, and I don't want to make register/unregister ordering a substitute for devl_lock().

The only requirement I have for this series is that devl_register() is called only once the driver is ready for devlink core to call eswitch_mode_set(). That follows from the earlier direction to have the core apply the default mode from devl_register() instead of adding an explicit driver call.

Quoted 4 lines

This is exactly what I'm objecting to. AFAIU we are trading off explicit call to get the default value for an implicit behavior depending on order of calls. We want to optimize for how easy it is to get the API wrong, not for LoC.

Right, the reason I moved in this direction is that in v1 I had the explicit driver call, and Jiri asked to make this transparent from devlink core instead.

Quoted 7 lines

If we don't have a clean way to implement this without driver changes let's add the explicit API to get the default value. If driver doesn't call it schedule a work to go via the callback once devl_lock() is dropped. That way drivers which care can optimize themselves by reading the default value upfront. Drivers which don't care will work correctly, and there's no API call order trap.

The workqueue fallback is possible, but I think it makes the semantics more complicated.

We would need to track devlink instances which still need the default applied, and the worker would have to skip/remove them once handled.

More importantly, the worker can race with userspace setting the eswitch mode, so we would also need some state to tell whether the user already changed the mode. That feels more fragile than an explicit driver call.

Quoted 4 lines

Not ideal, but isn't that best we can do here?
I still have flashbacks of the fallout from the call ordering games, we have too many drivers to keep this straight...

That's why I started with the explicit call in the first place.

I can switch back to this model: drivers which support boot time eswitch defaults will opt in and call the helper once they are ready. This keeps the support explicit per driver and avoids making it depend on where devl_register() happens in the init path.

With that, devlink can tell at register time whether the instance supports boot time eswitch defaults. If the user configured a default for an instance whose driver did not opt in, devlink can write to dmesg from devl_register().

Quoted 4 lines

Not perfect, but at least the user gets a visible failure instead of the config being silently ignored.

Mark

Jakub, Jiri, any thoughts?

I think the explicit helper is the cleanest option here, without any workqueue fallback inside devlink. It avoids depending on devl_register() ordering, and makes the support explicit per driver.

Does that sound like an acceptable direction?

Mark

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

So if the objection is to the commit message wording, I can fix that and drop the "notification barrier" language.

For unregister, I can probably leave the old ordering as-is. I moved it only to mirror the register path, which felt cleaner, but it is not required for the default-mode change and as the lock is held I see no issue with doing that.

maintainer
#8 lore raw

Re: [PATCH net-next V3 2/7] netdevsim: Register devlink after device init

Quoted 5 lines · On Tue, 16 Jun 2026 20:29:25 +0300 Mark Bloch wrote:

I think the explicit helper is the cleanest option here, without any workqueue fallback inside devlink. It avoids depending on devl_register() ordering, and makes the support explicit per driver.

Does that sound like an acceptable direction?

I'd much rather have the workqueue with the purely theoretical race with user space than a bunch of drivers that don't act on the cmdline params.

author maintainer
#9 lore raw

PATCH v3 3/7 net/mlx5: Clear FW reset-in-progress bit before reload

In reply to Mark Bloch

mlx5 sets MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS when acknowledging a sync reset request. This bit blocks devlink reload and other devlink operations while the firmware reset is running, but it was kept set until after the driver reload finished.

Clear the reset-in-progress bit once the reset unload flow is done and PCI access is back, before reloading the device. For a reset initiated through devlink, clear it before completing the reload waiter. For a reset reported through an asynchronous firmware event, keep the unload flow outside devl_lock, then take devl_lock before clearing the bit and reloading through the devl-locked load helper.

---
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 28 +++++++++++--------
 1 file changed, 17 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +17 -11
index 07440c58713a..7283e5b49eed 100644
@@ -238,24 +238,30 @@ static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
 {
 	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
 	struct devlink *devlink = priv_to_devlink(dev);
+	int err;
 
 	/* if this is the driver that initiated the fw reset, devlink completed the reload */
 	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
+		clear_bit(MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS,
+			  &fw_reset->reset_flags);
 		complete(&fw_reset->done);
-	} else {
-		mlx5_sync_reset_unload_flow(dev, false);
-		if (mlx5_health_wait_pci_up(dev))
-			mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
-		else
-			mlx5_load_one(dev, true);
-		devl_lock(devlink);
-		devlink_remote_reload_actions_performed(devlink, 0,
-							BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
-							BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
-		devl_unlock(devlink);
+		return;
 	}
 
+	mlx5_sync_reset_unload_flow(dev, false);
+	err = mlx5_health_wait_pci_up(dev);
+
+	devl_lock(devlink);
 	clear_bit(MLX5_FW_RESET_FLAGS_RESET_IN_PROGRESS, &fw_reset->reset_flags);
+	if (err)
+		mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
+	else
+		mlx5_load_one_devl_locked(dev, true);
+
+	devlink_remote_reload_actions_performed(devlink, 0,
+						BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
+						BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
+	devl_unlock(devlink);
 }
 
 static void mlx5_stop_sync_reset_poll(struct mlx5_core_dev *dev)
Signature · 1 line
2.34.1
author maintainer
#10 lore raw

PATCH v3 4/7 net/mlx5: Register devlink after device init

In reply to Mark Bloch

devl_register() makes the devlink instance visible to userspace. A later patch also makes registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move mlx5 devlink registration after mlx5 device initialization completes, including the lightweight init path, so registration-time devlink operations see initialized driver state.

Move devl_unregister() before the matching teardown paths, so unregister notifications are emitted from devl_unregister() before mlx5 removes the devlink objects.

Add a devl-locked uninit helper so failed nested devlink setup can unwind the initialized device before the instance is registered.

---
 .../net/ethernet/mellanox/mlx5/core/main.c    | 34 ++++++++++++++-----
 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +25 -9
index fd285aeb9630..4e3cb6ec8630 100644
@@ -1454,31 +1454,40 @@ int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev)
 	return err;
 }
 
+static void mlx5_uninit_one_devl_locked(struct mlx5_core_dev *dev);
+
 int mlx5_init_one(struct mlx5_core_dev *dev)
 {
 	struct devlink *devlink = priv_to_devlink(dev);
 	int err;
 
 	devl_lock(devlink);
+	err = mlx5_init_one_devl_locked(dev);
+	if (err)
+		goto unlock;
+
 	if (dev->shd) {
 		err = devl_nested_devlink_set(dev->shd, devlink);
 		if (err)
-			goto unlock;
+			goto err_uninit;
 	}
+
 	devl_register(devlink);
-	err = mlx5_init_one_devl_locked(dev);
-	if (err)
-		devl_unregister(devlink);
+	devl_unlock(devlink);
+	return 0;
+
+err_uninit:
+	mlx5_uninit_one_devl_locked(dev);
 unlock:
 	devl_unlock(devlink);
 	return err;
 }
 
-void mlx5_uninit_one(struct mlx5_core_dev *dev)
+static void mlx5_uninit_one_devl_locked(struct mlx5_core_dev *dev)
 {
 	struct devlink *devlink = priv_to_devlink(dev);
 
-	devl_lock(devlink);
+	devl_assert_locked(devlink);
 	mutex_lock(&dev->intf_state_mutex);
 
 	mlx5_hwmon_dev_unregister(dev);
@@ -1501,7 +1510,15 @@ void mlx5_uninit_one(struct mlx5_core_dev *dev)
 	mlx5_function_teardown(dev, true);
 out:
 	mutex_unlock(&dev->intf_state_mutex);
+}
+
+void mlx5_uninit_one(struct mlx5_core_dev *dev)
+{
+	struct devlink *devlink = priv_to_devlink(dev);
+
+	devl_lock(devlink);
 	devl_unregister(devlink);
+	mlx5_uninit_one_devl_locked(dev);
 	devl_unlock(devlink);
 }
 
@@ -1636,7 +1653,6 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)
 	int err;
 
 	devl_lock(devlink);
-	devl_register(devlink);
 	dev->state = MLX5_DEVICE_STATE_UP;
 	err = mlx5_function_enable(dev, true, mlx5_tout_ms(dev, FW_PRE_INIT_TIMEOUT));
 	if (err) {
@@ -1656,6 +1672,7 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)
 		goto query_hca_caps_err;
 	}
 
+	devl_register(devlink);
 	devl_unlock(devlink);
 	return 0;
 
@@ -1663,7 +1680,6 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)
 	mlx5_function_disable(dev, true);
 out:
 	dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR;
-	devl_unregister(devlink);
 	devl_unlock(devlink);
 	return err;
 }
@@ -1673,8 +1689,8 @@ void mlx5_uninit_one_light(struct mlx5_core_dev *dev)
 	struct devlink *devlink = priv_to_devlink(dev);
 
 	devl_lock(devlink);
-	mlx5_devlink_params_unregister(priv_to_devlink(dev));
 	devl_unregister(devlink);
+	mlx5_devlink_params_unregister(priv_to_devlink(dev));
 	devl_unlock(devlink);
 	if (dev->state != MLX5_DEVICE_STATE_UP)
 		return;
Signature · 1 line
2.34.1
author maintainer
#11 lore raw

PATCH v3 5/7 octeontx2-af: Register devlink after SR-IOV init

In reply to Mark Bloch

A later patch makes devlink registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move octeontx2 AF devlink registration after SR-IOV is enabled and the representor switch lock is initialized, so the AF eswitch mode set path sees the state it depends on.

If devlink registration fails after SR-IOV setup, unregister interrupts before disabling SR-IOV. This keeps the AF-VF mailbox IRQ handlers synchronized before the AF-VF mailbox workqueue is destroyed.

---
 .../net/ethernet/marvell/octeontx2/af/rvu.c   | 24 ++++++++++---------
 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/net/ethernet/marvell/octeontx2/af/rvu.c +13 -11
index 3cf131508ecf..c2b52eb4ffab 100644
@@ -3545,6 +3545,7 @@ static void rvu_update_module_params(struct rvu *rvu)
 static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct device *dev = &pdev->dev;
+	bool sriov_done = false;
 	struct rvu *rvu;
 	int    err;
 
@@ -3634,26 +3635,27 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto err_flr;
 	}
 
-	err = rvu_register_dl(rvu);
-	if (err) {
-		dev_err(dev, "%s: Failed to register devlink\n", __func__);
-		goto err_irq;
-	}
-
 	rvu_setup_rvum_blk_revid(rvu);
 
 	/* Enable AF's VFs (if any) */
 	err = rvu_enable_sriov(rvu);
 	if (err) {
 		dev_err(dev, "%s: Failed to enable sriov\n", __func__);
-		goto err_dl;
+		goto err_irq;
+	}
+	sriov_done = true;
+
+	mutex_init(&rvu->rswitch.switch_lock);
+
+	err = rvu_register_dl(rvu);
+	if (err) {
+		dev_err(dev, "%s: Failed to register devlink\n", __func__);
+		goto err_irq;
 	}
 
 	/* Initialize debugfs */
 	rvu_dbg_init(rvu);
 
-	mutex_init(&rvu->rswitch.switch_lock);
-
 	if (rvu->fwdata)
 		ptp_start(rvu, rvu->fwdata->sclk, rvu->fwdata->ptp_ext_clk_rate,
 			  rvu->fwdata->ptp_ext_tstamp);
@@ -3662,10 +3664,10 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	rvu_alloc_cint_qint_mem(rvu, &rvu->pf[RVU_AFPF], BLKADDR_NIX0,
 				(rvu->hw->block[BLKADDR_NIX0].lf.max));
 	return 0;
-err_dl:
-	rvu_unregister_dl(rvu);
 err_irq:
 	rvu_unregister_interrupts(rvu);
+	if (sriov_done)
+		rvu_disable_sriov(rvu);
 err_flr:
 	rvu_flr_wq_destroy(rvu);
 err_mbox:
Signature · 1 line
2.34.1
author maintainer
#12 lore raw

PATCH v3 6/7 octeontx2-pf: Register devlink after SR-IOV state init

In reply to Mark Bloch

A later patch makes devlink registration the point where devlink core may call eswitch_mode_set() to apply a boot-time default eswitch mode.

Move octeontx2 PF devlink registration after PF SR-IOV configuration state is initialized, so representor creation has the state it needs.

Add a separate unwind label so failures after devlink registration unregister devlink before cleaning up SR-IOV state.

---
 .../ethernet/marvell/octeontx2/nic/otx2_pf.c    | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +9 -8
index f9fbf0c17648..47d2c6a24636 100644
@@ -3278,14 +3278,14 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (err)
 		goto err_mcam_flow_del;
 
-	err = otx2_register_dl(pf);
-	if (err)
-		goto err_mcam_flow_del;
-
 	/* Initialize SR-IOV resources */
 	err = otx2_sriov_vfcfg_init(pf);
 	if (err)
-		goto err_pf_sriov_init;
+		goto err_shutdown_tc;
+
+	err = otx2_register_dl(pf);
+	if (err)
+		goto err_sriov_cleannup;
 
 	/* Enable link notifications */
 	otx2_cgx_config_linkevents(pf, true);
@@ -3293,7 +3293,7 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	pf->af_xdp_zc_qidx = bitmap_zalloc(qcount, GFP_KERNEL);
 	if (!pf->af_xdp_zc_qidx) {
 		err = -ENOMEM;
-		goto err_sriov_cleannup;
+		goto err_dl_unregister;
 	}
 
 #ifdef CONFIG_DCB
@@ -3310,10 +3310,11 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 err_free_zc_bmap:
 	bitmap_free(pf->af_xdp_zc_qidx);
 #endif
+err_dl_unregister:
+	otx2_unregister_dl(pf);
 err_sriov_cleannup:
 	otx2_sriov_vfcfg_cleanup(pf);
-err_pf_sriov_init:
-	otx2_unregister_dl(pf);
+err_shutdown_tc:
 	otx2_shutdown_tc(pf);
 err_mcam_flow_del:
 	otx2_mcam_flow_del(pf);
Signature · 1 line
2.34.1
author maintainer
#13 lore raw

PATCH v3 7/7 devlink: Add eswitch mode boot defaults

In reply to Mark Bloch

Add devlink_eswitch_mode= command line support for setting a default eswitch mode during device initialization.

The supported syntax selects either all devlink handles or one explicit comma-separated handle list:

devlink_eswitch_mode=*=<mode> devlink_eswitch_mode=<handle>[,<handle>...]=<mode>

where <mode> is one of legacy, switchdev or switchdev_inactive. All selected handles receive the same mode. Assigning different modes to different handle lists in the same parameter value is not supported.

The default is applied through the existing eswitch_mode_set() devlink operation, matching the userspace devlink eswitch mode command. devlink core applies it when a matching devlink instance is registered and after a successful devlink reload that performed DRIVER_REINIT, so rebuilt device state returns to the requested boot default.

Document the devlink_eswitch_mode= syntax and duplicate handle handling.

---
 .../admin-guide/kernel-parameters.txt         |  25 ++
 .../networking/devlink/devlink-defaults.rst   |  78 +++++
 Documentation/networking/devlink/index.rst    |   1 +
 net/devlink/core.c                            | 271 ++++++++++++++++++
 net/devlink/dev.c                             |   3 +
 net/devlink/devl_internal.h                   |   1 +
 6 files changed, 379 insertions(+)
 create mode 100644 Documentation/networking/devlink/devlink-defaults.rst
Documentation/admin-guide/kernel-parameters.txt +25 -0
index b3fdbbe3b3cc..b4fcc7f81166 100644
@@ -1246,6 +1246,31 @@ Kernel parameters
 	dell_smm_hwmon.fan_max=
 			[HW] Maximum configurable fan speed.
 
+	devlink_eswitch_mode=
+			[NET]
+			Format:
+			<selector>=<mode>
+
+			<selector>:
+			* | <handle>[,<handle>...]
+
+			<handle>:
+			<bus-name>/<dev-name>
+
+			Configure default devlink eswitch mode for matching
+			devlink instances during device initialization.
+
+			<mode>:
+			legacy | switchdev | switchdev_inactive
+
+			Examples:
+			devlink_eswitch_mode=*=switchdev
+			devlink_eswitch_mode=pci/0000:08:00.0=switchdev
+			devlink_eswitch_mode=pci/0000:08:00.0,pci/0000:09:00.1=switchdev_inactive
+
+			See Documentation/networking/devlink/devlink-defaults.rst
+			for the full syntax.
+
 	dfltcc=		[HW,S390]
 			Format: { on | off | def_only | inf_only | always }
 			on:       s390 zlib hardware support for compression on
Documentation/networking/devlink/devlink-defaults.rst new file +78 -0

78 lines not on this page · raw message

Documentation/networking/devlink/index.rst +1 -0
index f7ba7dcf477d..0d27a7008b14 100644
@@ -56,6 +56,7 @@ general.
    :maxdepth: 1
 
    devlink-dpipe
+   devlink-defaults
    devlink-eswitch-attr
    devlink-flash
    devlink-health
net/devlink/core.c +271 -0

308 lines not on this page · raw message

net/devlink/dev.c +3 -0
index 57b2b8f03543..0b4a831465e8 100644
@@ -478,6 +478,9 @@ int devlink_reload(struct devlink *devlink, struct net *dest_net,
 		return err;
 
 	WARN_ON(!(*actions_performed & BIT(action)));
+	if (*actions_performed & BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT))
+		devlink_apply_default_esw_mode(devlink);
+
 	/* Catch driver on updating the remote action within devlink reload */
 	WARN_ON(memcmp(remote_reload_stats, devlink->stats.remote_reload_stats,
 		       sizeof(remote_reload_stats)));
net/devlink/devl_internal.h +1 -0
index e4e48ee2da5a..12557b65248d 100644
@@ -71,6 +71,7 @@ extern struct genl_family devlink_nl_family;
 struct devlink *__devlink_alloc(const struct devlink_ops *ops, size_t priv_size,
 				struct net *net, struct device *dev,
 				const struct device_driver *dev_driver);
+void devlink_apply_default_esw_mode(struct devlink *devlink);
 
 #define devl_warn(devlink, format, args...)				\
 	do {								\
Signature · 1 line
2.34.1
#14 lore raw
In reply to Mark Bloch
Quoted 2 lines · Mark Bloch

This series adds a devlink_eswitch_mode= kernel command line parameter for applying a default devlink eswitch mode during device initialization.

Please trim your Cc list. There's nothing I can do about networking. You're CCing too many people unnecessarily.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette