net/mlx5: Fix E-Switch work queue deadlock with devlink lock
Subsystems (4)
Tariq Toukan
maintainerv2 cover net/mlx5: Fix E-Switch work queue deadlock with devlink lock
Hi,
See detailed description by Mark below [1].
Regards,
Tariq
[1]mlx5_eswitch_cleanup() calls destroy_workqueue() while holding the devlink lock through mlx5_uninit_one(). E-Switch workqueue workers also need the devlink lock, but previously took it before checking whether their work item was stale. Cleanup can therefore wait for a worker that is blocked on the same devlink lock.
Mode changes have the same ordering hazard: the mode-change path holds devlink lock while tearing down the current mode, and old work may still be pending on the E-Switch workqueue.
Fix this by making esw_wq_handler() check the generation counter before attempting to take devlink lock. The worker uses devl_trylock(); if the lock is busy and the work is still current, it sleeps on an E-Switch wait queue with a short timeout. Invalidation increments the generation counter and wakes the wait queue, so stale workers exit without spinning or blocking cleanup.
The generation counter already existed but was buried in mlx5_esw_functions and only covered function-change events. The three patches get from there to the fix in small steps.
Patch 1 moves the counter up to mlx5_eswitch. Pure refactor, no behavior change.
Patch 2 cleans up the work queue plumbing: factors out the repeated lock/check/dispatch boilerplate into a single esw_wq_handler() and adds mlx5_esw_add_work() as the one place to enqueue work.
Patch 3 is the actual fix: check the generation before the lock, use devl_trylock() instead of devl_lock(), add a wait queue so lock retries do not spin, and invalidate pending work at the earliest safe operation boundary. Cleanup invalidates before destroy_workqueue(), and mode teardown unregisters the work-producing notifiers before invalidating so new notifier work cannot capture the new generation.
V2:
Split out from a larger series. The representor lifecycle improvements to be sent separately.
Patch 3:
- Move generation invalidation after notifier unregister but before teardown, so old work is discarded early without allowing new notifier work to use the new generation.
- Replace cond_resched() polling with a wait queue to avoid CPU spinning while devlink lock is held by a long operation.
Link to V1:
https://lore.kernel.org/all/[email protected]/
Mark Bloch (3):
net/mlx5: E-Switch, move work queue generation counter net/mlx5: E-Switch, introduce generic work queue dispatch helper net/mlx5: E-Switch, fix deadlock between devlink lock and esw->wq
.../net/ethernet/mellanox/mlx5/core/eswitch.c | 20 +++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 5 +- .../mellanox/mlx5/core/eswitch_offloads.c | 96 ++++++++++++------- 3 files changed, 82 insertions(+), 39 deletions(-)
base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v2 1/3 net/mlx5: E-Switch, move work queue generation counter
From: Mark Bloch <[email protected]>
The generation counter in mlx5_esw_functions is used to detect stale work items on the E-Switch work queue. Move it from mlx5_esw_functions to the top-level mlx5_eswitch struct so it can guard all work types, not just function-change events.
This is a mechanical refactor: no behavioral change.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/eswitch.h | 2 +- drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 4 ++-- 3 files changed, 5 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +2 -1
@@ -1075,7 +1075,7 @@ static void mlx5_eswitch_event_handler_unregister(struct mlx5_eswitch *esw)if (esw->mode ==MLX5_ESWITCH_OFFLOADS &&mlx5_eswitch_is_funcs_handler (esw->dev)) {mlx5_eq_notifier_unregister (esw->dev, &esw->esw_funcs.nb); -atomic_inc (&esw->esw_funcs.generation); +atomic_inc (&esw->generation); } }
@@ -2072,6 +2072,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev)mutex_init (&esw->state_lock);init_rwsem (&esw->mode_lock);refcount_set (&esw->qos.refcnt,0 ); +atomic_set (&esw->generation,0 ); esw->enabled_vports =0 ; esw->offloads.inline_mode =MLX5_INLINE_MODE_NONE ;
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1 -1
@@ -340,7 +340,6 @@ struct mlx5_host_work {struct mlx5_esw_functions {struct mlx5_nb nb; -atomic_t generation;bool host_funcs_disabled;u16 num_vfs;u16 num_ec_vfs;
@@ -410,6 +409,7 @@ struct mlx5_eswitch {struct mlx5_devcom_comp_dev *devcom;u16 enabled_ipsec_vf_count;bool eswitch_operation_in_progress; +atomic_t generation; };void esw_offloads_disable (struct mlx5_eswitch *esw);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +2 -2
@@ -3667,7 +3667,7 @@ esw_vfs_changed_event_handler(struct mlx5_eswitch *esw, int work_gen,devl_lock (devlink);/* Stale work from one or more mode changes ago. Bail out. */ -if (work_gen !=atomic_read (&esw->esw_funcs.generation)) +if (work_gen !=atomic_read (&esw->generation)) goto unlock; new_num_vfs = MLX5_GET(query_esw_functions_out, out,
@@ -3729,7 +3729,7 @@ int mlx5_esw_funcs_changed_handler(struct notifier_block *nb, unsigned long type esw = container_of(esw_funcs, struct mlx5_eswitch, esw_funcs); host_work->esw = esw; - host_work->work_gen =atomic_read (&esw_funcs->generation); + host_work->work_gen =atomic_read (&esw->generation);INIT_WORK (&host_work->work, esw_functions_changed_event_handler);queue_work (esw->work_queue, &host_work->work);
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v2 2/3 net/mlx5: E-Switch, introduce generic work queue dispatch helper
In reply to Tariq ToukanFrom: Mark Bloch <[email protected]>
Each E-Switch work item requires the same boilerplate: acquire the devlink lock, check whether the work is stale, dispatch to the appropriate handler, and release the lock. Factor this out.
Add a func callback to mlx5_host_work so the generic handler esw_wq_handler() can dispatch to the right function without duplicating locking logic. Introduce mlx5_esw_add_work() as the single enqueue point: it stamps the work item with the current generation counter and queues it onto the E-Switch work queue.
Refactor esw_vfs_changed_event_handler() to match the new contract:
it no longer receives work_gen or out as parameters. It queries mlx5_esw_query_functions() itself and owns the kvfree() of the result. The devlink lock is acquired and released by esw_wq_handler() before dispatching, so the handler runs with the lock already held.
Update mlx5_esw_funcs_changed_handler() to use mlx5_esw_add_work().
--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 1 + .../mellanox/mlx5/core/eswitch_offloads.c | 77 +++++++++++-------- 2 files changed, 45 insertions(+), 33 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1 -0
@@ -336,6 +336,7 @@ struct mlx5_host_work {struct work_struct work;struct mlx5_eswitch *esw;int work_gen; +void (*func)(struct mlx5_eswitch *esw); };struct mlx5_esw_functions {
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +44 -33
125 lines not on this page · raw message
Signature · 1 line
2.44.0
[email protected]
AI reviewRe: [PATCH net-next V2 2/3] net/mlx5: E-Switch, introduce generic work queue dispatch helper
Sashiko has reviewed this patch and found no issues.
Quoted 18 lines
From: Mark Bloch <[email protected]>
Each E-Switch work item requires the same boilerplate: acquire the devlink lock, check whether the work is stale, dispatch to the appropriate handler, and release the lock. Factor this out.
Add a func callback to
mlx5_host_workso the generic handleresw_wq_handler()can dispatch to the right function without duplicating locking logic. Introducemlx5_esw_add_work()as the single enqueue point: it stamps the work item with the current generation counter and queues it onto the E-Switch work queue.Refactor
esw_vfs_changed_event_handler()to match the new contract:
it no longer receiveswork_genor out as parameters. It queriesmlx5_esw_query_functions()itself and owns thekvfree()of the result. The devlink lock is acquired and released byesw_wq_handler()before dispatching, so the handler runs with the lock already held.
Quoted 5 lines
Update
mlx5_esw_funcs_changed_handler()to usemlx5_esw_add_work().Signed-off-by Mark Bloch <[email protected]>Reviewed-by Cosmin Ratiu <[email protected]>Signed-off-by Tariq Toukan <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Tariq Toukan
maintainerPATCH v2 3/3 net/mlx5: E-Switch, fix deadlock between devlink lock and esw->wq
In reply to Tariq ToukanFrom: Mark Bloch <[email protected]>
mlx5_eswitch_cleanup() calls destroy_workqueue() while holding the devlink lock through mlx5_uninit_one(). E-Switch workqueue workers also need the devlink lock, but previously took it before checking whether their work item was stale. This can deadlock when cleanup waits for a worker that is blocked on the same devlink lock.
Mode changes have the same ordering hazard: the mode-change path holds devlink lock while tearing down the current mode, and old work may still be pending on the E-Switch workqueue.
Fix this by making esw_wq_handler() check the generation counter before attempting to take devlink lock. The worker uses devl_trylock(); if the lock is busy and the work is still current, it sleeps on an E-Switch wait queue with a short timeout. Invalidation increments the generation counter and wakes the wait queue, so stale workers exit without spinning or blocking cleanup.
Invalidate work at the earliest safe operation boundary. Cleanup invalidates before destroy_workqueue(), and QoS cleanup runs after the workqueue is destroyed. Mode teardown unregisters the work-producing notifiers first, then invalidates the queue before tearing down FDB/QoS/rate-node state. This prevents new notifier work from capturing the new generation while still making old work stale before expensive teardown starts.
mlx5_devlink_eswitch_mode_set() now relies on mlx5_eswitch_disable_locked() for the mode-change invalidation instead of incrementing the generation after disable. mlx5_eswitch_disable() gets the same coverage. SR-IOV enable/disable paths invalidate before VF state changes so work against the old VF count or mode is discarded.
Remove the conditional generation increment in mlx5_eswitch_event_handler_unregister(); mlx5_eswitch_disable_locked() now handles it unconditionally after the relevant notifiers are unregistered.
--- .../net/ethernet/mellanox/mlx5/core/eswitch.c | 19 +++++++++++++---- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 2 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 21 +++++++++++++++++-- 3 files changed, 36 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +15 -4
@@ -1070,13 +1070,17 @@ static void mlx5_eswitch_event_handler_register(struct mlx5_eswitch *esw) } } +static void mlx5_eswitch_invalidate_wq (struct mlx5_eswitch *esw) +{ +atomic_inc (&esw->generation); +wake_up_all (&esw->work_queue_wait); +} +static void mlx5_eswitch_event_handler_unregister (struct mlx5_eswitch *esw) {if (esw->mode ==MLX5_ESWITCH_OFFLOADS && -mlx5_eswitch_is_funcs_handler (esw->dev)) { +mlx5_eswitch_is_funcs_handler (esw->dev))mlx5_eq_notifier_unregister (esw->dev, &esw->esw_funcs.nb); -atomic_inc (&esw->generation); - } }static void mlx5_eswitch_clear_vf_vports_info (struct mlx5_eswitch *esw)
@@ -1701,6 +1705,8 @@ int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs)if (toggle_lag)mlx5_lag_disable_change (esw->dev); +mlx5_eswitch_invalidate_wq (esw); +if (!mlx5_esw_is_fdb_created (esw)) { ret =mlx5_eswitch_enable_locked (esw, num_vfs); }else {
@@ -1746,6 +1752,8 @@ void mlx5_eswitch_disable_sriov(struct mlx5_eswitch *esw, bool clear_vf) esw->mode ==MLX5_ESWITCH_LEGACY ?"LEGACY" :"OFFLOADS" , esw->esw_funcs.num_vfs, esw->esw_funcs.num_ec_vfs, esw->enabled_vports); +mlx5_eswitch_invalidate_wq (esw); +if (!mlx5_core_is_ecpf (esw->dev)) {mlx5_eswitch_unload_vf_vports (esw, esw->esw_funcs.num_vfs);if (clear_vf)
@@ -1785,6 +1793,7 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw)mlx5_eq_notifier_unregister (esw->dev, &esw->nb);mlx5_eswitch_event_handler_unregister (esw); +mlx5_eswitch_invalidate_wq (esw); esw_info(esw->dev, "Disable: mode(%s), nvfs(%d), necvfs(%d), active vports(%d)\n", esw->mode == MLX5_ESWITCH_LEGACY ? "LEGACY" : "OFFLOADS",
@@ -2072,6 +2081,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev)mutex_init (&esw->state_lock);init_rwsem (&esw->mode_lock);refcount_set (&esw->qos.refcnt,0 ); +init_waitqueue_head (&esw->work_queue_wait);atomic_set (&esw->generation,0 ); esw->enabled_vports =0 ;
@@ -2110,8 +2120,9 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw)esw_info (esw->dev,"cleanup\n" ); -mlx5_esw_qos_cleanup (esw); +mlx5_eswitch_invalidate_wq (esw);destroy_workqueue (esw->work_queue); +mlx5_esw_qos_cleanup (esw);WARN_ON (refcount_read (&esw->qos.refcnt));mutex_destroy (&esw->state_lock);WARN_ON (!xa_empty (&esw->offloads.vhca_map));
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +2 -0
@@ -36,6 +36,7 @@#include <linux/if_ether.h> #include <linux/if_link.h> #include <linux/atomic.h> +#include <linux/wait.h> #include <linux/xarray.h> #include <net/devlink.h> #include <linux/mlx5/device.h>
@@ -385,6 +386,7 @@ struct mlx5_eswitch {*/ struct rw_semaphore mode_lock;atomic64_t user_count; +wait_queue_head_t work_queue_wait;/* Protected with the E-Switch qos domain lock. */ struct {
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +19 -2
@@ -3694,21 +3694,38 @@ static void esw_wq_handler(struct work_struct *work)struct mlx5_host_work *host_work;struct mlx5_eswitch *esw;struct devlink *devlink; +int work_gen; host_work = container_of(work, struct mlx5_host_work, work); esw = host_work->esw; + work_gen = host_work->work_gen; devlink =priv_to_devlink (esw->dev); -devl_lock (devlink); +/* Do not block on devlink lock until stale work is filtered out. +* Teardown can invalidate the generation and then wait for this +* workqueue while holding devlink lock. +*/ +for (;;) { +if (work_gen !=atomic_read (&esw->generation)) + goto free; + +if (devl_trylock (devlink)) +break ; + +wait_event_timeout (esw->work_queue_wait, + work_gen !=atomic_read (&esw->generation), +msecs_to_jiffies (60 )); + }/* Stale work from one or more mode changes ago. Bail out. */ -if (host_work->work_gen !=atomic_read (&esw->generation)) +if (work_gen !=atomic_read (&esw->generation)) goto unlock; host_work->func (esw); unlock:devl_unlock (devlink); +free:kfree (host_work); }
Signature · 1 line
2.44.0
[email protected]
AI reviewRe: [PATCH net-next V2 3/3] net/mlx5: E-Switch, fix deadlock between devlink lock and esw->wq
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Open-coded trylock retry loop used to bypass a lock ordering deadlock, violating networking…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Open-coded trylock retry loop used to bypass a lock ordering deadlock, violating networking subsystem guidelines.
- [High] Unsynchronized atomic notifier unregistration creates a race condition that defeats the stale work filter, risking execution of old work on torn-down state. --
commit 1ce8a06ae391b110c8a8badeed88f41bec66031f
Author: Mark Bloch <[email protected]>
net/mlx5: E-Switch, fix deadlock between devlink lock and esw->wq
This commit aims to fix a deadlock during E-Switch cleanup by checking a generation counter before taking the devlink lock in the workqueue handler, and invalidating the workqueue around mode changes and teardown.
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +0
-0
[ ... ]
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw) · line 1796
Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +1
-0@@ -1785,6 +1793,7 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw)mlx5_eq_notifier_unregister (esw->dev, &esw->nb);mlx5_eswitch_event_handler_unregister (esw); +mlx5_eswitch_invalidate_wq (esw);
Since mlx5_eq_notifier_unregister() uses atomic_notifier_chain_unregister(), which does not wait for in-flight callbacks to complete on other CPUs, is there a race condition here?
If a concurrent hardware event callback is already executing when mlx5_eswitch_invalidate_wq() increments the generation counter, could the callback read the new generation value and assign it to the new work item?
If that happens, the stale work might bypass the filter in esw_wq_handler() and execute against torn-down state.
[ ... ]
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0
-0
[ ... ]
Quoted 17 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-1@@ -3694,21 +3694,38 @@ static void esw_wq_handler(struct work_struct *work)struct mlx5_host_work *host_work;struct mlx5_eswitch *esw;struct devlink *devlink; +int work_gen; host_work = container_of(work, struct mlx5_host_work, work); esw = host_work->esw; + work_gen = host_work->work_gen; devlink =priv_to_devlink (esw->dev); -devl_lock (devlink); +/* Do not block on devlink lock until stale work is filtered out. +* Teardown can invalidate the generation and then wait for this +* workqueue while holding devlink lock. +*/ +for (;;) {
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c static void esw_wq_handler(struct work_struct *work) · line 3718
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+if (work_gen !=atomic_read (&esw->generation)) + goto free; + +if (devl_trylock (devlink)) +break ; + +wait_event_timeout (esw->work_queue_wait, + work_gen !=atomic_read (&esw->generation), +msecs_to_jiffies (60 )); + }
Does this open-coded trylock retry loop mask a lock ordering issue?
The commit message mentions this is used to avoid an AB-BA deadlock when mlx5_eswitch_cleanup() calls destroy_workqueue() while holding the devlink lock. Using a trylock loop with a sleep fallback to bypass lock ordering issues reimplements blocking without lockdep's deadlock detection, which might indicate the teardown sequence itself needs adjustment.
Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0
-0/* Stale work from one or more mode changes ago. Bail out. */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Tue, 28 Apr 2026 08:10:14 +0300 you wrote:
Hi,
See detailed description by Mark below [1].
Regards,
Tariq[...]
Here is the summary with links:
- [net-next,V2,1/3] net/mlx5: E-Switch, move work queue generation counter
https://git.kernel.org/netdev/net-next/c/f950ddb57ce4 - [net-next,V2,2/3] net/mlx5: E-Switch, introduce generic work queue dispatch helper https://git.kernel.org/netdev/net-next/c/2a110ee54e89 - [net-next,V2,3/3] net/mlx5: E-Switch, fix deadlock between devlink lock and esw->wq https://git.kernel.org/netdev/net-next/c/6a92fe1956d2
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html