net/mlx5: Use effective affinity mask for IRQ selection
Subsystems (4)
PATCH v3 1/1 net/mlx5: Use effective affinity mask for IRQ selection
From: Fushuai Wang <[email protected]>
When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement, cpumask_subset() check would fail because the iter_mask contains offline CPUs that are not present in req_mask, causing sf creation to fail.
This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. For sf_ctrl_pool with ≤64 sf:
- xa_num_irqs = {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq via irq_pool_request_irq()
6. xa_alloc() fails because the pool is full(There is only one slot)
7. sf creation fails with error
Use irq_get_effective_affinity_mask() instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.
---v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix. v1->v2: Use mlx5_irq_get_affinity_mask() api
previous discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#u
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4 -1
@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *reqlockdep_assert_held (&pool->lock);xa_for_each_range (&pool->irqs, index, iter, start, end) { -struct cpumask *iter_mask =mlx5_irq_get_affinity_mask (iter);int iter_refcount =mlx5_irq_read_locked (iter); +const struct cpumask *iter_mask; + iter_mask =irq_get_effective_affinity_mask (mlx5_irq_get_irq (iter)); +if (!iter_mask) +continue ;if (!cpumask_subset (iter_mask, req_mask))/* skip IRQs with a mask which is not subset of req_mask */ continue ;
Signature · 1 line
2.36.1
Quoted 28 lines · Fushuai Wang
External email: Use caution opening links or attachments
From: Fushuai Wang <[email protected]>
When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement,
cpumask_subset()check would fail because theiter_maskcontains offline CPUs that are not present inreq_mask, causing sf creation to fail.This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. Forsf_ctrl_poolwith ≤64 sf:
-xa_num_irqs= {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq viairq_pool_request_irq()
6.xa_alloc()fails because the pool is full(There is only one slot)
7. sf creation fails with error
Quoted 5 lines
Use
irq_get_effective_affinity_mask()instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.Fixes 061f5b23588a ("net/mlx5: SF, Use all available cpu for setting cpu affinity")Suggested-by Shay Drory <[email protected]>
Quoted 33 lines
Signed-off-by Fushuai Wang <[email protected]>---
v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix.v1->v2: Usemlx5_irq_get_affinity_mask()apiprevious discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#udrivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4
-2@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *reqlockdep_assert_held (&pool->lock);xa_for_each_range (&pool->irqs, index, iter, start, end) { -struct cpumask *iter_mask =mlx5_irq_get_affinity_mask (iter);int iter_refcount =mlx5_irq_read_locked (iter); +const struct cpumask *iter_mask; + iter_mask =irq_get_effective_affinity_mask (mlx5_irq_get_irq (iter)); +if (!iter_mask) +continue ;if (!cpumask_subset (iter_mask, req_mask))/* skip IRQs with a mask which is not subset of req_mask */ continue ; --2.36.1
Quoted 50 lines · Fushuai Wang
From: Fushuai Wang <[email protected]>
When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement,
cpumask_subset()check would fail because theiter_maskcontains offline CPUs that are not present inreq_mask, causing sf creation to fail.This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. Forsf_ctrl_poolwith ≤64 sf:
-xa_num_irqs= {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq viairq_pool_request_irq()
6.xa_alloc()fails because the pool is full(There is only one slot)
7. sf creation fails with errorUse
irq_get_effective_affinity_mask()instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.Fixes 061f5b23588a ("net/mlx5: SF, Use all available cpu for setting cpu affinity")Suggested-by Shay Drory <[email protected]>Signed-off-by Fushuai Wang <[email protected]>---
v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix.v1->v2: Usemlx5_irq_get_affinity_mask()apiprevious discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#udrivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +0
-0@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *reqlockdep_assert_held (&pool->lock);xa_for_each_range (&pool->irqs, index, iter, start, end) {
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req · line 116
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4
-1-struct cpumask *iter_mask =mlx5_irq_get_affinity_mask (iter);int iter_refcount =mlx5_irq_read_locked (iter); +const struct cpumask *iter_mask; + iter_mask =irq_get_effective_affinity_mask (mlx5_irq_get_irq (iter)); +if (!iter_mask) +continue ;if (!cpumask_subset (iter_mask, req_mask))/* skip IRQs with a mask which is not subset of req_mask */ continue ;
Thanks for your patch.
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 9 lines · On Fri, 5 Jun 2026 18:21:12 +0800 you wrote:
From: Fushuai Wang <[email protected]>
When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement,
cpumask_subset()check would fail because theiter_maskcontains offline CPUs that are not present inreq_mask, causing sf creation to fail.[...]
Here is the summary with links:
- [net,v3] net/mlx5: Use effective affinity mask for IRQ selection
https://git.kernel.org/netdev/net/c/a7767290e77c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html