Skip to contentSkip to conversation

net/mlx5: Use effective affinity mask for IRQ selection

PATCH Applied Fushuai Wang 4 participants 4 messages Jun 5 – Jun 9

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 net/mlx5: Use effective affinity mask for IRQ selection

From: Fushuai Wang <[email protected]>

When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement, cpumask_subset() check would fail because the iter_mask contains offline CPUs that are not present in req_mask, causing sf creation to fail.

This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. For sf_ctrl_pool with ≤64 sf:
- xa_num_irqs = {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq via irq_pool_request_irq()
6. xa_alloc() fails because the pool is full(There is only one slot)
7. sf creation fails with error

Use irq_get_effective_affinity_mask() instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.

Fixes 061f5b23588a ("net/mlx5: SF, Use all available cpu for setting cpu affinity")

---
v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix. v1->v2: Use mlx5_irq_get_affinity_mask() api

previous discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#u

 drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4 -1
index 994fe83da4be..a0bb8ee44e35 100644
@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req
 
 	lockdep_assert_held(&pool->lock);
 	xa_for_each_range(&pool->irqs, index, iter, start, end) {
-		struct cpumask *iter_mask = mlx5_irq_get_affinity_mask(iter);
 		int iter_refcount = mlx5_irq_read_locked(iter);
+		const struct cpumask *iter_mask;
 
+		iter_mask = irq_get_effective_affinity_mask(mlx5_irq_get_irq(iter));
+		if (!iter_mask)
+			continue;
 		if (!cpumask_subset(iter_mask, req_mask))
 			/* skip IRQs with a mask which is not subset of req_mask */
 			continue;
Signature · 1 line
2.36.1
Reviewed-by on v3 1/1
#1 lore raw
Quoted 28 lines · Fushuai Wang

External email: Use caution opening links or attachments

From: Fushuai Wang <[email protected]>

When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement, cpumask_subset() check would fail because the iter_mask contains offline CPUs that are not present in req_mask, causing sf creation to fail.

This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. For sf_ctrl_pool with ≤64 sf:
- xa_num_irqs = {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq via irq_pool_request_irq()
6. xa_alloc() fails because the pool is full(There is only one slot)
7. sf creation fails with error

Quoted 5 lines

Use irq_get_effective_affinity_mask() instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.

Fixes 061f5b23588a ("net/mlx5: SF, Use all available cpu for setting cpu affinity")
Quoted 33 lines

---
v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix. v1->v2: Use mlx5_irq_get_affinity_mask() api

previous discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#u

  drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4 -2
index 994fe83da4be..a0bb8ee44e35 100644
@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req
 
         lockdep_assert_held(&pool->lock);
         xa_for_each_range(&pool->irqs, index, iter, start, end) {
-               struct cpumask *iter_mask = mlx5_irq_get_affinity_mask(iter);
                 int iter_refcount = mlx5_irq_read_locked(iter);
+               const struct cpumask *iter_mask;
 
+               iter_mask = irq_get_effective_affinity_mask(mlx5_irq_get_irq(iter));
+               if (!iter_mask)
+                       continue;
                 if (!cpumask_subset(iter_mask, req_mask))
                         /* skip IRQs with a mask which is not subset of req_mask */
                         continue;
--

2.36.1

maintainer Reviewed-by on v3 1/1
#2 lore raw
In reply to Fushuai Wang
Quoted 50 lines · Fushuai Wang

From: Fushuai Wang <[email protected]>

When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement, cpumask_subset() check would fail because the iter_mask contains offline CPUs that are not present in req_mask, causing sf creation to fail.

This is an example:
1. When mlx5 driver loads, it initializes the IRQ pools. For sf_ctrl_pool with ≤64 sf:
- xa_num_irqs = {N, N} (There is only one slot)
2. When the first SF is created:
- The ctrl IRQ is allocated with mask=cpu_online_mask={0-191}
2. We take CPU 20 offline
3. Existing ctl irq still have mask={0-191}
4. Create a new SF:
- req_mask={0-19,21-191}
- iter_mask={0-191}
- {0-191} is NOT a subset of {0-19,21-191}
- least_loaded_irq=NULL
5. Try to allocate a new irq via irq_pool_request_irq()
6. xa_alloc() fails because the pool is full(There is only one slot)
7. sf creation fails with error

Use irq_get_effective_affinity_mask() instead, which returns the IRQ's actual effective affinity that already excludes offline CPUs.

Fixes 061f5b23588a ("net/mlx5: SF, Use all available cpu for setting cpu affinity")

---
v2->v3: Separate the patchset to two patches, reverse xmas tree coding style fix. v1->v2: Use mlx5_irq_get_affinity_mask() api

previous discussion:
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/T/#u

  drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +0 -0
index 994fe83da4be..a0bb8ee44e35 100644
@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req
 
  	lockdep_assert_held(&pool->lock);
  	xa_for_each_range(&pool->irqs, index, iter, start, end) {

drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req · line 116

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c +4 -1
-		struct cpumask *iter_mask = mlx5_irq_get_affinity_mask(iter);
  		int iter_refcount = mlx5_irq_read_locked(iter);
+		const struct cpumask *iter_mask;
 
+		iter_mask = irq_get_effective_affinity_mask(mlx5_irq_get_irq(iter));
+		if (!iter_mask)
+			continue;
  		if (!cpumask_subset(iter_mask, req_mask))
  			/* skip IRQs with a mask which is not subset of req_mask */
  			continue;

Thanks for your patch.

#3 lore raw
In reply to Fushuai Wang
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 9 lines · On Fri, 5 Jun 2026 18:21:12 +0800 you wrote:

From: Fushuai Wang <[email protected]>

When a sf is created after a CPU has been taken offline, the IRQ pool may contain IRQs with affinity masks that include the offline CPU. Since only online CPUs should be considered for IRQ placement, cpumask_subset() check would fail because the iter_mask contains offline CPUs that are not present in req_mask, causing sf creation to fail.

[...]

Here is the summary with links:
- [net,v3] net/mlx5: Use effective affinity mask for IRQ selection

    https://git.kernel.org/netdev/net/c/a7767290e77c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html