Skip to contentSkip to conversation

eth: bnxt: preserve IRQ affinity across IRQ reallocation

PATCH Applied Jakub Kicinski 3 participants 7 messages Aug 13 – Aug 17

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

cover eth: bnxt: preserve IRQ affinity across IRQ reallocation

bnxt currently discards the IRQ affinity when changing ring count:

  # ethtool -l ens9np0
  [...] Combined:	8 [...]
  # ynl --family netdev --dump napi-get --json '{"ifindex": 2}'
  [...]
   {'defer-hard-irqs': 0,
  'gro-flush-timeout': 0,
  'id': 70,
  'ifindex': 2,
  'irq': 170,                << IRQ 170 is for NAPI 1 (second to last)
  'irq-suspend-timeout': 0,
  'threaded': 'disabled'},
 {'defer-hard-irqs': 0,
  'gro-flush-timeout': 0,
  'id': 69,
  'ifindex': 2,
  'irq': 169,
  'irq-suspend-timeout': 0,
  'threaded': 'disabled'}]
  # cat /proc/irq/170/smp_affinity_list 
  1     <<< system config script set CPU 1 for this IRQ
  # ethtool -L ens9np0 combined 1
  # ethtool -L ens9np0 combined 8
  # cat /proc/irq/170/smp_affinity_list 
  0-31  <<< system has 32 CPUs

After this series:

  # cat /proc/irq/170/smp_affinity_list 
  1
  # ethtool -L ens9np0 combined 1
  # ethtool -L ens9np0 combined 8
  # cat /proc/irq/170/smp_affinity_list 
  1

We recently added the ability to networking core to track the affinity. bnxt doesn't use it because it needs TPH programming as well. Let's align its local behavior.

The loss of IRQ config is a real production problem, but it also breaks some of the NIPA tests.

Jakub Kicinski (2):
eth: bnxt: decrease indent in bnxt_init_int_mode()
eth: bnxt: preserve IRQ affinity across IRQ reallocation

 drivers/net/ethernet/broadcom/bnxt/bnxt.h |  11 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 131 +++++++++++++++-------
 2 files changed, 98 insertions(+), 44 deletions(-)
Signature · 1 line
2.55.0
author maintainer
#1 lore raw

PATCH 1/2 eth: bnxt: decrease indent in bnxt_init_int_mode()

Handle the IRQ table allocation failure right away instead of wrapping the rest of the function in an if. Purely to make upcoming changes more readable.

While refactoring, drop the init of rc which is not necessary.

No functional changes.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 36 +++++++++++------------
 1 file changed, 18 insertions(+), 18 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +18 -18
index 1e4944f3e606..d3531cd283a5 100644
@@ -11592,7 +11592,7 @@ static int bnxt_get_num_msix(struct bnxt *bp)
 
 static int bnxt_init_int_mode(struct bnxt *bp)
 {
-	int i, total_vecs, max, rc = 0, min = 1, ulp_msix, tx_cp, tbl_size;
+	int i, total_vecs, max, rc, min = 1, ulp_msix, tx_cp, tbl_size;
 
 	total_vecs = bnxt_get_num_msix(bp);
 	max = bnxt_get_max_func_irqs(bp);
@@ -11617,26 +11617,26 @@ static int bnxt_init_int_mode(struct bnxt *bp)
 	if (pci_msix_can_alloc_dyn(bp->pdev))
 		tbl_size = max;
 	bp->irq_tbl = kzalloc_objs(*bp->irq_tbl, tbl_size);
-	if (bp->irq_tbl) {
-		for (i = 0; i < total_vecs; i++)
-			bp->irq_tbl[i].vector = pci_irq_vector(bp->pdev, i);
-
-		bp->total_irqs = total_vecs;
-		/* Trim rings based upon num of vectors allocated */
-		rc = bnxt_trim_rings(bp, &bp->rx_nr_rings, &bp->tx_nr_rings,
-				     total_vecs - ulp_msix, min == 1);
-		if (rc)
-			goto msix_setup_exit;
-
-		tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings);
-		bp->cp_nr_rings = (min == 1) ?
-				  max_t(int, tx_cp, bp->rx_nr_rings) :
-				  tx_cp + bp->rx_nr_rings;
-
-	} else {
+	if (!bp->irq_tbl) {
 		rc = -ENOMEM;
 		goto msix_setup_exit;
 	}
+
+	for (i = 0; i < total_vecs; i++)
+		bp->irq_tbl[i].vector = pci_irq_vector(bp->pdev, i);
+
+	bp->total_irqs = total_vecs;
+	/* Trim rings based upon num of vectors allocated */
+	rc = bnxt_trim_rings(bp, &bp->rx_nr_rings, &bp->tx_nr_rings,
+			     total_vecs - ulp_msix, min == 1);
+	if (rc)
+		goto msix_setup_exit;
+
+	tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings);
+	bp->cp_nr_rings = (min == 1) ?
+			  max_t(int, tx_cp, bp->rx_nr_rings) :
+			  tx_cp + bp->rx_nr_rings;
+
 	return 0;
 
 msix_setup_exit:
Signature · 1 line
2.55.0
author maintainer
#3 lore raw

PATCH 2/2 eth: bnxt: preserve IRQ affinity across IRQ reallocation

In reply to Jakub Kicinski

Reconfiguring the rings frees the MSI-X vectors and allocates them again. The IRQ descriptors go away with them, so the affinity user space set is silently replaced by the driver's default NUMA spread. This is painful to deal with for user space as seemingly arbitrary NIC configuration changes lead to loss of configuration.

In NIPA (netdev CI) this results in the toeplitz test reporting:

Exception| net.lib.py.ksft.KsftFailEx: IRQ170 is not mapped to a single core: 0-31

if the test run after another test which reconfigured the device. We configure the IRQ mapping at boot, but if the driver is not preserving the config - it gets lost.

Record the affinity in the notifier and apply it when the IRQs are requested again. The notifier has to be registered unconditionally now, so far it was only installed when TPH was enabled. Drivers which let the core manage the affinity (idpf, ice, iavf via netif_set_affinity_auto()) work exactly like this, napi_restore_config() reapplies napi_config.affinity_mask on every napi_enable().

Note that the affinity is supposed to follow the NAPI / queue, same as the napi_config behavior in drivers mentioned above. If the user changes the affinity when the device is down - we will override it on up. That's expected, the IRQs are not associated with queues when device is down (no name, no entry in /proc/interrupts, no entry in netdev netlink).

map_idx is ulp_msix + i, so the slot shifts whenever RoCE takes or releases vectors and the mask would end up on a different ring. Key using the completion ring id, which maps to the NAPI instance.

Note2: this restores the side effect fcf42409c6e1 ("bnxt_en: use irq_update_affinity_hint()") removed, but not the problem it was fixing. The complaint there was that reopening the device resets the affinity and can move an IRQ onto a CPU irqbalance was told to stay away from. We now replay what user space or irqbalance last asked for, the driver's own placement is only used for a ring nobody has configured.

Note3: the combined irq_set_affinity_and_hint() looks like it may hide the failure from __irq_set_affinity(), but let's assume the IRQ maintainers know what their doing - either this can't happen or is intentional.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h | 11 ++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 95 +++++++++++++++++------
 2 files changed, 80 insertions(+), 26 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.h +9 -2
index dc8ec5e5733e..ab894f8addef 100644
@@ -1261,9 +1261,7 @@ struct bnxt_irq {
 	irq_handler_t	handler;
 	unsigned int	vector;
 	u8		requested:1;
-	u8		have_cpumask:1;
 	char		name[IFNAMSIZ + BNXT_IRQ_NAME_EXTRA];
-	cpumask_var_t	cpu_mask;
 
 	struct bnxt	*bp;
 	int		msix_nr;
@@ -2482,6 +2480,15 @@ struct bnxt {
 	 pci_channel_offline((bp)->pdev))
 
 	struct bnxt_irq	*irq_tbl;
+	/* IRQ affinity, indexed by completion ring. Kept across IRQ
+	 * reallocation, the MSI-X vector index is not stable.
+	 */
+	cpumask_var_t		*ring_cpu_mask;
+	/* Rings for which the mask above was configured from the outside,
+	 * rather than being our own default placement.
+	 */
+	unsigned long		*ring_affinity_set;
+	int			max_irqs;
 	int			total_irqs;
 	int			ulp_num_msix_want;
 	u8			mac_addr[ETH_ALEN];
drivers/net/ethernet/broadcom/bnxt/bnxt.c +71 -24

170 lines not on this page · raw message

Signature · 1 line
2.55.0
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH net-next 1/2] eth: bnxt: decrease indent in bnxt_init_int_mode()

In reply to Jakub Kicinski
Quoted 5 lines · Jakub Kicinski

Handle the IRQ table allocation failure right away instead of wrapping the rest of the function in an if. Purely to make upcoming changes more readable.

Quoted 9 lines

While refactoring, drop the init of rc which is not necessary.

No functional changes.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 36 +++++++++++------------
 1 file changed, 18 insertions(+), 18 deletions(-)
maintainer Reviewed-by on v1 2/2
#4 lore raw
In reply to Jakub Kicinski
Quoted 7 lines

Note2: this restores the side effect fcf42409c6e1 ("bnxt_en: use irq_update_affinity_hint()") removed, but not the problem it was fixing. The complaint there was that reopening the device resets the affinity and can move an IRQ onto a CPU irqbalance was told to stay away from. We now replay what user space or irqbalance last asked for, the driver's own placement is only used for a ring nobody has configured.

Spl thanks for this Note2. This saved me time!

Quoted 6 lines

Note3: the combined irq_set_affinity_and_hint() looks like it may hide the failure from __irq_set_affinity(), but let's assume the IRQ maintainers know what their doing - either this can't happen or is intentional.

Quoted 6 lines
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h | 11 ++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 95 +++++++++++++++++------
 2 files changed, 80 insertions(+), 26 deletions(-)

LGTM.

author maintainer
#5 lore raw
In reply to Jakub Kicinski

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_request_irq(struct bnxt *bp) · line 12004

Quoted 2 lines · On Thu, 13 Aug 2026 12:32:48 -0700 Jakub Kicinski wrote:
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -0
+		 * Set after installing the notifier, if we race with the user
+		 * it's better to overwrite than miss the notification.

'shiko warns about the race between the notifier and the init. This is effectively the same pre-existing race we I'm flagging in this comment. We ignore updates while the device is down completely, I don't think it's worth trying to reliably delay the racing user programming after init has finished.

#6 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 21 lines · On Thu, 13 Aug 2026 12:32:46 -0700 you wrote:

bnxt currently discards the IRQ affinity when changing ring count:

  # ethtool -l ens9np0
  [...] Combined:	8 [...]
  # ynl --family netdev --dump napi-get --json '{"ifindex": 2}'
  [...]
   {'defer-hard-irqs': 0,
  'gro-flush-timeout': 0,
  'id': 70,
  'ifindex': 2,
  'irq': 170,                << IRQ 170 is for NAPI 1 (second to last)
  'irq-suspend-timeout': 0,
  'threaded': 'disabled'},
 {'defer-hard-irqs': 0,
  'gro-flush-timeout': 0,
  'id': 69,
  'ifindex': 2,
  'irq': 169,
  'irq-suspend-timeout': 0,
  'threaded': 'disabled'}]
Quoted 1 line

[...]

Here is the summary with links:
- [net-next,1/2] eth: bnxt: decrease indent in bnxt_init_int_mode()

    https://git.kernel.org/netdev/net-next/c/2b49709667de
  - [net-next,2/2] eth: bnxt: preserve IRQ affinity across IRQ reallocation
    https://git.kernel.org/netdev/net-next/c/fb0502649043

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html