eth: bnxt: preserve IRQ affinity across IRQ reallocation
Jakub Kicinski
maintainercover eth: bnxt: preserve IRQ affinity across IRQ reallocation
bnxt currently discards the IRQ affinity when changing ring count:
# ethtool -l ens9np0
[...] Combined: 8 [...]
# ynl --family netdev --dump napi-get --json '{"ifindex": 2}'
[...]
{'defer-hard-irqs': 0,
'gro-flush-timeout': 0,
'id': 70,
'ifindex': 2,
'irq': 170, << IRQ 170 is for NAPI 1 (second to last)
'irq-suspend-timeout': 0,
'threaded': 'disabled'},
{'defer-hard-irqs': 0,
'gro-flush-timeout': 0,
'id': 69,
'ifindex': 2,
'irq': 169,
'irq-suspend-timeout': 0,
'threaded': 'disabled'}]# cat /proc/irq/170/smp_affinity_list 1 <<< system config script set CPU 1 for this IRQ # ethtool -L ens9np0 combined 1 # ethtool -L ens9np0 combined 8 # cat /proc/irq/170/smp_affinity_list 0-31 <<< system has 32 CPUs
After this series:
# cat /proc/irq/170/smp_affinity_list 1 # ethtool -L ens9np0 combined 1 # ethtool -L ens9np0 combined 8 # cat /proc/irq/170/smp_affinity_list 1
We recently added the ability to networking core to track the affinity. bnxt doesn't use it because it needs TPH programming as well. Let's align its local behavior.
The loss of IRQ config is a real production problem, but it also breaks some of the NIPA tests.
Jakub Kicinski (2):
eth: bnxt: decrease indent in bnxt_init_int_mode()
eth: bnxt: preserve IRQ affinity across IRQ reallocation
drivers/net/ethernet/broadcom/bnxt/bnxt.h | 11 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 131 +++++++++++++++------- 2 files changed, 98 insertions(+), 44 deletions(-)
Signature · 1 line
2.55.0
Jakub Kicinski
maintainerPATCH 1/2 eth: bnxt: decrease indent in bnxt_init_int_mode()
Handle the IRQ table allocation failure right away instead of wrapping the rest of the function in an if. Purely to make upcoming changes more readable.
While refactoring, drop the init of rc which is not necessary.
No functional changes.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 36 +++++++++++------------ 1 file changed, 18 insertions(+), 18 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +18 -18
@@ -11592,7 +11592,7 @@ static int bnxt_get_num_msix(struct bnxt *bp)static int bnxt_init_int_mode (struct bnxt *bp) { -int i, total_vecs, max, rc =0 , min =1 , ulp_msix, tx_cp, tbl_size; +int i, total_vecs, max, rc, min =1 , ulp_msix, tx_cp, tbl_size; total_vecs =bnxt_get_num_msix (bp); max =bnxt_get_max_func_irqs (bp);
@@ -11617,26 +11617,26 @@ static int bnxt_init_int_mode(struct bnxt *bp)if (pci_msix_can_alloc_dyn (bp->pdev)) tbl_size = max; bp->irq_tbl =kzalloc_objs (*bp->irq_tbl, tbl_size); -if (bp->irq_tbl) { -for (i =0 ; i < total_vecs; i++) - bp->irq_tbl[i].vector =pci_irq_vector (bp->pdev, i); - - bp->total_irqs = total_vecs; -/* Trim rings based upon num of vectors allocated */ - rc =bnxt_trim_rings (bp, &bp->rx_nr_rings, &bp->tx_nr_rings, - total_vecs - ulp_msix, min ==1 ); -if (rc) - goto msix_setup_exit; - - tx_cp =bnxt_num_tx_to_cp (bp, bp->tx_nr_rings); - bp->cp_nr_rings = (min ==1 ) ? -max_t (int, tx_cp, bp->rx_nr_rings) : - tx_cp + bp->rx_nr_rings; - - }else { +if (!bp->irq_tbl) { rc = -ENOMEM ; goto msix_setup_exit; } + +for (i =0 ; i < total_vecs; i++) + bp->irq_tbl[i].vector =pci_irq_vector (bp->pdev, i); + + bp->total_irqs = total_vecs; +/* Trim rings based upon num of vectors allocated */ + rc =bnxt_trim_rings (bp, &bp->rx_nr_rings, &bp->tx_nr_rings, + total_vecs - ulp_msix, min ==1 ); +if (rc) + goto msix_setup_exit; + + tx_cp =bnxt_num_tx_to_cp (bp, bp->tx_nr_rings); + bp->cp_nr_rings = (min ==1 ) ? +max_t (int, tx_cp, bp->rx_nr_rings) : + tx_cp + bp->rx_nr_rings; +return 0 ; msix_setup_exit:
Signature · 1 line
2.55.0
Jakub Kicinski
maintainerPATCH 2/2 eth: bnxt: preserve IRQ affinity across IRQ reallocation
In reply to Jakub KicinskiReconfiguring the rings frees the MSI-X vectors and allocates them again. The IRQ descriptors go away with them, so the affinity user space set is silently replaced by the driver's default NUMA spread. This is painful to deal with for user space as seemingly arbitrary NIC configuration changes lead to loss of configuration.
In NIPA (netdev CI) this results in the toeplitz test reporting:
Exception| net.lib.py.ksft.KsftFailEx: IRQ170 is not mapped to a single core: 0-31
if the test run after another test which reconfigured the device. We configure the IRQ mapping at boot, but if the driver is not preserving the config - it gets lost.
Record the affinity in the notifier and apply it when the IRQs are requested again. The notifier has to be registered unconditionally now, so far it was only installed when TPH was enabled. Drivers which let the core manage the affinity (idpf, ice, iavf via netif_set_affinity_auto()) work exactly like this, napi_restore_config() reapplies napi_config.affinity_mask on every napi_enable().
Note that the affinity is supposed to follow the NAPI / queue, same as the napi_config behavior in drivers mentioned above. If the user changes the affinity when the device is down - we will override it on up. That's expected, the IRQs are not associated with queues when device is down (no name, no entry in /proc/interrupts, no entry in netdev netlink).
map_idx is ulp_msix + i, so the slot shifts whenever RoCE takes or releases vectors and the mask would end up on a different ring. Key using the completion ring id, which maps to the NAPI instance.
Note2: this restores the side effect fcf42409c6e1 ("bnxt_en: use irq_update_affinity_hint()") removed, but not the problem it was fixing. The complaint there was that reopening the device resets the affinity and can move an IRQ onto a CPU irqbalance was told to stay away from. We now replay what user space or irqbalance last asked for, the driver's own placement is only used for a ring nobody has configured.
Note3: the combined irq_set_affinity_and_hint() looks like it may hide the failure from __irq_set_affinity(), but let's assume the IRQ maintainers know what their doing - either this can't happen or is intentional.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 11 ++- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 95 +++++++++++++++++------ 2 files changed, 80 insertions(+), 26 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.h +9 -2
@@ -1261,9 +1261,7 @@ struct bnxt_irq {irq_handler_t handler;unsigned int vector; u8 requested:1; - u8 have_cpumask:1;char name[IFNAMSIZ +BNXT_IRQ_NAME_EXTRA ]; -cpumask_var_t cpu_mask;struct bnxt *bp;int msix_nr;
@@ -2482,6 +2480,15 @@ struct bnxt { pci_channel_offline((bp)->pdev))struct bnxt_irq *irq_tbl; +/* IRQ affinity, indexed by completion ring. Kept across IRQ +* reallocation, the MSI-X vector index is not stable. +*/ +cpumask_var_t *ring_cpu_mask; +/* Rings for which the mask above was configured from the outside, +* rather than being our own default placement. +*/ +unsigned long *ring_affinity_set; +int max_irqs;int total_irqs;int ulp_num_msix_want;u8 mac_addr[ETH_ALEN ];
drivers/net/ethernet/broadcom/bnxt/bnxt.c +71 -24
170 lines not on this page · raw message
Signature · 1 line
2.55.0
Re: [PATCH net-next 1/2] eth: bnxt: decrease indent in bnxt_init_int_mode()
In reply to Jakub KicinskiQuoted 5 lines · Jakub Kicinski
Handle the IRQ table allocation failure right away instead of wrapping the rest of the function in an if. Purely to make upcoming changes more readable.
Quoted 9 lines
While refactoring, drop the init of rc which is not necessary.
No functional changes.
Signed-off-by Jakub Kicinski <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 36 +++++++++++------------ 1 file changed, 18 insertions(+), 18 deletions(-)
Quoted 7 lines
Note2: this restores the side effect
fcf42409c6e1("bnxt_en: useirq_update_affinity_hint()") removed, but not the problem it was fixing. The complaint there was that reopening the device resets the affinity and can move an IRQ onto a CPU irqbalance was told to stay away from. We now replay what user space or irqbalance last asked for, the driver's own placement is only used for a ring nobody has configured.
Spl thanks for this Note2. This saved me time!
Quoted 6 lines
Note3: the combined
irq_set_affinity_and_hint()looks like it may hide the failure from__irq_set_affinity(), but let's assume the IRQ maintainers know what their doing - either this can't happen or is intentional.
Quoted 6 lines
Signed-off-by Jakub Kicinski <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 11 ++- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 95 +++++++++++++++++------ 2 files changed, 80 insertions(+), 26 deletions(-)
LGTM.
Jakub Kicinski
maintainerdrivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_request_irq(struct bnxt *bp) · line 12004
Quoted 2 lines · On Thu, 13 Aug 2026 12:32:48 -0700 Jakub Kicinski wrote:
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2
-0+* Set after installing the notifier, if we race with the user +* it's better to overwrite than miss the notification.
'shiko warns about the race between the notifier and the init. This is effectively the same pre-existing race we I'm flagging in this comment. We ignore updates while the device is down completely, I don't think it's worth trying to reliably delay the racing user programming after init has finished.
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 21 lines · On Thu, 13 Aug 2026 12:32:46 -0700 you wrote:
bnxt currently discards the IRQ affinity when changing ring count:
# ethtool -l ens9np0 [...] Combined: 8 [...] # ynl --family netdev --dump napi-get --json '{"ifindex": 2}' [...] {'defer-hard-irqs': 0, 'gro-flush-timeout': 0, 'id': 70, 'ifindex': 2, 'irq': 170, << IRQ 170 is for NAPI 1 (second to last) 'irq-suspend-timeout': 0, 'threaded': 'disabled'}, {'defer-hard-irqs': 0, 'gro-flush-timeout': 0, 'id': 69, 'ifindex': 2, 'irq': 169, 'irq-suspend-timeout': 0, 'threaded': 'disabled'}]
Quoted 1 line
[...]
Here is the summary with links:
- [net-next,1/2] eth: bnxt: decrease indent in bnxt_init_int_mode()
https://git.kernel.org/netdev/net-next/c/2b49709667de - [net-next,2/2] eth: bnxt: preserve IRQ affinity across IRQ reallocation https://git.kernel.org/netdev/net-next/c/fb0502649043
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html