Skip to contentSkip to conversation

Fixes in igbvf driver

PATCH In review Wander Lairson Costa 6 participants 11 messages Sep 20, 2024 – Oct 21, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Fixes in igbvf driver

The first patch fixes a bug manifested manifested in the igbvf driver when interrupting handling for the igb driver delays.

The second is just a cleanup for igbvf.

Wander Lairson Costa (2):
igb: Disable threaded IRQ for igb_msix_other
igbvf: remove unused spinlock

 drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
 drivers/net/ethernet/intel/igbvf/igbvf.h  | 3 ---
 drivers/net/ethernet/intel/igbvf/netdev.c | 3 ---
 3 files changed, 1 insertion(+), 7 deletions(-)
Signature · 1 line
2.46.1
#1 lore raw

PATCH 1/2 igb: Disable threaded IRQ for igb_msix_other

During testing of SR-IOV, Red Hat QE encountered an issue where the ip link up command intermittently fails for the igbvf interfaces when using the PREEMPT_RT variant. Investigation revealed that e1000_write_posted_mbx returns an error due to the lack of an ACK from e1000_poll_for_ack.

The underlying issue arises from the fact that IRQs are threaded by default under PREEMPT_RT. While the exact hardware details are not available, it appears that the IRQ handled by igb_msix_other must be processed before e1000_poll_for_ack times out. However, e1000_write_posted_mbx is called with preemption disabled, leading to a scenario where the IRQ is serviced only after the failure of e1000_write_posted_mbx.

To resolve this, we set IRQF_NO_THREAD for the affected interrupt, ensuring that the kernel handles it immediately, thereby preventing the aforementioned error.

Reproducer:

    #!/bin/bash

    # echo 2 > /sys/class/net/ens14f0/device/sriov_numvfs
    ipaddr_vlan=3
    nic_test=ens14f0
    vf=${nic_test}v0

    while true; do
	    ip link set ${nic_test} mtu 1500
	    ip link set ${vf} mtu 1500
	    ip link set $vf up
	    ip link set ${nic_test} vf 0 vlan ${ipaddr_vlan}
	    ip addr add 172.30.${ipaddr_vlan}.1/24 dev ${vf}
	    ip addr add 2021:db8:${ipaddr_vlan}::1/64 dev ${vf}
	    if ! ip link show $vf | grep 'state UP'; then
		    echo 'Error found'
		    break
	    fi
	    ip link set $vf down
    done
---
 drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
index 1ef4cb871452..8a1696d7289f 100644
@@ -907,7 +907,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
 	int i, err = 0, vector = 0, free_vector = 0;
 
 	err = request_irq(adapter->msix_entries[vector].vector,
-			  igb_msix_other, 0, netdev->name, adapter);
+			  igb_msix_other, IRQF_NO_THREAD, netdev->name, adapter);
 	if (err)
 		goto err_out;
 
Signature · 1 line
2.46.1
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] igb: Disable threaded IRQ for igb_msix_other

Quoted 52 lines · Wander Lairson Costa

During testing of SR-IOV, Red Hat QE encountered an issue where the ip link up command intermittently fails for the igbvf interfaces when using the PREEMPT_RT variant. Investigation revealed that e1000_write_posted_mbx returns an error due to the lack of an ACK from e1000_poll_for_ack.

The underlying issue arises from the fact that IRQs are threaded by default under PREEMPT_RT. While the exact hardware details are not available, it appears that the IRQ handled by igb_msix_other must be processed before e1000_poll_for_ack times out. However, e1000_write_posted_mbx is called with preemption disabled, leading to a scenario where the IRQ is serviced only after the failure of e1000_write_posted_mbx.

To resolve this, we set IRQF_NO_THREAD for the affected interrupt, ensuring that the kernel handles it immediately, thereby preventing the aforementioned error.

Reproducer:

     #!/bin/bash

     # echo 2 > /sys/class/net/ens14f0/device/sriov_numvfs
     ipaddr_vlan=3
     nic_test=ens14f0
     vf=${nic_test}v0

     while true; do
	    ip link set ${nic_test} mtu 1500
	    ip link set ${vf} mtu 1500
	    ip link set $vf up
	    ip link set ${nic_test} vf 0 vlan ${ipaddr_vlan}
	    ip addr add 172.30.${ipaddr_vlan}.1/24 dev ${vf}
	    ip addr add 2021:db8:${ipaddr_vlan}::1/64 dev ${vf}
	    if ! ip link show $vf | grep 'state UP'; then
		    echo 'Error found'
		    break
	    fi
	    ip link set $vf down
     done
---
  drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 1ef4cb871452..8a1696d7289f 100644

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 913

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -907,7 +907,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
  	int i, err = 0, vector = 0, free_vector = 0;
 
  	err = request_irq(adapter->msix_entries[vector].vector,
-			  igb_msix_other, 0, netdev->name, adapter);
+			  igb_msix_other, IRQF_NO_THREAD, netdev->name, adapter);
  	if (err)
  		goto err_out;
 

Thank you for small, localized fix with a good description. Our VAL will check it also on non-RT OS.

PS: for future intel ethernet submissions please split out fixes and refactors, and tag each commit with the [iwl-net] or [iwl-next] tags

Tested-by Rafal Romanowski
#3 lore raw

Re: [Intel-wired-lan] [PATCH 1/2] igb: Disable threaded IRQ for igb_msix_other

From: Intel-wired-lan <[email protected]> on behalf of Przemek Kitszel <[email protected]>
Sent: Monday, September 23, 2024 11:07 AM
To: Wander Lairson Costa <[email protected]>
Cc: Yuying Ma <[email protected]>; moderated list:INTEL ETHERNET DRIVERS <[email protected]>; open list <[email protected]>; Eric Dumazet <[email protected]>; Nguyen, Anthony L <[email protected]>; open list:NETWORKING DRIVERS <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; David S. Miller <[email protected]>
Subject: Re: [Intel-wired-lan] [PATCH 1/2] igb: Disable threaded IRQ for igb_msix_other

Quoted 52 lines · Wander Lairson Costa

During testing of SR-IOV, Red Hat QE encountered an issue where the ip link up command intermittently fails for the igbvf interfaces when using the PREEMPT_RT variant. Investigation revealed that e1000_write_posted_mbx returns an error due to the lack of an ACK from e1000_poll_for_ack.

The underlying issue arises from the fact that IRQs are threaded by default under PREEMPT_RT. While the exact hardware details are not available, it appears that the IRQ handled by igb_msix_other must be processed before e1000_poll_for_ack times out. However, e1000_write_posted_mbx is called with preemption disabled, leading to a scenario where the IRQ is serviced only after the failure of e1000_write_posted_mbx.

To resolve this, we set IRQF_NO_THREAD for the affected interrupt, ensuring that the kernel handles it immediately, thereby preventing the aforementioned error.

Reproducer:

     #!/bin/bash

     # echo 2 > /sys/class/net/ens14f0/device/sriov_numvfs
     ipaddr_vlan=3
     nic_test=ens14f0
     vf=${nic_test}v0

     while true; do
           ip link set ${nic_test} mtu 1500
           ip link set ${vf} mtu 1500
           ip link set $vf up
           ip link set ${nic_test} vf 0 vlan ${ipaddr_vlan}
           ip addr add 172.30.${ipaddr_vlan}.1/24 dev ${vf}
           ip addr add 2021:db8:${ipaddr_vlan}::1/64 dev ${vf}
           if ! ip link show $vf | grep 'state UP'; then
                   echo 'Error found'
                   break
           fi
           ip link set $vf down
     done
---
  drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 1ef4cb871452..8a1696d7289f 100644

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 913

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -907,7 +907,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
       int i, err = 0, vector = 0, free_vector = 0;
 
       err = request_irq(adapter->msix_entries[vector].vector,
-                       igb_msix_other, 0, netdev->name, adapter);
+                       igb_msix_other, IRQF_NO_THREAD, netdev->name, adapter);
       if (err)
               goto err_out;
 
#4 lore raw

PATCH 2/2 igbvf: remove unused spinlock

In reply to Wander Lairson Costa

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

---
 drivers/net/ethernet/intel/igbvf/igbvf.h  | 3 ---
 drivers/net/ethernet/intel/igbvf/netdev.c | 3 ---
 2 files changed, 6 deletions(-)
drivers/net/ethernet/intel/igbvf/igbvf.h +0 -3
index 6ad35a00a287..ca6e44245a7b 100644
@@ -169,8 +169,6 @@ struct igbvf_adapter {
 	u16 link_speed;
 	u16 link_duplex;
 
-	spinlock_t tx_queue_lock; /* prevent concurrent tail updates */
-
 	/* track device up/down/testing state */
 	unsigned long state;
 
@@ -220,7 +218,6 @@ struct igbvf_adapter {
 	/* OS defined structs */
 	struct net_device *netdev;
 	struct pci_dev *pdev;
-	spinlock_t stats_lock; /* prevent concurrent stats updates */
 
 	/* structs defined in e1000_hw.h */
 	struct e1000_hw hw;
drivers/net/ethernet/intel/igbvf/netdev.c +0 -3
index 925d7286a8ee..02044aa2181b 100644
@@ -1656,12 +1656,9 @@ static int igbvf_sw_init(struct igbvf_adapter *adapter)
 	if (igbvf_alloc_queues(adapter))
 		return -ENOMEM;
 
-	spin_lock_init(&adapter->tx_queue_lock);
-
 	/* Explicitly disable IRQ since the NIC can be in any state. */
 	igbvf_irq_disable(adapter);
 
-	spin_lock_init(&adapter->stats_lock);
 	spin_lock_init(&adapter->hw.mbx_lock);
 
 	set_bit(__IGBVF_DOWN, &adapter->state);
Signature · 1 line
2.46.1
Reviewed-by on v1 2/2
#5 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Dear Wander,

Thank you for your patch.

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 4 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

It’d be great if you added a Fixes: tag.

Quoted 33 lines
---
  drivers/net/ethernet/intel/igbvf/igbvf.h  | 3 ---
  drivers/net/ethernet/intel/igbvf/netdev.c | 3 ---
  2 files changed, 6 deletions(-)
drivers/net/ethernet/intel/igbvf/igbvf.h +0 -3
index 6ad35a00a287..ca6e44245a7b 100644
@@ -169,8 +169,6 @@ struct igbvf_adapter {
  	u16 link_speed;
  	u16 link_duplex;
 
-	spinlock_t tx_queue_lock; /* prevent concurrent tail updates */
-
  	/* track device up/down/testing state */
  	unsigned long state;
 
@@ -220,7 +218,6 @@ struct igbvf_adapter {
  	/* OS defined structs */
  	struct net_device *netdev;
  	struct pci_dev *pdev;
-	spinlock_t stats_lock; /* prevent concurrent stats updates */
 
  	/* structs defined in e1000_hw.h */
  	struct e1000_hw hw;
drivers/net/ethernet/intel/igbvf/netdev.c +0 -0
index 925d7286a8ee..02044aa2181b 100644
@@ -1656,12 +1656,9 @@ static int igbvf_sw_init(struct igbvf_adapter *adapter)
  	if (igbvf_alloc_queues(adapter))
  		return -ENOMEM;

drivers/net/ethernet/intel/igbvf/netdev.c static int igbvf_sw_init(struct igbvf_adapter *adapter) · line 1664

Quoted 10 lines
drivers/net/ethernet/intel/igbvf/netdev.c +0 -3
 
-	spin_lock_init(&adapter->tx_queue_lock);
-
  	/* Explicitly disable IRQ since the NIC can be in any state. */
  	igbvf_irq_disable(adapter);
 
-	spin_lock_init(&adapter->stats_lock);
  	spin_lock_init(&adapter->hw.mbx_lock);
 
  	set_bit(__IGBVF_DOWN, &adapter->state);

With that addressed:

Kind regards,

Paul

maintainer
#6 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Quoted 5 lines · Paul Menzel

Dear Wander,

Thank you for your patch.

Quoted 7 lines

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 4 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

It’d be great if you added a Fixes: tag.

Alternatively you could split this series into two, and send this patch to iwl-next tree, without the fixes tag. For me this patch is just a cleanup, not a fix.

Quoted 1 line

[...]

Quoted 9 lines

With that addressed:

Kind regards,

Paul

#7 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Quoted 11 lines · Przemek Kitszel
Quoted 9 lines · Paul Menzel

Dear Wander,

Thank you for your patch.

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 3 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

It’d be great if you added a Fixes: tag.

Alternatively you could split this series into two, and send this patch to iwl-next tree, without the fixes tag. For me this patch is just a cleanup, not a fix.

Quoted 1 line

Should I send a new version of the patches separately?

Quoted 12 lines

[...]

Quoted 9 lines

With that addressed:

Kind regards,

Paul

maintainer
#8 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Quoted 17 lines · Wander Lairson Costa
Quoted 15 lines · Przemek Kitszel
Quoted 12 lines · Paul Menzel

Dear Wander,

Thank you for your patch.

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 4 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

It’d be great if you added a Fixes: tag.

Quoted 8 lines
Quoted 6 lines

Alternatively you could split this series into two, and send this patch to iwl-next tree, without the fixes tag. For me this patch is just a cleanup, not a fix.

Quoted 1 line

Should I send a new version of the patches separately?

The patches apply to the respective trees when split out so I can take these without a re-send. Patch 1 will need a Fixes: for it though...

I'm seeing it as: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")?

Thanks,
Tony

Quoted 13 lines
Quoted 12 lines

[...]

Quoted 9 lines

With that addressed:

Kind regards,

Paul

#9 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Quoted 28 lines · Tony Nguyen
Quoted 24 lines · Wander Lairson Costa
Quoted 21 lines · Przemek Kitszel
Quoted 5 lines · Paul Menzel

Dear Wander,

Thank you for your patch.

Quoted 7 lines

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 4 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

It’d be great if you added a Fixes: tag.

Alternatively you could split this series into two, and send this patch to iwl-next tree, without the fixes tag. For me this patch is just a cleanup, not a fix.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Should I send a new version of the patches separately?

The patches apply to the respective trees when split out so I can take these without a re-send. Patch 1 will need a Fixes: for it though...

I'm seeing it as: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")?

Can you add the tag when you apply the patch or should I add it?

Quoted 17 lines

Thanks,
Tony

Quoted 13 lines
Quoted 12 lines

[...]

Quoted 9 lines

With that addressed:

Kind regards,

Paul

#10 lore raw

Re: [Intel-wired-lan] [PATCH 2/2] igbvf: remove unused spinlock

Quoted 31 lines · Wander Lairson Costa
Quoted 30 lines · Tony Nguyen
Quoted 25 lines · Wander Lairson Costa
Quoted 11 lines · Przemek Kitszel
Quoted 9 lines · Paul Menzel

Dear Wander,

Thank you for your patch.

Am 20.09.24 um 20:59 schrieb Wander Lairson Costa:

Quoted 3 lines

tx_queue_lock and stats_lock are declared and initialized, but never used. Remove them.

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

It’d be great if you added a Fixes: tag.

Alternatively you could split this series into two, and send this patch to iwl-next tree, without the fixes tag. For me this patch is just a cleanup, not a fix.

Quoted 1 line

Should I send a new version of the patches separately?

Quoted 9 lines
Quoted 6 lines

The patches apply to the respective trees when split out so I can take these without a re-send. Patch 1 will need a Fixes: for it though...

I'm seeing it as: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")?

Can you add the tag when you apply the patch or should I add it?

I will add the fixes tag when I said it.

Thanks,
Jake