igb: fix runtime PM resume issues
Subsystems (4)
v2 cover igb: fix runtime PM resume issues
igb can be runtime suspended when the link goes down (with power/control set to auto). Resuming it calls functions that need RTNL without holding it (patches 1 and 2), and devices that can't signal a wake-up are suspended anyway, so their port stays down when the link comes back (patch 3).
Tested with the 82576 emulated by QEMU.
Changes in v2:
- patch 1: keep setting the queue counts in __igb_open(), only skipping them on runtime resume (Sashiko)
- mention the tested hardware in the commit messages
v1: https://lore.kernel.org/netdev/[email protected]/
Antonio Quartulli (3):
igb: fix RTNL assertion when resuming from runtime suspend
igb: link queues to NAPIs under RTNL after runtime resume
igb: only allow runtime PM if the device can wake up
drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 69 ++++++++++++++++++----- 2 files changed, 57 insertions(+), 13 deletions(-)
Signature · 1 line
2.56.0
PATCH v2 1/3 igb: fix RTNL assertion when resuming from runtime suspend
Since commit ac8c58f5b535 ("igb: fix deadlock caused by taking RTNL in RPM resume path") __igb_resume() doesn't take RTNL for runtime resume, but __igb_open() still calls netif_set_real_num_{tx,rx}_queues(), which require it. An asynchronous runtime resume (e.g. after the link comes back) then triggers "RTNL: assertion failed".
Skip them when resuming. The counts are set again on every regular open, which happens under RTNL; the only case left out is a fallback from MSI-X to MSI during a resume, which leaves the stack with more queues than the device uses until the next open.
Tested with the 82576 emulated by QEMU.
---
v2:
- keep setting the queue counts in __igb_open() after igb_request_irq(), only skipping them on runtime resume: setting them in igb_open() before the device was resumed broke the MSI-X to MSI fallback and XDP setup on a runtime suspended device (Sashiko)
- drop the reference to igc, which has the same ordering problem
Note: igc sets the queue counts in igc_open() before igc_request_irq(), which can fall back to MSI, since 6f31d6b643a3 ("igc: Refactor runtime power management flow"). We didn't have igc hardware to test a fix with, so no igc patch is sent for now.
drivers/net/ethernet/intel/igb/igb_main.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +13 -9
@@ -4178,16 +4178,20 @@ static int __igb_open(struct net_device *netdev, bool resuming)if (err) goto err_req_irq; -/* Notify the stack of the actual queue counts. */ - err =netif_set_real_num_tx_queues (adapter->netdev, - adapter->num_tx_queues); -if (err) - goto err_set_queues; +/* Notify the stack of the actual queue counts. This needs RTNL, +* which isn't held when resuming from runtime suspend. +*/ +if (!resuming) { + err =netif_set_real_num_tx_queues (adapter->netdev, + adapter->num_tx_queues); +if (err) + goto err_set_queues; - err =netif_set_real_num_rx_queues (adapter->netdev, - adapter->num_rx_queues); -if (err) - goto err_set_queues; + err =netif_set_real_num_rx_queues (adapter->netdev, + adapter->num_rx_queues); +if (err) + goto err_set_queues; + }/* From here on the code is the same as igb_up() */ clear_bit (__IGB_DOWN, &adapter->state);
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH net v2 1/3] igb: fix RTNL assertion when resuming from runtime suspend
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] igb: stale real queue counts after resume MSI fallback in __igb_open() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] igb: stale real queue counts after resume MSI fallback in __igb_open()
--
commit 6c8e67bb8e01233489112f0bf629f1764d87d2c6
Author: Antonio Quartulli <[email protected]>
igb: fix RTNL assertion when resuming from runtime suspend
This commit skips calling netif_set_real_num_tx_queues() and netif_set_real_num_rx_queues() in __igb_open() when resuming is true to avoid triggering an RTNL assertion during runtime resume.
Quoted 24 lines
drivers/net/ethernet/intel/igb/igb_main.c +8
-7@@ -4178,16 +4178,20 @@ static int __igb_open(struct net_device *netdev, bool resuming)if (err) goto err_req_irq; -/* Notify the stack of the actual queue counts. */ - err =netif_set_real_num_tx_queues (adapter->netdev, - adapter->num_tx_queues); -if (err) - goto err_set_queues; +/* Notify the stack of the actual queue counts. This needs RTNL, +* which isn't held when resuming from runtime suspend. +*/ +if (!resuming) { + err =netif_set_real_num_tx_queues (adapter->netdev, + adapter->num_tx_queues); +if (err) + goto err_set_queues; - err =netif_set_real_num_rx_queues (adapter->netdev, - adapter->num_rx_queues);
drivers/net/ethernet/intel/igb/igb_main.c static int __igb_open(struct net_device *netdev, bool resuming) · line 4197
Quoted 10 lines
drivers/net/ethernet/intel/igb/igb_main.c +5
-2-if (err) - goto err_set_queues; + err =netif_set_real_num_rx_queues (adapter->netdev, + adapter->num_rx_queues); +if (err) + goto err_set_queues; + }/* From here on the code is the same as igb_up() */ clear_bit (__IGB_DOWN, &adapter->state);
[Severity: Medium]
Can leaving netdev->real_num_tx_queues and netdev->real_num_rx_queues stale when MSI-X falls back to MSI during resume cause concurrent unlocked TX on adapter->tx_ring[0] and a NULL pointer dereference in igb_xsk_pool_disable()?
During suspend, __igb_shutdown() -> igb_clear_interrupt_scheme() frees all q_vectors and clears adapter->tx_ring[] and adapter->rx_ring[]. If MSI-X allocation fails on resume (including system resume via igb_resume() -> __igb_resume(dev, false), which already holds rtnl_lock() but still passes resuming = true), the driver falls back to MSI with adapter->num_tx_queues and adapter->num_rx_queues set to 1, allocating only index 0 while netdev->real_num_tx_queues and netdev->real_num_rx_queues remain at N > 1.
In the TX path, the stack selects subqueues across 0 .. real_num_tx_queues - 1, each serialized only by its own netdev->_tx[q]._xmit_lock, but igb_tx_queue_mapping() folds all of them onto adapter->tx_ring[0]:
drivers/net/ethernet/intel/igb/igb_main.c:igb_tx_queue_mapping() {
unsigned int r_idx = skb->queue_mapping;
if (r_idx >= adapter->num_tx_queues)
r_idx = r_idx % adapter->num_tx_queues;
return adapter->tx_ring[r_idx];
}CPU0 (subqueue 0) CPU1 (subqueue 1)netdev_start_xmit() netdev_start_xmit()
HARD_TX_LOCK(dev, txq0) HARD_TX_LOCK(dev, txq1)
igb_xmit_frame() igb_xmit_frame()
igb_tx_queue_mapping() igb_tx_queue_mapping()
-> adapter->tx_ring[0] -> adapter->tx_ring[0]
igb_xmit_frame_ring(tx_ring[0]) igb_xmit_frame_ring(tx_ring[0])
first = &tx_ring-> first = &tx_ring->
tx_buffer_info[next_to_use] tx_buffer_info[next_to_use]
<-- concurrent ring corruptionMultiple CPUs transmitting on different subqueues concurrently run igb_xmit_frame_ring() and igb_tx_map() on adapter->tx_ring[0], corrupting tx_ring->next_to_use, tx_buffer_info, hardware descriptors, and BQL state.
In addition, igb_xsk_pool_disable() checks xsk_get_pool_from_qid(), which validates qid against real_num_rx_queues and real_num_tx_queues; disabling an AF_XDP pool bound to qid >= 1 after a resume fallback passes qid to igb_txrx_ring_disable(), dereferencing NULL adapter->tx_ring[qid] and adapter->rx_ring[qid].
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH net v2 1/3] igb: fix RTNL assertion when resuming from runtime suspend
In reply to Antonio QuartulliQuoted 10 lines · Antonio Quartulli
Since commit
ac8c58f5b535("igb: fix deadlock caused by taking RTNL in RPM resume path")__igb_resume()doesn't take RTNL for runtime resume, but__igb_open()still calls netif_set_real_num_{tx,rx}_queues(), which require it. An asynchronous runtime resume (e.g. after the link comes back) then triggers "RTNL: assertion failed".Skip them when resuming. The counts are set again on every regular open, which happens under RTNL; the only case left out is a fallback from MSI-X to MSI during a resume, which leaves the stack with more queues than the device uses until the next open.
RTNL is not held only on runtime resume, it is held on a normal resume. __igb_open() skips netif_set_real_num_{tx,rx}_queues() for both resume types. Maybe it makes sense to skip them only on runtime resume when the RTNL is not held?
May I ask why is it okay to leave out the fallback case, can't it lead to trouble if the counts are not updated but a fallback happens?
Best regards,
Janis
Re: [PATCH net v2 1/3] igb: fix RTNL assertion when resuming from runtime suspend
Hi Janis, thanks for the review!
Quoted 12 lines · Janis Edvarts Lacis
Quoted 10 lines · Antonio Quartulli
Since commit
ac8c58f5b535("igb: fix deadlock caused by taking RTNL in RPM resume path")__igb_resume()doesn't take RTNL for runtime resume, but__igb_open()still calls netif_set_real_num_{tx,rx}_queues(), which require it. An asynchronous runtime resume (e.g. after the link comes back) then triggers "RTNL: assertion failed".Skip them when resuming. The counts are set again on every regular open, which happens under RTNL; the only case left out is a fallback from MSI-X to MSI during a resume, which leaves the stack with more queues than the device uses until the next open.
Quoted 4 lines
RTNL is not held only on runtime resume, it is held on a normal resume.
__igb_open()skips netif_set_real_num_{tx,rx}_queues() for both resume types. Maybe it makes sense to skip them only on runtime resume when the RTNL is not held?
You're right.
RTNL is held on normal (system sleep) resume, so the counts can and should be set there.
Quoted 3 lines
May I ask why is it okay to leave out the fallback case, can't it lead to trouble if the counts are not updated but a fallback happens?
Ouch..you're right again.
The fallback case can't be left out: with fewer rings than the stack's real queue count, igb_tx_queue_mapping() maps several TX queues onto tx_ring[0], each under its own xmit lock, and igb_xsk_pool_disable() can index a ring that doesn't exist (Sashiko reported the same on this version....).
To summarize: in v3 I'll set the counts directly on normal resume and on runtime resume from a work item that takes RTNL (as patch 2 already does for the NAPI links).
Until the work item has run, igb_xmit_frame() will drop packets for queues the device doesn't have instead of mapping them onto tx_ring[0] (the stack drops them anyway when the count is lowered), and the XSK disable path will skip rings that don't exist.
Best,
Quoted 3 lines
Best regards,
Janis
Signature · 5 lines
Antonio Quartulli CEO and Co-Founder Mandelbit Srl https://www.mandelbit.com
PATCH v2 2/3 igb: link queues to NAPIs under RTNL after runtime resume
In reply to Antonio Quartulli__igb_open() links queues to NAPIs with netif_queue_set_napi(), which requires RTNL, but RTNL isn't held when resuming from runtime suspend:
this triggers "RTNL: assertion failed".
Don't link them in __igb_open() when resuming: do it right away on system resume (RTNL held) and from a work item taking RTNL on runtime resume. The links can't be kept across a suspend, because __igb_shutdown() frees the q_vectors. These are only freed under RTNL with __IGB_DOWN set, so the work item checks __IGB_DOWN under RTNL.
Tested with the 82576 emulated by QEMU.
---
v2: no code changes; mention the tested hardware.
Note: igc may have the same issue: __igc_open() also links queues to NAPIs (igc_set_queue_napi()) on the RTNL-less runtime resume path. We didn't have igc hardware to test a fix with, so no igc patch is sent for now.
drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 38 +++++++++++++++++++++-- 2 files changed, 37 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +1 -0
@@ -585,6 +585,7 @@ struct igb_adapter {struct work_struct reset_task;struct work_struct watchdog_task; +struct work_struct queue_napi_task;bool fc_autoneg;u8 tx_timeout_factor;struct timer_list blink_timer;
drivers/net/ethernet/intel/igb/igb_main.c +36 -2
@@ -2115,6 +2115,30 @@ void igb_set_queue_napi(struct igb_adapter *adapter, int vector, NETDEV_QUEUE_TYPE_TX, napi); } +static void igb_link_queue_napis (struct igb_adapter *adapter) +{ +int i; + +for (i =0 ; i < adapter->num_q_vectors; i++) +igb_set_queue_napi (adapter, i, &adapter->q_vector[i]->napi); +} + +/* Linking queues to NAPIs requires RTNL, which isn't held when resuming +* from runtime suspend: do it from here instead. The q_vectors are only +* freed under RTNL with __IGB_DOWN set, so checking it under RTNL is safe. +*/ +static void igb_queue_napi_task (struct work_struct *work) +{ + struct igb_adapter *adapter = container_of(work, struct igb_adapter, + queue_napi_task); + +rtnl_lock (); +if (!test_bit (__IGB_DOWN, &adapter->state) && +netif_running (adapter->netdev)) +igb_link_queue_napis (adapter); +rtnl_unlock (); +} +/** * igb_up - Open the interface and prepare it to handle traffic * @adapter: board private structure
@@ -3426,6 +3450,7 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)INIT_WORK (&adapter->reset_task, igb_reset_task);INIT_WORK (&adapter->watchdog_task, igb_watchdog_task); +INIT_WORK (&adapter->queue_napi_task, igb_queue_napi_task);/* Initialize link properties that are user-changeable */ adapter->fc_autoneg = true;
@@ -3890,6 +3915,7 @@ static void igb_remove(struct pci_dev *pdev)cancel_work_sync (&adapter->reset_task);cancel_work_sync (&adapter->watchdog_task); +cancel_work_sync (&adapter->queue_napi_task);#ifdef CONFIG_IGB_DCA if (adapter->flags &IGB_FLAG_DCA_ENABLED ) {
@@ -4199,7 +4225,8 @@ static int __igb_open(struct net_device *netdev, bool resuming)for (i =0 ; i < adapter->num_q_vectors; i++) { napi = &adapter->q_vector[i]->napi;napi_enable (napi); -igb_set_queue_napi (adapter, i, napi); +if (!resuming) +igb_set_queue_napi (adapter, i, napi); }/* Clear any pending interrupts. */
@@ -9637,8 +9664,15 @@ static int __igb_resume(struct device *dev, bool rpm)if (!rpm)rtnl_lock (); -if (!err &&netif_running (netdev)) +if (!err &&netif_running (netdev)) { err =__igb_open (netdev, true); +if (!err) { +if (rpm) +schedule_work (&adapter->queue_napi_task); +else +igb_link_queue_napis (adapter); + } + }if (!err)netif_device_attach (netdev);
Signature · 1 line
2.56.0
Re: [PATCH net v2 2/3] igb: link queues to NAPIs under RTNL after runtime resume
Quoted 16 lines · Antonio Quartulli
__igb_open()links queues to NAPIs withnetif_queue_set_napi(), which requires RTNL, but RTNL isn't held when resuming from runtime suspend:
this triggers "RTNL: assertion failed".Don't link them in
__igb_open()when resuming: do it right away on system resume (RTNL held) and from a work item taking RTNL on runtime resume. The links can't be kept across a suspend, because__igb_shutdown()frees theq_vectors. These are only freed under RTNL with__IGB_DOWNset, so the work item checks__IGB_DOWNunder RTNL.Tested with the 82576 emulated by QEMU.
Fixes b75a1dea500f ("igb: Link queues to NAPI instances")Cc [email protected] # 6.16+Signed-off-by Antonio Quartulli <[email protected]>
Quoted 6 lines
---
v2: no code changes; mention the tested hardware.Note: igc may have the same issue:
__igc_open()also links queues to NAPIs (igc_set_queue_napi()) on the RTNL-less runtime resume path. We didn't have igc hardware to test a fix with, so no igc patch is sent for now.
I have an igc I can test on if you want to send me the script you used to test this. Feel free to message me off list if you like.
Quoted 3 lines
drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 38 +++++++++++++++++++++-- 2 files changed, 37 insertions(+), 2 deletions(-)
PATCH v2 3/3 igb: only allow runtime PM if the device can wake up
In reply to Antonio Quartulliigb lets the device be runtime suspended even if it can't signal a wake-up from D3. Such a device doesn't notice the link coming back, so the port stays down until it is resumed from user space.
Allow runtime PM only if pci_dev_run_wake() is true, as e1000e and r8169 do. Devices that can't wake up no longer runtime suspend, even with power/control set to auto.
Tested with the 82576 emulated by QEMU.
---
v2: no code changes; mention the tested hardware.
Note: igc may have the same issue: igc_probe() also allows runtime PM regardless of pci_dev_run_wake(). We didn't have igc hardware to test a fix with, so no igc patch is sent for now.
drivers/net/ethernet/intel/igb/igb_main.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +7 -2
@@ -3677,7 +3677,11 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)dev_pm_set_driver_flags (&pdev->dev,DPM_FLAG_NO_DIRECT_COMPLETE ); -pm_runtime_put_noidle (&pdev->dev); +/* A device that can't wake up wouldn't notice the link coming back +* while runtime suspended +*/ +if (pci_dev_run_wake (pdev)) +pm_runtime_put_noidle (&pdev->dev);return 0 ; err_register:
@@ -3900,7 +3904,8 @@ static void igb_remove(struct pci_dev *pdev)struct igb_adapter *adapter =netdev_priv (netdev);struct e1000_hw *hw = &adapter->hw; -pm_runtime_get_noresume (&pdev->dev); +if (pci_dev_run_wake (pdev)) +pm_runtime_get_noresume (&pdev->dev);#ifdef CONFIG_IGB_HWMON igb_sysfs_exit (adapter);#endif
Signature · 1 line
2.56.0