net: mana: Avoid waiting on dead HWC after device reset
Subsystems (6)
PATCH v4 1/1 net: mana: Avoid waiting on dead HWC after device reset
After device reset is issued due to TX drain timeout in mana_dealloc_queues(), hardware channel (HWC) state is non-functional. However, the driver continues to issue HWC commands (mana_disable_vport_rx, mana_fence_rqs, mana_destroy_wq_obj) which will each wait for a completion that never arrives.
The first such command (MANA_CONFIG_VPORT_RX) blocks for the full hwc_timeout (60s as set by the firmware on the tested system), after which the timeout is reduced to 1 ms for subsequent commands. This causes the calling thread to hold the RTNL lock for over 180 seconds (120s TX drain + 60s HWC timeout), triggering hung task warnings and blocking all other networking operations on the system.
Fix this by:
1. Replacing the sleep-based TX drain with a CQE polling approach:
disable NAPI per queue and poll its CQ synchronously, with up to 3 more polls 100 ms apart to collect late completions. Reset the device only if sends remain unacknowledged after that, then free uncompleted SKBs. The device is reset at most once per call, and not at all if an earlier reset already marked the HWC dead. Once a queue is found stuck or the HWC is dead, the remaining queues are polled only once. This replaces the 120s drain timeout with at most 300 ms of sleep per queue. If queue is still stuck, mana_gd_dev_reset() waits for device to return. The 300 ms budget is not based on a hardware limit but in testing under load, the completions still outstanding on a queue had all arrived by its second poll.
2. Setting the hwc_dead flag after a successful device reset. If the reset fails, the device may not have been reset and may still access memory, so teardown commands are still posted.
3. Checking the hwc_dead flag in mana_hwc_send_request() before taking an HWC message slot and again before posting the WQE, and returning -ETIMEDOUT immediately without submitting commands to hardware that cannot respond. A request that was already posted, or that passed these checks just before hwc_dead was set (for example from the gf_stats worker or mana_ib, which do not hold RTNL), still waits for its own timeout, as before this patch.
4. Skipping mana_fence_rqs() and the SHM channel teardown handshake once hwc_dead is set.
---
Changes in v4:
- Keep netif_napi_del_locked() in mana_destroy_txq() after the WQ
object is destroyed; track the dealloc-time disable with
txq->napi_disabled.
- Drop lowering hwc_timeout on reset failure.
- Fix commit message based on review comments.
Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Resolve sashiko comments, poll last 100 ms before device reset.
- Stop the drain retries once the HWC is dead.
- Fail fast in mana_hwc_send_request() only when hwc_dead is set, not
when hwc_timeout is 0, so requests made after mana_serv_reset() are
still posted as before this patch.
- When the device reset fails, do not set hwc_dead, so teardown
commands are still posted, but lower hwc_timeout to 1 ms so they do
not each wait the full timeout
- Read hwc_timeout once with READ_ONCE() and lower it with cmpxchg(),
so a concurrent 0 from mana_serv_reset() is not overwritten.
- Use READ_ONCE() for all hwc_dead reads.
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Add bounded wait for late TX completions before deciding that a
queue is stuck, poll the CQ up to 3 times, 100 ms apart.
- check hwc_timeout and hwc_dead before taking an HWC message slot,
not only before posting the WQE.
Link to v1: https://lore.kernel.org/all/[email protected]/
.../net/ethernet/microsoft/mana/gdma_main.c | 6 +- .../net/ethernet/microsoft/mana/gdma_pci.c | 2 +- .../net/ethernet/microsoft/mana/hw_channel.c | 40 ++++-- drivers/net/ethernet/microsoft/mana/mana_en.c | 118 +++++++++++++----- include/net/mana/hw_channel.h | 1 + include/net/mana/mana.h | 3 + 6 files changed, 130 insertions(+), 40 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -2
@@ -141,7 +141,7 @@ int mana_gd_init_registers(struct gdma_context *gc)return mana_gd_init_vf_regs (gc); } -/* Suppress logging when we set timeout to zero */ +/* Suppress logging when we set timeout to zero or hwc is dead */ bool mana_need_log (struct gdma_context *gc,int err) {struct hw_channel_context *hwc;
@@ -153,7 +153,9 @@ bool mana_need_log(struct gdma_context *gc, int err)return true; hwc = gc->hwc.driver_data; -if (hwc && hwc->hwc_timeout ==0 ) +if (hwc && + (READ_ONCE (hwc->hwc_timeout) ==0 || +READ_ONCE (hwc->hwc_dead)))return false;return true;
drivers/net/ethernet/microsoft/mana/gdma_pci.c +1 -1
@@ -102,7 +102,7 @@ static void mana_serv_reset(struct pci_dev *pdev) }/* HWC is not responding in this case, so don't wait */ - hwc->hwc_timeout =0 ; +WRITE_ONCE (hwc->hwc_timeout,0 );dev_info (&pdev->dev,"MANA reset cycle start\n" );
drivers/net/ethernet/microsoft/mana/hw_channel.c +32 -8
@@ -192,7 +192,7 @@ static void mana_hwc_init_event_handler(void *ctx, struct gdma_queue *q_self,switch (type) {case HWC_DATA_CFG_HWC_TIMEOUT : - hwc->hwc_timeout = val; +WRITE_ONCE (hwc->hwc_timeout, val);break ;case HWC_DATA_HW_LINK_CONNECT :
@@ -816,10 +816,16 @@ void mana_hwc_destroy_channel(struct gdma_context *gc)return ;/* gc->max_num_cqs is set in mana_hwc_init_event_handler(). If it's -* non-zero, the HWC worked and we should tear down the HWC here. +* non-zero, the HWC worked and we should tear down the HWC here, +* unless device reset has already marked it dead: the device then +* no longer holds the HWC state and the handshake would only spin. */ if (gc->max_num_cqs >0 ) { -mana_smc_teardown_hwc (&gc->shm_channel, false); +if (READ_ONCE (hwc->hwc_dead)) +dev_warn (hwc->dev, +"HWC unavailable, skip SHM teardown handshake\n" ); +else +mana_smc_teardown_hwc (&gc->shm_channel, false);WRITE_ONCE (gc->max_num_cqs,0 ); }
@@ -867,9 +873,17 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,u32 dest_vrcq;u32 dest_vrq;u32 command; +u32 timeout;u16 msg_id;int err; +/* Fail fast before taking the HWC slot. After a device reset, requests +* already holding slots will release them once their own timeout +* expires. +*/ +if (READ_ONCE (hwc->hwc_dead)) +return -ETIMEDOUT ; +mana_hwc_get_msg_index (hwc, &msg_id); tx_wr = &txq->msg_buf->reqs[msg_id];
@@ -902,21 +916,31 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, dest_vrq = hwc->dest_vrq_id; dest_vrcq = hwc->dest_vrcq_id; +/* If the HWC is unavailable after device reset, do not post the WQE: +* the hardware cannot respond, so fail fast instead of waiting for +* the full timeout. +*/ +if (READ_ONCE (hwc->hwc_dead)) { + err = -ETIMEDOUT ; + goto out; + } + err =mana_hwc_post_tx_wqe (txq, tx_wr, dest_vrq, dest_vrcq, false);if (err) {dev_err (hwc->dev,"HWC: Failed to post send WQE: %d\n" , err); goto out; } + timeout =READ_ONCE (hwc->hwc_timeout);if (!wait_for_completion_timeout (&ctx->comp_event, - (msecs_to_jiffies (hwc->hwc_timeout)))) { -if (hwc->hwc_timeout !=0 ) + (msecs_to_jiffies (timeout)))) { +if (timeout)dev_err (hwc->dev,"Command 0x%x timed out: %u ms\n" , - command, hwc->hwc_timeout); + command, timeout);/* Reduce further waiting if HWC no response */ -if (hwc->hwc_timeout >1 ) - hwc->hwc_timeout =1 ; +if (timeout >1 ) +cmpxchg (&hwc->hwc_timeout, timeout,1 ); err = -ETIMEDOUT ; goto out;
drivers/net/ethernet/microsoft/mana/mana_en.c +89 -29
181 lines not on this page · raw message
include/net/mana/hw_channel.h +1 -0
@@ -179,6 +179,7 @@ struct hw_channel_context {struct gdma_dev *gdma_dev;struct device *dev; +bool hwc_dead;u16 num_inflight_msg;u32 max_req_msg_size;
include/net/mana/mana.h +3 -0
@@ -143,6 +143,7 @@ struct mana_txq {atomic_t pending_sends;bool napi_initialized; +bool napi_disabled;struct mana_stats_tx stats; };
@@ -344,6 +345,8 @@ struct mana_cq {/* Cumulative TX completions fed to DIM. Updated and read only in * NAPI context (mana_poll_tx_cq() / mana_update_tx_dim()), so they * measure the hardware completion rate and need no u64_stats_sync. +* The TX drain in mana_dealloc_queues() also updates them, but only +* after NAPI is disabled and the DIM work is cancelled. */ u64 tx_dim_pkts;u64 tx_dim_bytes;
--
2.53.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues.
Quoted 13 lines
After device reset is issued due to TX drain timeout in
mana_dealloc_queues(), hardware channel (HWC) state is non-functional. However, the driver continues to issue HWC commands (mana_disable_vport_rx,mana_fence_rqs,mana_destroy_wq_obj) which will each wait for a completion that never arrives.The first such command (
MANA_CONFIG_VPORT_RX) blocks for the fullhwc_timeout(60s as set by the firmware on the tested system), after which the timeout is reduced to 1 ms for subsequent commands. This causes the calling thread to hold the RTNL lock for over 180 seconds (120s TX drain + 60s HWC timeout), triggering hung task warnings and blocking all other networking operations on the system.
Quoted 8 lines
Fix this by:
1. Replacing the sleep-based TX drain with a CQE polling approach:[ ... ]Fixes a7dfeda6fdec ("net: mana: Fix MANA VF unload when hardware is unresponsive")Co-developed-by Dipayaan Roy <[email protected]>Signed-off-by Dipayaan Roy <[email protected]>Signed-off-by Sahil Chandna <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1