Skip to contentSkip to conversation

net: mana: Avoid waiting on dead HWC after device reset

PATCH New Sahil Chandna 2 participants 2 messages Nov 18

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 net: mana: Avoid waiting on dead HWC after device reset

After device reset is issued due to TX drain timeout in mana_dealloc_queues(), hardware channel (HWC) state is non-functional. However, the driver continues to issue HWC commands (mana_disable_vport_rx, mana_fence_rqs, mana_destroy_wq_obj) which will each wait for a completion that never arrives.

The first such command (MANA_CONFIG_VPORT_RX) blocks for the full hwc_timeout (60s as set by the firmware on the tested system), after which the timeout is reduced to 1 ms for subsequent commands. This causes the calling thread to hold the RTNL lock for over 180 seconds (120s TX drain + 60s HWC timeout), triggering hung task warnings and blocking all other networking operations on the system.

Fix this by:
1. Replacing the sleep-based TX drain with a CQE polling approach:
disable NAPI per queue and poll its CQ synchronously, with up to 3 more polls 100 ms apart to collect late completions. Reset the device only if sends remain unacknowledged after that, then free uncompleted SKBs. The device is reset at most once per call, and not at all if an earlier reset already marked the HWC dead. Once a queue is found stuck or the HWC is dead, the remaining queues are polled only once. This replaces the 120s drain timeout with at most 300 ms of sleep per queue. If queue is still stuck, mana_gd_dev_reset() waits for device to return. The 300 ms budget is not based on a hardware limit but in testing under load, the completions still outstanding on a queue had all arrived by its second poll.
2. Setting the hwc_dead flag after a successful device reset. If the reset fails, the device may not have been reset and may still access memory, so teardown commands are still posted.
3. Checking the hwc_dead flag in mana_hwc_send_request() before taking an HWC message slot and again before posting the WQE, and returning -ETIMEDOUT immediately without submitting commands to hardware that cannot respond. A request that was already posted, or that passed these checks just before hwc_dead was set (for example from the gf_stats worker or mana_ib, which do not hold RTNL), still waits for its own timeout, as before this patch.
4. Skipping mana_fence_rqs() and the SHM channel teardown handshake once hwc_dead is set.

Fixes a7dfeda6fdec ("net: mana: Fix MANA VF unload when hardware is unresponsive")

---
Changes in v4:
- Keep netif_napi_del_locked() in mana_destroy_txq() after the WQ
object is destroyed; track the dealloc-time disable with
txq->napi_disabled.
- Drop lowering hwc_timeout on reset failure.
- Fix commit message based on review comments.
Link to v3: https://lore.kernel.org/all/[email protected]/

Changes in v3:
- Resolve sashiko comments, poll last 100 ms before device reset.
- Stop the drain retries once the HWC is dead.
- Fail fast in mana_hwc_send_request() only when hwc_dead is set, not
when hwc_timeout is 0, so requests made after mana_serv_reset() are
still posted as before this patch.
- When the device reset fails, do not set hwc_dead, so teardown
commands are still posted, but lower hwc_timeout to 1 ms so they do
not each wait the full timeout
- Read hwc_timeout once with READ_ONCE() and lower it with cmpxchg(),
so a concurrent 0 from mana_serv_reset() is not overwritten.
- Use READ_ONCE() for all hwc_dead reads.
Link to v2: https://lore.kernel.org/all/[email protected]/

Changes in v2:
- Add bounded wait for late TX completions before deciding that a
queue is stuck, poll the CQ up to 3 times, 100 ms apart.
- check hwc_timeout and hwc_dead before taking an HWC message slot,
not only before posting the WQE.
Link to v1: https://lore.kernel.org/all/[email protected]/

 .../net/ethernet/microsoft/mana/gdma_main.c   |   6 +-
 .../net/ethernet/microsoft/mana/gdma_pci.c    |   2 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  |  40 ++++--
 drivers/net/ethernet/microsoft/mana/mana_en.c | 118 +++++++++++++-----
 include/net/mana/hw_channel.h                 |   1 +
 include/net/mana/mana.h                       |   3 +
 6 files changed, 130 insertions(+), 40 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -2
index d6ae89c2ccdd..db330426320e 100644
@@ -141,7 +141,7 @@ int mana_gd_init_registers(struct gdma_context *gc)
 		return mana_gd_init_vf_regs(gc);
 }
 
-/* Suppress logging when we set timeout to zero */
+/* Suppress logging when we set timeout to zero or hwc is dead */
 bool mana_need_log(struct gdma_context *gc, int err)
 {
 	struct hw_channel_context *hwc;
@@ -153,7 +153,9 @@ bool mana_need_log(struct gdma_context *gc, int err)
 		return true;
 
 	hwc = gc->hwc.driver_data;
-	if (hwc && hwc->hwc_timeout == 0)
+	if (hwc &&
+	    (READ_ONCE(hwc->hwc_timeout) == 0 ||
+	     READ_ONCE(hwc->hwc_dead)))
 		return false;
 
 	return true;
drivers/net/ethernet/microsoft/mana/gdma_pci.c +1 -1
index 208dadbcf977..f9d75b5d9e43 100644
@@ -102,7 +102,7 @@ static void mana_serv_reset(struct pci_dev *pdev)
 	}
 
 	/* HWC is not responding in this case, so don't wait */
-	hwc->hwc_timeout = 0;
+	WRITE_ONCE(hwc->hwc_timeout, 0);
 
 	dev_info(&pdev->dev, "MANA reset cycle start\n");
 
drivers/net/ethernet/microsoft/mana/hw_channel.c +32 -8
index fb9f38f65911..1671944684b8 100644
@@ -192,7 +192,7 @@ static void mana_hwc_init_event_handler(void *ctx, struct gdma_queue *q_self,
 
 		switch (type) {
 		case HWC_DATA_CFG_HWC_TIMEOUT:
-			hwc->hwc_timeout = val;
+			WRITE_ONCE(hwc->hwc_timeout, val);
 			break;
 
 		case HWC_DATA_HW_LINK_CONNECT:
@@ -816,10 +816,16 @@ void mana_hwc_destroy_channel(struct gdma_context *gc)
 		return;
 
 	/* gc->max_num_cqs is set in mana_hwc_init_event_handler(). If it's
-	 * non-zero, the HWC worked and we should tear down the HWC here.
+	 * non-zero, the HWC worked and we should tear down the HWC here,
+	 * unless device reset has already marked it dead: the device then
+	 * no longer holds the HWC state and the handshake would only spin.
 	 */
 	if (gc->max_num_cqs > 0) {
-		mana_smc_teardown_hwc(&gc->shm_channel, false);
+		if (READ_ONCE(hwc->hwc_dead))
+			dev_warn(hwc->dev,
+				 "HWC unavailable, skip SHM teardown handshake\n");
+		else
+			mana_smc_teardown_hwc(&gc->shm_channel, false);
 		WRITE_ONCE(gc->max_num_cqs, 0);
 	}
 
@@ -867,9 +873,17 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 	u32 dest_vrcq;
 	u32 dest_vrq;
 	u32 command;
+	u32 timeout;
 	u16 msg_id;
 	int err;
 
+	/* Fail fast before taking the HWC slot. After a device reset, requests
+	 * already holding slots will release them once their own timeout
+	 * expires.
+	 */
+	if (READ_ONCE(hwc->hwc_dead))
+		return -ETIMEDOUT;
+
 	mana_hwc_get_msg_index(hwc, &msg_id);
 
 	tx_wr = &txq->msg_buf->reqs[msg_id];
@@ -902,21 +916,31 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 	dest_vrq = hwc->dest_vrq_id;
 	dest_vrcq = hwc->dest_vrcq_id;
 
+	/* If the HWC is unavailable after device reset, do not post the WQE:
+	 * the hardware cannot respond, so fail fast instead of waiting for
+	 * the full timeout.
+	 */
+	if (READ_ONCE(hwc->hwc_dead)) {
+		err = -ETIMEDOUT;
+		goto out;
+	}
+
 	err = mana_hwc_post_tx_wqe(txq, tx_wr, dest_vrq, dest_vrcq, false);
 	if (err) {
 		dev_err(hwc->dev, "HWC: Failed to post send WQE: %d\n", err);
 		goto out;
 	}
 
+	timeout = READ_ONCE(hwc->hwc_timeout);
 	if (!wait_for_completion_timeout(&ctx->comp_event,
-					 (msecs_to_jiffies(hwc->hwc_timeout)))) {
-		if (hwc->hwc_timeout != 0)
+					 (msecs_to_jiffies(timeout)))) {
+		if (timeout)
 			dev_err(hwc->dev, "Command 0x%x timed out: %u ms\n",
-				command, hwc->hwc_timeout);
+				command, timeout);
 
 		/* Reduce further waiting if HWC no response */
-		if (hwc->hwc_timeout > 1)
-			hwc->hwc_timeout = 1;
+		if (timeout > 1)
+			cmpxchg(&hwc->hwc_timeout, timeout, 1);
 
 		err = -ETIMEDOUT;
 		goto out;
drivers/net/ethernet/microsoft/mana/mana_en.c +89 -29

181 lines not on this page · raw message

include/net/mana/hw_channel.h +1 -0
index 1ab2d66c891e..0bc222c32244 100644
@@ -179,6 +179,7 @@ struct hw_channel_context {
 	struct gdma_dev *gdma_dev;
 	struct device *dev;
 
+	bool hwc_dead;
 	u16 num_inflight_msg;
 	u32 max_req_msg_size;
 
include/net/mana/mana.h +3 -0
index 218d540bc994..0ed638bfe8b2 100644
@@ -143,6 +143,7 @@ struct mana_txq {
 	atomic_t pending_sends;
 
 	bool napi_initialized;
+	bool napi_disabled;
 
 	struct mana_stats_tx stats;
 };
@@ -344,6 +345,8 @@ struct mana_cq {
 	/* Cumulative TX completions fed to DIM. Updated and read only in
 	 * NAPI context (mana_poll_tx_cq() / mana_update_tx_dim()), so they
 	 * measure the hardware completion rate and need no u64_stats_sync.
+	 * The TX drain in mana_dealloc_queues() also updates them, but only
+	 * after NAPI is disabled and the DIM work is cancelled.
 	 */
 	u64 tx_dim_pkts;
 	u64 tx_dim_bytes;

--
2.53.0

#1 lore raw
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines

After device reset is issued due to TX drain timeout in mana_dealloc_queues(), hardware channel (HWC) state is non-functional. However, the driver continues to issue HWC commands (mana_disable_vport_rx, mana_fence_rqs, mana_destroy_wq_obj) which will each wait for a completion that never arrives.

The first such command (MANA_CONFIG_VPORT_RX) blocks for the full hwc_timeout (60s as set by the firmware on the tested system), after which the timeout is reduced to 1 ms for subsequent commands. This causes the calling thread to hold the RTNL lock for over 180 seconds (120s TX drain + 60s HWC timeout), triggering hung task warnings and blocking all other networking operations on the system.

Quoted 8 lines

Fix this by:
1. Replacing the sleep-based TX drain with a CQE polling approach:

[ ... ]
Fixes a7dfeda6fdec ("net: mana: Fix MANA VF unload when hardware is unresponsive")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1