Skip to contentSkip to conversation

net: mana: Add CQE polling as first-level TX timeout recovery

PATCH In review Sahil Chandna 4 participants 8 messages Nov 12 – Nov 18

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: mana: Add CQE polling as first-level TX timeout recovery

The current mana_tx_timeout() handler unconditionally schedules a full port queue reset (detach/attach) whenever the TX watchdog fires. This is unnecessary for transient stalls where completions are simply delayed or an interrupt was missed.

Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset.

If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue.

This makes the TX timeout recovery three-tiered:

Level 1: CQE poll via napi_schedule() (this patch) Level 2: Port queue reset (detach/attach) Level 3: device reset(FLR)

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 31 ++++++++++++++++---
 .../ethernet/microsoft/mana/mana_ethtool.c    |  2 ++
 include/net/mana/mana.h                       |  4 +++
 3 files changed, 33 insertions(+), 4 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +27 -4
index e6d494df22e4..0d8c8b8fbc83 100644
@@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work)
 	if (err)
 		netdev_err(ndev, "mana_attach failed: %d\n", err);
 
+	/* Let the next TX timeout try a CQE poll recovery first */
+	WRITE_ONCE(apc->last_tx_timeout, 0);
+
 dealloc_pre_rxbufs:
 	mana_pre_dealloc_rxbufs(apc);
 clear_flag:
@@ -964,6 +967,7 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue)
 	struct mana_port_context *apc = netdev_priv(netdev);
 	struct mana_context *ac = apc->ac;
 	struct gdma_context *gc = ac->gdma_dev->gdma_context;
+	unsigned long last = READ_ONCE(apc->last_tx_timeout);
 
 	/* Debug knob for bringup/qualification: when set, log the timeout and
 	 * skip the reset so the failing state is preserved for telemetry.
@@ -980,11 +984,30 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue)
 	if (test_bit(GC_IN_SERVICE, &gc->flags))
 		return;
 
-	/* Note: If there are pending queue reset work for this port(apc),
-	 * subsequent request queued up from here are ignored. This is because
-	 * we are using the same work instance per port(apc).
+	if (!apc->port_is_up)
+		return;
+
+	/* If the previous timeout fired less than
+	 * MANA_TX_TIMEOUT_RESET_THRESHOLD ago, the earlier CQE poll did not
+	 * help and queue stalled again	: escalate to a port reset.
+	 * Note: If there is pending queue reset work for this port (apc),
+	 * subsequent requests queued up from here are ignored, because the
+	 * same work instance is used per port (apc).
 	 */
-	queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
+	if (last &&
+	    time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) {
+		queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
+		return;
+	}
+
+	/* First timeout, or the previous one was long ago: try to recover by
+	 * polling the TX CQ for completions before escalating to a reset.
+	 */
+	if (txqueue < apc->num_queues && apc->tx_qp && apc->tx_qp[txqueue]) {
+		WRITE_ONCE(apc->last_tx_timeout, jiffies);
+		apc->eth_stats.tx_cqe_poll_recovery++;
+		napi_schedule(&apc->tx_qp[txqueue]->tx_cq.napi);
+	}
 }
 
 static int mana_shaper_set(struct net_shaper_binding *binding,
drivers/net/ethernet/microsoft/mana/mana_ethtool.c +2 -0
index a91eb12badca..ff4799c61a6c 100644
@@ -22,6 +22,8 @@ static const struct mana_stats_desc mana_eth_stats[] = {
 				       tx_linear_pkt_cnt)},
 	{"rx_cqe_unknown_type", offsetof(struct mana_ethtool_stats,
 					rx_cqe_unknown_type)},
+	{"tx_cqe_poll_recovery", offsetof(struct mana_ethtool_stats,
+					tx_cqe_poll_recovery)},
 };
 
 static const struct mana_stats_desc mana_hc_stats[] = {
include/net/mana/mana.h +4 -0
index 09725e3256fa..5de6eaa02102 100644
@@ -94,6 +94,8 @@ enum mana_priv_flag_bits {
 
 #define MANA_RX_FRAG_ALIGNMENT 64
 
+#define MANA_TX_TIMEOUT_RESET_THRESHOLD (2 * 60 * HZ)
+
 struct mana_stats_rx {
 	u64 packets;
 	u64 bytes;
@@ -447,6 +449,7 @@ struct mana_tx_qp {
 struct mana_ethtool_stats {
 	u64 stop_queue;
 	u64 wake_queue;
+	u64 tx_cqe_poll_recovery;
 	u64 tx_cqe_err;
 	u64 tx_cqe_unknown_type;
 	u64 tx_linear_pkt_cnt;
@@ -566,6 +569,7 @@ struct mana_port_context {
 	struct net_device *ndev;
 	struct work_struct queue_reset_work;
 
+	unsigned long last_tx_timeout;
 	/* Debug knob to log TX timeout but skip recovery reset */
 	bool tx_timeout_skip_reset;
 

--
2.53.0

#1 lore raw

Hello Sahil,

Quoted 6 lines · Sahil Chandna

Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset.

Which interrupt is being missed here?

Quoted 3 lines

If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue.

How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no?

#2 lore raw
Quoted 12 lines · Breno Leitao

Hello Sahil,

Quoted 6 lines · Sahil Chandna

Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset.

Which interrupt is being missed here?

Hi Breno,

It is the TX CQ completion interrupt. After NAPI finishes, it re-arms the TX CQ. When a new CQE arrives, the hardware post EQE, which raises the MSI-X and schedules NAPI. If that notification for the TX CQ is lost , NAPI never runs for the TX CQ and pending_sends never drains and the queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs and re-arms it.

Quoted 5 lines
Quoted 3 lines

If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue.

How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no?

The watchdog_timeo is TX stall timeout setting i.e if a queue remains stalled for this much time, it trigger watchdog timeout. Here, 2 minute is not timeout/wait. The 2 minutes here is for deciding whether a new queue stall event is a repeat of the previous incident or a new, unrelated one that should get the cheap poll first. For e.g. if a queue gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger manual NAPI, if another stall occurs (it can be on any queue) within 2 minutes of last one, the existing recovery mechanism of mana detach/attach will be tried.The patch is adding a first level less destructive recovery first.
For selecting 2 minutes, I do not have any hardware basis for it afaik, but I can confirm and check with our host team as well.
Hope this helps !
Regards,
Sahil

#3 lore raw
Quoted 13 lines · Sahil Chandna
Quoted 12 lines · Breno Leitao

Hello Sahil,

Quoted 6 lines · Sahil Chandna

Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset.

Which interrupt is being missed here?

Quoted 8 lines

Hi Breno,

It is the TX CQ completion interrupt. After NAPI finishes, it re-arms the TX CQ. When a new CQE arrives, the hardware post EQE, which raises the MSI-X and schedules NAPI. If that notification for the TX CQ is lost , NAPI never runs for the TX CQ and pending_sends never drains and the queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs and re-arms it.

Thanks. Which one gets lost, the EQE or the MSI-X?

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue.

How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no?

Quoted 12 lines

The watchdog_timeo is TX stall timeout setting i.e if a queue remains stalled for this much time, it trigger watchdog timeout. Here, 2 minute is not timeout/wait. The 2 minutes here is for deciding whether a new queue stall event is a repeat of the previous incident or a new, unrelated one that should get the cheap poll first. For e.g. if a queue gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger manual NAPI, if another stall occurs (it can be on any queue) within 2 minutes of last one, the existing recovery mechanism of mana detach/attach will be tried.The patch is adding a first level less destructive recovery first.
For selecting 2 minutes, I do not have any hardware basis for it afaik, but I can confirm and check with our host team as well.

So the goal is to tell an isolated stall from a wider problem.

A port-wide timestamp only approximates that. It cannot tell the same queue from another one, and since dev_watchdog() reports one queue per run, two queues that stall together show up ~15s apart: the first gets a kick, the second gets the port reset anyway, no?

Could mana_tx_timeout() check this directly? Look at the other TX queues the way dev_watchdog() does (stopped, with trans_start older than watchdog_timeo). If more than one is stalled, reset right away.

Otherwise kick this queue, and reset if the same queue times out again within a couple of watchdog periods, as gve_tx_timeout_try_q_kick() does. That also gets rid of the 2 minutes.

#4 lore raw
Quoted 16 lines · Breno Leitao
Quoted 15 lines · Sahil Chandna
Quoted 12 lines · Breno Leitao

Hello Sahil,

Quoted 6 lines · Sahil Chandna

Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset.

Which interrupt is being missed here?

Hi Breno,

Quoted 8 lines
Quoted 6 lines

It is the TX CQ completion interrupt. After NAPI finishes, it re-arms the TX CQ. When a new CQE arrives, the hardware post EQE, which raises the MSI-X and schedules NAPI. If that notification for the TX CQ is lost , NAPI never runs for the TX CQ and pending_sends never drains and the queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs and re-arms it.

Thanks. Which one gets lost, the EQE or the MSI-X?

Hi Breno,

Sorry for the delay, i was checking on these with my team. Regarding what is lost here. The patch is trying to solve the scenario where TX CQ never arrive due to some reason, it is not handling the MSI-X lost scenario and one of the netdev bot comment highlights the same. Though i will work on framing commit message more clearly in next version. Thanks !

Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue.

How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no?

Quoted 12 lines

The watchdog_timeo is TX stall timeout setting i.e if a queue remains stalled for this much time, it trigger watchdog timeout. Here, 2 minute is not timeout/wait. The 2 minutes here is for deciding whether a new queue stall event is a repeat of the previous incident or a new, unrelated one that should get the cheap poll first. For e.g. if a queue gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger manual NAPI, if another stall occurs (it can be on any queue) within 2 minutes of last one, the existing recovery mechanism of mana detach/attach will be tried.The patch is adding a first level less destructive recovery first.
For selecting 2 minutes, I do not have any hardware basis for it afaik, but I can confirm and check with our host team as well.

Quoted 2 lines

So the goal is to tell an isolated stall from a wider problem.

Yes, and take just a minimal step to solve it rather than entire port detach/attach.

Quoted 5 lines

A port-wide timestamp only approximates that. It cannot tell the same queue from another one, and since dev_watchdog() reports one queue per run, two queues that stall together show up ~15s apart: the first gets a kick, the second gets the port reset anyway, no?

yes, that is correct. The approach in this patch solves first instance of stall on any queue with CQE polling and if that does not solve it (based on another queue stall observed) go for existing mechanism (detach/attach), but I agree to your point here that another queue which stall after one queue had level 1 CQE polling would have a full detach/attach rather than level 1 polling. I will work on this in v2.

Quoted 7 lines

Could mana_tx_timeout() check this directly? Look at the other TX queues the way dev_watchdog() does (stopped, with trans_start older than watchdog_timeo). If more than one is stalled, reset right away.

Otherwise kick this queue, and reset if the same queue times out again within a couple of watchdog periods, as gve_tx_timeout_try_q_kick() does. That also gets rid of the 2 minutes.

Thanks, I will go through this approach and update in v2. I am currently discussing this internally.

Regards,
Sahil

#5 lore raw
In reply to Sahil Chandna
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [Critical] net: mana: missing netdev_lock() in…

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Critical] net: mana: missing netdev_lock() in mana_detach() callers
[1]
--

commit 6f4e15a96596e896188b6e028bb49a0d4c04269b
Author: Sahil Chandna <[email protected]>

net: mana: Add CQE polling as first-level TX timeout recovery

This commit adds a first-level TX timeout recovery mechanism that polls the stalled queue's TX CQ via napi_schedule() before escalating to a port-level queue reset.

drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_per_port_queue_reset_work_handler(struct work_struct *work) · line 340

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
index e6d494df22e4d..0d8c8b8fbc83f 100644
@@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work)
 	if (err)
 		netdev_err(ndev, "mana_attach failed: %d\n", err);
 
+	/* Let the next TX timeout try a CQE poll recovery first */
+	WRITE_ONCE(apc->last_tx_timeout, 0);

[Severity: Critical]
This problem wasn't introduced by this patch, but should mana_per_port_queue_reset_work_handler() (along with mana_remove(), mana_probe() on resume, and mana_xdp_set()) hold netdev_lock(ndev) around mana_detach() and mana_attach() [1]?

Because mana_devops sets .net_shaper_ops, netdev_need_ops_lock() is enabled on MANA net_device instances. Queue teardown in mana_detach() calls napi_disable_locked() and netif_napi_del_locked() via mana_destroy_txq() and mana_destroy_rxq(), while mana_attach() calls netif_napi_add_locked(), netif_napi_add_weight_locked(), and napi_enable_locked(). Calling these under rtnl_lock() alone violates netdev_assert_locked(), leaves NAPI state unserialized against netdev-genl operations, and allows ops-locked ethtool callbacks (such as mana_get_ethtool_stats() or mana_set_coalesce()) running under netdev_lock() without rtnl_lock() to race with apc->rxqs and apc->tx_qp teardown, leading to a use-after-free or NULL pointer dereference.

[1] https://sashiko.dev/#/bug/linux-2946b7e7-6855-49c0-921d-a6daa482c10c

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#6 lore raw
Quoted 45 lines · On 13-11-2026 16:28, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Critical] net: mana: missing netdev_lock() in mana_detach() callers
[1]
--

commit 6f4e15a96596e896188b6e028bb49a0d4c04269b
Author: Sahil Chandna <[email protected]>

net: mana: Add CQE polling as first-level TX timeout recovery

This commit adds a first-level TX timeout recovery mechanism that polls the stalled queue's TX CQ via napi_schedule() before escalating to a port-level queue reset.

drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_per_port_queue_reset_work_handler(struct work_struct *work) · line 340

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -0
index e6d494df22e4d..0d8c8b8fbc83f 100644
@@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work)
 	if (err)
 		netdev_err(ndev, "mana_attach failed: %d\n", err);
 
+	/* Let the next TX timeout try a CQE poll recovery first */
+	WRITE_ONCE(apc->last_tx_timeout, 0);

[Severity: Critical]
This problem wasn't introduced by this patch, but should mana_per_port_queue_reset_work_handler() (along with mana_remove(), mana_probe() on resume, and mana_xdp_set()) hold netdev_lock(ndev) around mana_detach() and mana_attach() [1]?

Because mana_devops sets .net_shaper_ops, netdev_need_ops_lock() is enabled on MANA net_device instances. Queue teardown in mana_detach() calls napi_disable_locked() and netif_napi_del_locked() via mana_destroy_txq() and mana_destroy_rxq(), while mana_attach() calls netif_napi_add_locked(), netif_napi_add_weight_locked(), and napi_enable_locked(). Calling these under rtnl_lock() alone violates netdev_assert_locked(), leaves NAPI state unserialized against netdev-genl operations, and allows ops-locked ethtool callbacks (such as mana_get_ethtool_stats() or mana_set_coalesce()) running under netdev_lock() without rtnl_lock() to race with apc->rxqs and apc->tx_qp teardown, leading to a use-after-free or NULL pointer dereference.

Quoted 2 lines

[1] https://sashiko.dev/#/bug/linux-2946b7e7-6855-49c0-921d-a6daa482c10c

Thanks or reporting this, the issue looks real and I will work on this in a separate patch.

#7 lore raw
In reply to Sahil Chandna
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider.

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 3

If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).

- [Medium] Level 1 of the new recovery only calls napi_schedule() on the TX CQ that stalled, in mana_tx_timeout()…
- [Low] The commit message says the patch makes TX timeout recovery 'three-tiered' and lists 'Level 3: device reset(FLR)' as the step…
- [Low] apc->last_tx_timeout is cleared in only one place: the post- attach tail of mana_per_port_queue_reset_work_handler()…
- [Low] The new comment in mana_tx_timeout() (mana_en.c:990-996) says escalation happens because 'the earlier CQE poll did not help and…

--

net: mana: Add CQE polling as first-level TX timeout recovery

This commit changes mana_tx_timeout() so that the first TX timeout, or one arriving more than 2 minutes after the previous one, schedules NAPI on the stalled queue's TX CQ instead of resetting the port. A further timeout on any queue within 2 minutes escalates to the existing per-port detach/attach reset, and a new tx_cqe_poll_recovery ethtool counter tracks the CQE poll attempts.

Quoted 5 lines

This makes the TX timeout recovery three-tiered:

Level 1: CQE poll via napi_schedule() (this patch) Level 2: Port queue reset (detach/attach) Level 3: device reset(FLR)

[Severity: Low]
Is the Level 3 step accurate for the TX timeout path? When mana_pre_alloc_rxbufs(), mana_detach() or mana_attach() fails in mana_per_port_queue_reset_work_handler(), the handler only logs:

	err = mana_attach(ndev);
	if (err)
		netdev_err(ndev, "mana_attach failed: %d\n", err);

It doesn't call mana_schedule_serv_work() or set GC_IN_SERVICE. In mana_en.c, mana_schedule_serv_work() is only called from mana_gf_stats_work_handler() on an HWC timeout. gdma_main.c calls it for hardware service EQEs.

So when a port reset fails or doesn't help, the next timeout triggers another port reset, or after 2 minutes goes back to Level 1. It never reaches an FLR.

Could the commit message describe the FLR as a separate existing mechanism, rather than as the third tier of this escalation?

drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_per_port_queue_reset_work_handler(struct work_struct *work) · line 341

Quoted 11 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
index e6d494df22e4d..0d8c8b8fbc83f 100644
@@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work)
 	if (err)
 		netdev_err(ndev, "mana_attach failed: %d\n", err);
 
+	/* Let the next TX timeout try a CQE poll recovery first */
+	WRITE_ONCE(apc->last_tx_timeout, 0);
+

[Severity: Low]
Is this the only place last_tx_timeout gets cleared? The other paths that tear down and rebuild the TX queues don't touch it. These include mana_close()/mana_open(), the other mana_detach()/mana_attach() callers (channel, MTU and XDP changes), and the service/FLR resume through mana_probe().

Suppose a Level 1 CQE poll ran at T0 and one of those paths then rebuilt the queues. A TX timeout on the new queues before
T0 + MANA_TX_TIMEOUT_RESET_THRESHOLD still sees a recent, non-zero last in mana_tx_timeout():

	if (last &&
	    time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) {
		queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
		return;
	}

It would then go straight to a port reset, without a CQE poll on queues that were never polled.

Should the timestamp also be cleared when queues are rebuilt outside this worker, for example in mana_open() or mana_attach()?

Quoted 3 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +0 -0
 dealloc_pre_rxbufs:
 	mana_pre_dealloc_rxbufs(apc);
 clear_flag:

[ ... ]

drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue) · line 992

Quoted 13 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +6 -3
@@ -980,11 +984,30 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue)
 	if (test_bit(GC_IN_SERVICE, &gc->flags))
 		return;
 
-	/* Note: If there are pending queue reset work for this port(apc),
-	 * subsequent request queued up from here are ignored. This is because
-	 * we are using the same work instance per port(apc).
+	if (!apc->port_is_up)
+		return;
+
+	/* If the previous timeout fired less than
+	 * MANA_TX_TIMEOUT_RESET_THRESHOLD ago, the earlier CQE poll did not
+	 * help and queue stalled again	: escalate to a port reset.

[Severity: Low]
This isn't a bug, but there's a stray tab between "again" and the colon.

The comment also suggests the same queue stalled again. However, last_tx_timeout is a single per-port field in struct mana_port_context, so a timeout on any queue within the threshold escalates. That includes a queue that never had a CQE poll.

For example, queue A might be polled, and queue B might time out about 15 seconds later (watchdog_timeo). B then goes straight to a port reset.

Could the comment say that a timeout on any queue of the port escalates?

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +6 -1
+	 * Note: If there is pending queue reset work for this port (apc),
+	 * subsequent requests queued up from here are ignored, because the
+	 * same work instance is used per port (apc).
 	 */
-	queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
+	if (last &&
+	    time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) {
+		queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);

drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue) · line 1009

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +10 -0
+		return;
+	}
+
+	/* First timeout, or the previous one was long ago: try to recover by
+	 * polling the TX CQ for completions before escalating to a reset.
+	 */
+	if (txqueue < apc->num_queues && apc->tx_qp && apc->tx_qp[txqueue]) {
+		WRITE_ONCE(apc->last_tx_timeout, jiffies);
+		apc->eth_stats.tx_cqe_poll_recovery++;
+		napi_schedule(&apc->tx_qp[txqueue]->tx_cq.napi);

[Severity: Medium]
Does this recover the case where the MSI-X interrupt for the queue's EQ was lost? This only schedules NAPI on the TX CQ, and the NAPI path re-arms only the CQ in mana_cq_handler():

	if (w < cq->budget) {
		mana_gd_ring_cq(gdma_queue, SET_ARM_BIT);

The only EQ re-arm is at the end of mana_gd_process_eq_events(), which runs from the EQ interrupt:

	mana_gd_ring_doorbell(gc, eq->gdma_dev->doorbell, eq->type, eq->id,
			      head, SET_ARM_BIT, 0);

The TX CQ and the RX CQ of queue i share the same parent EQ:

mana_create_txq():

		spec.cq.parent_eq = apc->eqs[i].eq;

mana_create_rxq() (called with &apc->eqs[i]):

	spec.cq.parent_eq = eq->eq;

Suppose the EQ interrupt was lost and the EQ stays un-armed. TX would make progress briefly. New TX CQEs would then post EQEs into an un-armed EQ, no interrupt would fire, and TX would stall again. This path never polls the RX CQ on the same EQ, so RX on queue i would stay stalled.

Before this patch, the first watchdog firing did a detach/attach, which rebuilt and re-armed the EQ. Now escalation only happens if the next TX timeout arrives within 2 minutes.

With light TX load on that queue, timeouts could be more than 2 minutes apart. Each one would then be handled as another Level 1 poll, and RX on queue i might never recover.

The driver code doesn't show the exact hardware semantics of the GDMA EQ arm bit, so this depends on whether an interrupt consumes the arm state.

Does the commit message claim of "recovering missed-interrupt scenarios without queue reset" apply only to CQ-level misses? Or should Level 1 also handle the EQ and the RX CQ that shares it?

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +1 -0
+	}
 }
 
 static int mana_shaper_set(struct net_shaper_binding *binding,

[ ... ]

Signature · 1 line
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261112105752.978446-1-sahilchandna%40linux.microsoft.com