Skip to contentSkip to conversation

net: mana: Fix MANA VF unload when host is unresponsive

PATCH Superseded souradeep chakrabarti 7 participants 14 messages Jul 3, 2023 – Jul 10, 2023

Subsystems (6)

v4 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

---
V3 -> V4:
* Fixed the commit message to describe the context.
* Removed the vf_unload_timeout, as it is not required.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 26 ++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +23 -3
index a499e460594b..d26f1da70411 100644
@@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct net_device *ndev)
 {
 	struct mana_port_context *apc = netdev_priv(ndev);
 	struct gdma_dev *gd = apc->ac->gdma_dev;
+	unsigned long timeout;
 	struct mana_txq *txq;
+	struct sk_buff *skb;
+	struct mana_cq *cq;
 	int i, err;
 
 	if (apc->port_is_up)
@@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct net_device *ndev)
 	 * to false, but it doesn't matter since mana_start_xmit() drops any
 	 * new packets due to apc->port_is_up being false.
 	 *
-	 * Drain all the in-flight TX packets
+	 * Drain all the in-flight TX packets.
+	 * A timeout of 120 seconds for all the queues is used.
+	 * This will break the while loop when h/w is not responding.
+	 * This value of 120 has been decided here considering max
+	 * number of queues.
 	 */
+
+	timeout = jiffies + 120 * HZ;
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);
+		}
 	}
 
+	for (i = 0; i < apc->num_queues; i++) {
+		txq = &apc->tx_qp[i].txq;
+		cq = &apc->tx_qp[i].tx_cq;
+		while (atomic_read(&txq->pending_sends)) {
+			skb = skb_dequeue(&txq->pending_skbs);
+			mana_unmap_skb(skb, apc);
+			napi_consume_skb(skb, cq->budget);
+			atomic_sub(1, &txq->pending_sends);
+		}
+	}
 	/* We're 100% sure the queues can no longer be woken up, because
 	 * we're sure now mana_poll_tx_cq() can't be running.
 	 */
Signature · 1 line
2.34.1
#1 lore raw
FYI: kernel test robot notices the stable kernel rule is not satisfied.

Hi,

Thanks for your patch.

FYI: kernel test robot notices the stable kernel rule is not satisfied.

Rule: 'Cc: [email protected]' or 'commit <sha1> upstream.'
Subject: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#2 lore raw
In reply to souradeep chakrabarti
Quoted 77 lines

-----Original Message-----
From: souradeep chakrabarti <[email protected]>
Sent: Monday, July 3, 2023 4:50 AM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>; Souradeep Chakrabarti <[email protected]>
Subject: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

---
V3 -> V4:
* Fixed the commit message to describe the context.
* Removed the vf_unload_timeout, as it is not required.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 26 ++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
index a499e460594b..d26f1da70411 100644
@@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct net_device
 *ndev)
 {
 	struct mana_port_context *apc = netdev_priv(ndev);
 	struct gdma_dev *gd = apc->ac->gdma_dev;
+	unsigned long timeout;
 	struct mana_txq *txq;
+	struct sk_buff *skb;
+	struct mana_cq *cq;
 	int i, err;
 
drivers/net/ethernet/microsoft/mana/mana_en.c +9 -3
 	if (apc->port_is_up)
@@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct net_device
 *ndev)
 	 * to false, but it doesn't matter since mana_start_xmit() drops any
 	 * new packets due to apc->port_is_up being false.
 	 *
-	 * Drain all the in-flight TX packets
+	 * Drain all the in-flight TX packets.
+	 * A timeout of 120 seconds for all the queues is used.
+	 * This will break the while loop when h/w is not responding.
+	 * This value of 120 has been decided here considering max
+	 * number of queues.
 	 */
+
+	timeout = jiffies + 120 * HZ;
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device · line 2392

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +8 -0
+		}
 	}
 
+	for (i = 0; i < apc->num_queues; i++) {
+		txq = &apc->tx_qp[i].txq;
+		cq = &apc->tx_qp[i].tx_cq;
+		while (atomic_read(&txq->pending_sends)) {
+			skb = skb_dequeue(&txq->pending_skbs);
+			mana_unmap_skb(skb, apc);
+			napi_consume_skb(skb, cq->budget);

This is not in NAPI context, so it should be dev_consume_skb_any()

Thanks,
- Haiyang

#3 lore raw
In reply to souradeep chakrabarti

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 01:49:31 -0700

Quoted 1 line

From: Souradeep Chakrabarti <[email protected]>

Please sync your Git name and Git mail account settings, so that your own patches won't have "From:" when sending. From what I see, you need to correct first letters of name and surname to capital in the Git email settings block.

Quoted 11 lines

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

Where's "Fixes:" tagging the blamed commit?

Quoted 26 lines

---
V3 -> V4:
* Fixed the commit message to describe the context.
* Removed the vf_unload_timeout, as it is not required.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 26 ++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
index a499e460594b..d26f1da70411 100644
@@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct net_device *ndev)
 {
 	struct mana_port_context *apc = netdev_priv(ndev);
 	struct gdma_dev *gd = apc->ac->gdma_dev;
+	unsigned long timeout;
 	struct mana_txq *txq;
+	struct sk_buff *skb;
+	struct mana_cq *cq;
 	int i, err;
 
 	if (apc->port_is_up)
@@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct net_device *ndev)
 	 * to false, but it doesn't matter since mana_start_xmit() drops any
 	 * new packets due to apc->port_is_up being false.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2376

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +7 -1
 	 *
-	 * Drain all the in-flight TX packets
+	 * Drain all the in-flight TX packets.
+	 * A timeout of 120 seconds for all the queues is used.
+	 * This will break the while loop when h/w is not responding.
+	 * This value of 120 has been decided here considering max
+	 * number of queues.
 	 */
+
+	timeout = jiffies + 120 * HZ;

Why not initialize it right when declaring?

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2383

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);> +		}
 	}

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2387

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+	for (i = 0; i < apc->num_queues; i++) {
+		txq = &apc->tx_qp[i].txq;
+		cq = &apc->tx_qp[i].tx_cq;

cq can be just &txq->tx_cq.

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2391

Quoted 4 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +4 -0
+		while (atomic_read(&txq->pending_sends)) {
+			skb = skb_dequeue(&txq->pending_skbs);
+			mana_unmap_skb(skb, apc);
+			napi_consume_skb(skb, cq->budget);

(you already have comment about this one)

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2397

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+			atomic_sub(1, &txq->pending_sends);
+		}
+	}
 	/* We're 100% sure the queues can no longer be woken up, because
 	 * we're sure now mana_poll_tx_cq() can't be running.
 	 */

Thanks,
Olek

#4 lore raw
Quoted 21 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Monday, July 3, 2023 10:18 PM
To: souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Souradeep Chakrabarti <[email protected]>
Subject: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 01:49:31 -0700

Quoted 6 lines
Quoted 1 line

From: Souradeep Chakrabarti <[email protected]>

Please sync your Git name and Git mail account settings, so that your own patches won't have "From:" when sending. From what I see, you need to correct first letters of name and surname to capital in the Git email settings block.

Thank you for pointing, I will fix it.

Quoted 7 lines
Quoted 7 lines

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Quoted 8 lines
Quoted 6 lines

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

Signed-off-by: Souradeep Chakrabarti
<[email protected]>

Where's "Fixes:" tagging the blamed commit?

This is present from the day zero of the mana driver code. It has not been introduced in the code by any commit.

Quoted 28 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct

Quoted 27 lines

---
V3 -> V4:
* Fixed the commit message to describe the context.
* Removed the vf_unload_timeout, as it is not required.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 26
++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
index a499e460594b..d26f1da70411 100644
@@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct
 net_device *ndev)  {
 	struct mana_port_context *apc = netdev_priv(ndev);
 	struct gdma_dev *gd = apc->ac->gdma_dev;
+	unsigned long timeout;
 	struct mana_txq *txq;
+	struct sk_buff *skb;
+	struct mana_cq *cq;
 	int i, err;
 
 	if (apc->port_is_up)
@@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct

net_device *ndev)

Quoted 14 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2376

Quoted 12 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +7 -1
 	 * to false, but it doesn't matter since mana_start_xmit() drops any
 	 * new packets due to apc->port_is_up being false.
 	 *
-	 * Drain all the in-flight TX packets
+	 * Drain all the in-flight TX packets.
+	 * A timeout of 120 seconds for all the queues is used.
+	 * This will break the while loop when h/w is not responding.
+	 * This value of 120 has been decided here considering max
+	 * number of queues.
 	 */
+
+	timeout = jiffies + 120 * HZ;

Why not initialize it right when declaring?

I will fix it in next version.

Quoted 17 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2383

Quoted 9 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);> +		}
 	}

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time, before freeing sk_buffs. The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by xmit.

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions it :

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues are allowed in this specific hardware, it is a safe assumption.

Quoted 7 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2387

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+	for (i = 0; i < apc->num_queues; i++) {
+		txq = &apc->tx_qp[i].txq;
+		cq = &apc->tx_qp[i].tx_cq;

cq can be just &txq->tx_cq.

mana_txq structure does not have a pointer to mana_cq.

Quoted 17 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2391

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +4 -0
+		while (atomic_read(&txq->pending_sends)) {
+			skb = skb_dequeue(&txq->pending_skbs);
+			mana_unmap_skb(skb, apc);
+			napi_consume_skb(skb, cq->budget);

(you already have comment about this one)

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2397

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+			atomic_sub(1, &txq->pending_sends);
+		}
+	}
 	/* We're 100% sure the queues can no longer be woken up, because
 	 * we're sure now mana_poll_tx_cq() can't be running.
 	 */

Thanks,
Olek

maintainer
#5 lore raw
Quoted 37 lines · Souradeep Chakrabarti
Quoted 19 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Monday, July 3, 2023 10:18 PM
To: souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Souradeep Chakrabarti <[email protected]>
Subject: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 01:49:31 -0700

Quoted 6 lines
Quoted 1 line

From: Souradeep Chakrabarti <[email protected]>

Please sync your Git name and Git mail account settings, so that your own patches won't have "From:" when sending. From what I see, you need to correct first letters of name and surname to capital in the Git email settings block.

Thank you for pointing, I will fix it.

Quoted 11 lines
Quoted 11 lines

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Signed-off-by: Souradeep Chakrabarti
<[email protected]>

Where's "Fixes:" tagging the blamed commit?

This is present from the day zero of the mana driver code. It has not been introduced in the code by any commit.

Then the fixes tag should be:

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")

Cheers,

Paolo

maintainer
#6 lore raw
In reply to Souradeep Chakrabarti
Quoted 141 lines

-----Original Message-----
From: Souradeep Chakrabarti <[email protected]>
Sent: Monday, July 3, 2023 3:55 PM
To: Alexander Lobakin <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Monday, July 3, 2023 10:18 PM
To: souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang

Quoted 8 lines

<[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Souradeep Chakrabarti

<[email protected]>

Quoted 6 lines

Subject: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 01:49:31 -0700

Quoted 6 lines
Quoted 1 line

From: Souradeep Chakrabarti <[email protected]>

Please sync your Git name and Git mail account settings, so that your own patches won't have "From:" when sending. From what I see, you need to correct first letters of name and surname to capital in the Git email settings block.

Thank you for pointing, I will fix it.

Quoted 7 lines
Quoted 7 lines

When unloading the MANA driver, mana_dealloc_queues() waits for the MANA hardware to complete any inflight packets and set the pending send count to zero. But if the hardware has failed, mana_dealloc_queues() could wait forever.

Quoted 8 lines
Quoted 6 lines

Fix this by adding a timeout to the wait. Set the timeout to 120 seconds, which is a somewhat arbitrary value that is more than long enough for functional hardware to complete any sends.

Signed-off-by: Souradeep Chakrabarti
<[email protected]>

Where's "Fixes:" tagging the blamed commit?

This is present from the day zero of the mana driver code. It has not been introduced in the code by any commit.

Quoted 28 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct

Quoted 27 lines

---
V3 -> V4:
* Fixed the commit message to describe the context.
* Removed the vf_unload_timeout, as it is not required.

---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 26
++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
index a499e460594b..d26f1da70411 100644
@@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct
 net_device *ndev)  {
 	struct mana_port_context *apc = netdev_priv(ndev);
 	struct gdma_dev *gd = apc->ac->gdma_dev;
+	unsigned long timeout;
 	struct mana_txq *txq;
+	struct sk_buff *skb;
+	struct mana_cq *cq;
 	int i, err;
 
 	if (apc->port_is_up)
@@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct

net_device *ndev)

Quoted 14 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2376

Quoted 12 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +7 -1
 	 * to false, but it doesn't matter since mana_start_xmit() drops any
 	 * new packets due to apc->port_is_up being false.
 	 *
-	 * Drain all the in-flight TX packets
+	 * Drain all the in-flight TX packets.
+	 * A timeout of 120 seconds for all the queues is used.
+	 * This will break the while loop when h/w is not responding.
+	 * This value of 120 has been decided here considering max
+	 * number of queues.
 	 */
+
+	timeout = jiffies + 120 * HZ;

Why not initialize it right when declaring?

I will fix it in next version.

Quoted 16 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2383

Quoted 9 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);> +		}
 	}

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work

instead

Quoted 1 line

of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the
pending_sends of each tx queues for an indefinite time, before freeing sk_buffs.
The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by xmit.

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq()
not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions it :

2346 /* No packet can be transmitted now since apc->port_is_up is false.

Quoted 8 lines
2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues
are allowed in this specific hardware, it is a safe assumption.

I agree. Also, this waiting time is usually much shorter than 120 sec. The long wait only happens in rare and unexpected NIC HW non-responding cases. To further reduce the resource consumption, we can double the usleep_range() time in every iteration. So, the number of iterations will be greatly reduced before reaching 120 sec.

Thanks,
- Haiyang

#7 lore raw
In reply to Souradeep Chakrabarti

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 19:55:06 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Monday, July 3, 2023 10:18 PM

[...]

Quoted 27 lines
Quoted 9 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2383

Quoted 8 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
 	for (i = 0; i < apc->num_queues; i++) {
 		txq = &apc->tx_qp[i].txq;
-
-		while (atomic_read(&txq->pending_sends) > 0)
+		while (atomic_read(&txq->pending_sends) > 0 &&
+		       time_before(jiffies, timeout)) {
 			usleep_range(1000, 2000);> +		}
 	}
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time, before freeing sk_buffs. The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by xmit.

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions it :

Quoted 7 lines

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues

This is quite opposite to what you're saying above. How should I connect these two:

Here the intent is not waiting for 120 seconds

+

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds?
My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

Quoted 9 lines

are allowed in this specific hardware, it is a safe assumption.

Quoted 7 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2387

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+	for (i = 0; i < apc->num_queues; i++) {
+		txq = &apc->tx_qp[i].txq;
+		cq = &apc->tx_qp[i].tx_cq;

cq can be just &txq->tx_cq.

mana_txq structure does not have a pointer to mana_cq.

Sorry, misread, my bad.

Quoted 8 lines
Quoted 8 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2391

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +4 -0
+		while (atomic_read(&txq->pending_sends)) {
+			skb = skb_dequeue(&txq->pending_skbs);
+			mana_unmap_skb(skb, apc);
+			napi_consume_skb(skb, cq->budget);

(you already have comment about this one)

Quoted 9 lines
Quoted 9 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2397

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+			atomic_sub(1, &txq->pending_sends);
+		}
+	}
 	/* We're 100% sure the queues can no longer be woken up, because
 	 * we're sure now mana_poll_tx_cq() can't be running.
 	 */

Thanks,
Olek

Thanks,
Olek

#8 lore raw
Quoted 78 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Wednesday, July 5, 2023 8:06 PM
To: Souradeep Chakrabarti <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

From: Souradeep Chakrabarti <[email protected]>
Date: Mon, 3 Jul 2023 19:55:06 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Monday, July 3, 2023 10:18 PM

[...]

Quoted 16 lines
Quoted 9 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2383

Quoted 8 lines
    for (i = 0; i < apc->num_queues; i++) {
            txq = &apc->tx_qp[i].txq;
drivers/net/ethernet/microsoft/mana/mana_en.c +2 -2
-
-           while (atomic_read(&txq->pending_sends) > 0)
+           while (atomic_read(&txq->pending_sends) > 0 &&
+                  time_before(jiffies, timeout)) {
                    usleep_range(1000, 2000);> +            }
    }
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Quoted 2 lines

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time,

before freeing sk_buffs.

Quoted 2 lines

The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by

xmit.

Quoted 7 lines

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions it :

Quoted 8 lines

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues

This is quite opposite to what you're saying above. How should I connect these two:

Here the intent is not waiting for 120 seconds

+

Quoted 8 lines

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds?
My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

For each queue each takes few milli second, in a normal condition. So based on maximum number of allowed queues for our h/w it won't go beyond a sec.
The 120s only happens rarely during some NIC HW issue -unexpected. So this timeout will only trigger in a very rare scenario.

Quoted 31 lines
Quoted 9 lines

are allowed in this specific hardware, it is a safe assumption.

Quoted 7 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2387

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+   for (i = 0; i < apc->num_queues; i++) {
+           txq = &apc->tx_qp[i].txq;
+           cq = &apc->tx_qp[i].tx_cq;

cq can be just &txq->tx_cq.

mana_txq structure does not have a pointer to mana_cq.

Sorry, misread, my bad.

Quoted 15 lines
Quoted 15 lines

drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2391

Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +4 -0
+           while (atomic_read(&txq->pending_sends)) {
+                   skb = skb_dequeue(&txq->pending_skbs);
+                   mana_unmap_skb(skb, apc);
+                   napi_consume_skb(skb, cq->budget);

(you already have comment about this one)

Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+                   atomic_sub(1, &txq->pending_sends);
+           }
+   }
    /* We're 100% sure the queues can no longer be woken up, because
     * we're sure now mana_poll_tx_cq() can't be running.
     */
Quoted 2 lines
Quoted 2 lines

Thanks,
Olek

Thanks,
Olek

#9 lore raw

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 10:41:03 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Wednesday, July 5, 2023 8:06 PM

[...]

Quoted 44 lines
Quoted 36 lines
Quoted 9 lines
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time,

before freeing sk_buffs.

Quoted 2 lines

The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by

xmit.

Quoted 7 lines

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions it :

Quoted 8 lines

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues

This is quite opposite to what you're saying above. How should I connect these two:

Here the intent is not waiting for 120 seconds

+

Quoted 8 lines

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds?
My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

Quoted 5 lines

For each queue each takes few milli second, in a normal condition. So based on maximum number of allowed queues for our h/w it won't go beyond a sec.
The 120s only happens rarely during some NIC HW issue -unexpected. So this timeout will only trigger in a very rare scenario.

So set the timeout to 2 seconds if it makes no difference?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

are allowed in this specific hardware, it is a safe assumption.

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+   for (i = 0; i < apc->num_queues; i++) {
+           txq = &apc->tx_qp[i].txq;
+           cq = &apc->tx_qp[i].tx_cq;

[...]

Thanks,
Olek

#10 lore raw
Quoted 78 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Thursday, July 6, 2023 5:09 PM
To: Souradeep Chakrabarti <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 10:41:03 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Wednesday, July 5, 2023 8:06 PM

[...]

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time,

before freeing sk_buffs.

Quoted 3 lines

The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by

Quoted 7 lines
Quoted 7 lines

xmit.

Quoted 6 lines

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions

it :

Quoted 26 lines
Quoted 26 lines
Quoted 9 lines

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues

This is quite opposite to what you're saying above. How should I connect these
two:

Here the intent is not waiting for 120 seconds

+

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds? My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

Quoted 7 lines
Quoted 5 lines

For each queue each takes few milli second, in a normal condition. So based on maximum number of allowed queues for our h/w it won't go beyond a sec.
The 120s only happens rarely during some NIC HW issue -unexpected. So this timeout will only trigger in a very rare scenario.

So set the timeout to 2 seconds if it makes no difference?

It can go near 120 seconds in a very rare MANA h/w scenario. That normally won't happen. But during that scenario, we may need 120 seconds.

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

are allowed in this specific hardware, it is a safe assumption.

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3 -0
+   for (i = 0; i < apc->num_queues; i++) {
+           txq = &apc->tx_qp[i].txq;
+           cq = &apc->tx_qp[i].tx_cq;

[...]

Thanks,
Olek

#11 lore raw

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 11:43:58 +0000

Quoted 84 lines
Quoted 84 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Thursday, July 6, 2023 5:09 PM
To: Souradeep Chakrabarti <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 10:41:03 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Wednesday, July 5, 2023 8:06 PM

[...]

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time,

before freeing sk_buffs.

Quoted 3 lines

The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by

Quoted 7 lines
Quoted 7 lines

xmit.

Quoted 6 lines

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it mentions

it :

Quoted 31 lines
Quoted 18 lines
Quoted 9 lines

2346 /* No packet can be transmitted now since apc->port_is_up is false.

2347      * There is still a tiny chance that mana_poll_tx_cq() can re-enable
2348      * a txq because it may not timely see apc->port_is_up being cleared
2349      * to false, but it doesn't matter since mana_start_xmit() drops any
2350      * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum number of queues

This is quite opposite to what you're saying above. How should I connect these
two:

Here the intent is not waiting for 120 seconds

+

Quoted 8 lines

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds? My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

For each queue each takes few milli second, in a normal condition. So based on maximum number of allowed queues for our h/w it won't go beyond a sec.
The 120s only happens rarely during some NIC HW issue -unexpected. So this timeout will only trigger in a very rare scenario.

Quoted 3 lines
Quoted 1 line

So set the timeout to 2 seconds if it makes no difference?

It can go near 120 seconds in a very rare MANA h/w scenario. That normally won't happen. But during that scenario, we may need 120 seconds.

This waiting loop is needed to let the pending Tx packets be sent. If they weren't sent in 1 second, it most likely makes no sense already whether they will be sent at all or not -- the destination host won't wait for them for so long.
You say that it may happen only in case of HW issue. If so, I assume you need to fix it some way, e.g. do a HW reset or so? If so, why bother waiting for Tx completions if Tx is hung? You free all skbs later either way, so there are no leaks.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

are allowed in this specific hardware, it is a safe assumption.

Quoted 5 lines
Quoted 5 lines
(quoted diff) +3 -0
+   for (i = 0; i < apc->num_queues; i++) {
+           txq = &apc->tx_qp[i].txq;
+           cq = &apc->tx_qp[i].tx_cq;

[...]

Thanks,
Olek

Thanks,
Olek

maintainer
#12 lore raw
Quoted 119 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Thursday, July 6, 2023 7:49 AM
To: Souradeep Chakrabarti <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 11:43:58 +0000

Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Thursday, July 6, 2023 5:09 PM
To: Souradeep Chakrabarti <[email protected]>; souradeep chakrabarti <[email protected]>
Cc: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>;

Ajay

Quoted 2 lines
Quoted 2 lines

Sharma <[email protected]>; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 31 lines
Quoted 31 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXTERNAL] Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is unresponsive

From: Souradeep Chakrabarti <[email protected]>
Date: Thu, 6 Jul 2023 10:41:03 +0000

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Alexander Lobakin <[email protected]>
Sent: Wednesday, July 5, 2023 8:06 PM

[...]

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000 iterations. I know usleep_range() often is much less precise, but still. Do you really need that much time? Has this been measured during the tests that it can take up to 120 seconds or is it just some random value that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work instead of wasting resources.

Here the intent is not waiting for 120 seconds, rather than avoid continue checking the pending_sends of each tx queues for an indefinite time,

before freeing sk_buffs.

Quoted 3 lines

The pending_sends can only get decreased for a tx queue, if mana_poll_tx_cq() gets called for a completion notification and increased by

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xmit.

Quoted 6 lines

In this particular bug, apc->port_is_up is not set to false, causing xmit to keep increasing the pending_sends for the queue and mana_poll_tx_cq() not getting called for the queue.

If we see the comment in the function mana_dealloc_queues(), it

mentions

Quoted 3 lines
Quoted 3 lines

it :

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

2346 /* No packet can be transmitted now since apc->port_is_up is

false.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2347 * There is still a tiny chance that mana_poll_tx_cq() can re-

enable

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2348 * a txq because it may not timely see apc->port_is_up being

cleared

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2349 * to false, but it doesn't matter since mana_start_xmit() drops

any

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

2350 * new packets due to apc->port_is_up being false.

The value 120 seconds has been decided here based on maximum

number

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 1 line

of queues

This is quite opposite to what you're saying above. How should I connect these
two:

Here the intent is not waiting for 120 seconds

+

The value 120 seconds has been decided here based on maximum number of queues

?
Can cleaning the Tx queues really last for 120 seconds? My understanding is that timeouts need to be sensible and not go to the outer space. What is the medium value you got during the tests?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

For each queue each takes few milli second, in a normal condition. So based on maximum number of allowed queues for our h/w it won't go beyond a sec.
The 120s only happens rarely during some NIC HW issue -unexpected. So this timeout will only trigger in a very rare scenario.

So set the timeout to 2 seconds if it makes no difference?

It can go near 120 seconds in a very rare MANA h/w scenario. That normally

won't happen.

Quoted 1 line

But during that scenario, we may need 120 seconds.

Quoted 8 lines

This waiting loop is needed to let the pending Tx packets be sent. If they weren't sent in 1 second, it most likely makes no sense already whether they will be sent at all or not -- the destination host won't wait for them for so long.
You say that it may happen only in case of HW issue. If so, I assume you need to fix it some way, e.g. do a HW reset or so? If so, why bother waiting for Tx completions if Tx is hung? You free all skbs later either way, so there are no leaks.

At that point, we don't actually care if the pending packets are sent or not. But if we free the queues too soon, and the HW is slow for unexpected reasons, a delayed completion notice will DMA into the freed memory and cause corruption. That's why we have a longer waiting time.

Souradeep, you may double check with hostnet team to see what's the best waiting time to ensure no more HW activities.

Thanks,
- Haiyang

#13 lore raw
Quoted 9 lines · Haiyang Zhang
Quoted 8 lines

This waiting loop is needed to let the pending Tx packets be sent. If they weren't sent in 1 second, it most likely makes no sense already whether they will be sent at all or not -- the destination host won't wait for them for so long.
You say that it may happen only in case of HW issue. If so, I assume you need to fix it some way, e.g. do a HW reset or so? If so, why bother waiting for Tx completions if Tx is hung? You free all skbs later either way, so there are no leaks.

Quoted 4 lines

At that point, we don't actually care if the pending packets are sent or not. But if we free the queues too soon, and the HW is slow for unexpected reasons, a delayed completion notice will DMA into the freed memory and cause corruption. That's why we have a longer waiting time.

Aieiiie that is a horrible HW design to not have a strong fence of DMA.

"just wait and hope the HW doesn't UAF the kernel with DMA" is really awful.

Jason