net: mana: Fix MANA VF unload when host is unresponsive
Subsystems (6)
v3 cover net: mana: Fix MANA VF unload when host is unresponsive
From: Souradeep Chakrabarti <[email protected]>
VF unload gets stuck in MANA driver, when the host is not responding. The function mana_dealloc_queues() tries to clear the inflight packets, and gets stuck in while loop. Another problem in this scenario is the timeout from hwc send request.
These patch add fix for the same.
In mana driver we are adding a timeout in the while loop, to fix it. Also we are adding a new attribute in mana_context, which gets set when mana_hwc_send_request() hits a timeout because of host unresponsiveness.
Souradeep Chakrabarti (2):
net: mana: Fix MANA VF unload when host is unresponsive
net: mana: Fix MANA VF unload when host is unresponsive
.../net/ethernet/microsoft/mana/gdma_main.c | 4 +++- .../net/ethernet/microsoft/mana/hw_channel.c | 12 +++++++++++- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- include/net/mana/mana.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/2 net: mana: Fix MANA VF unload when host is unresponsive
From: Souradeep Chakrabarti <[email protected]>
This patch addresses the VF unload issue, where mana_dealloc_queues() gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.
Microsoft Azure Network Adapter)
---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces from mana_dealloc_queues().
--- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +17 -2
@@ -2329,7 +2329,10 @@ static int mana_dealloc_queues(struct net_device *ndev) {struct mana_port_context *apc =netdev_priv (ndev);struct gdma_dev *gd = apc->ac->gdma_dev; +unsigned long timeout;struct mana_txq *txq; +struct sk_buff *skb; +struct mana_cq *cq;int i, err;if (apc->port_is_up)
@@ -2348,13 +2351,25 @@ static int mana_dealloc_queues(struct net_device *ndev)* * Drain all the in-flight TX packets */ + + timeout = jiffies +120 *HZ ;for (i =0 ; i < apc->num_queues; i++) { txq = &apc->tx_qp[i].txq; - -while (atomic_read (&txq->pending_sends) >0 ) +while (atomic_read (&txq->pending_sends) >0 && +time_before (jiffies, timeout))usleep_range (1000 ,2000 ); } +for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + cq = &apc->tx_qp[i].tx_cq; +while (atomic_read (&txq->pending_sends)) { + skb =skb_dequeue (&txq->pending_skbs); +mana_unmap_skb (skb, apc); +napi_consume_skb (skb, cq->budget); +atomic_sub (1 , &txq->pending_sends); + } + }/* We're 100% sure the queues can no longer be woken up, because * we're sure now mana_poll_tx_cq() can't be running. */
Signature · 1 line
2.34.1
PATCH v3 1/2 net: mana: Fix MANA VF unload when host is unresponsive
From: Souradeep Chakrabarti <[email protected]>
This patch addresses the VF unload issue, where mana_dealloc_queues() gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.
Microsoft Azure Network Adapter)
---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces from mana_dealloc_queues().
--- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
drivers/net/ethernet/microsoft/mana/mana_en.c +17 -2
@@ -2329,7 +2329,10 @@ static int mana_dealloc_queues(struct net_device *ndev) {struct mana_port_context *apc =netdev_priv (ndev);struct gdma_dev *gd = apc->ac->gdma_dev; +unsigned long timeout;struct mana_txq *txq; +struct sk_buff *skb; +struct mana_cq *cq;int i, err;if (apc->port_is_up)
@@ -2348,13 +2351,25 @@ static int mana_dealloc_queues(struct net_device *ndev)* * Drain all the in-flight TX packets */ + + timeout = jiffies +120 *HZ ;for (i =0 ; i < apc->num_queues; i++) { txq = &apc->tx_qp[i].txq; - -while (atomic_read (&txq->pending_sends) >0 ) +while (atomic_read (&txq->pending_sends) >0 && +time_before (jiffies, timeout))usleep_range (1000 ,2000 ); } +for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + cq = &apc->tx_qp[i].tx_cq; +while (atomic_read (&txq->pending_sends)) { + skb =skb_dequeue (&txq->pending_skbs); +mana_unmap_skb (skb, apc); +napi_consume_skb (skb, cq->budget); +atomic_sub (1 , &txq->pending_sends); + } + }/* We're 100% sure the queues can no longer be woken up, because * we're sure now mana_poll_tx_cq() can't be running. */
Signature · 1 line
2.34.1
kernel test robot
build botFYI: kernel test robot notices the stable kernel rule is not satisfied.
Hi,
Thanks for your patch.
FYI: kernel test robot notices the stable kernel rule is not satisfied.
Rule: 'Cc: [email protected]' or 'commit <sha1> upstream.'
Subject: [PATCH 1/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsive
The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 47 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This patch addresses the VF unload issue, where
mana_dealloc_queues()gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces frommana_dealloc_queues().--- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)drivers/net/ethernet/microsoft/mana/mana_en.c +7
-2@@ -2329,7 +2329,10 @@ static int mana_dealloc_queues(struct net_device *ndev) {struct mana_port_context *apc =netdev_priv (ndev);struct gdma_dev *gd = apc->ac->gdma_dev; +unsigned long timeout;struct mana_txq *txq; +struct sk_buff *skb; +struct mana_cq *cq;int i, err;if (apc->port_is_up)@@ -2348,13 +2351,25 @@ static int mana_dealloc_queues(struct net_device *ndev)* * Drain all the in-flight TX packets */ + + timeout = jiffies +120 *HZ ;for (i =0 ; i < apc->num_queues; i++) { txq = &apc->tx_qp[i].txq; - -while (atomic_read (&txq->pending_sends) >0 ) +while (atomic_read (&txq->pending_sends) >0 && +time_before (jiffies, timeout))usleep_range (1000 ,2000 ); }
drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2372
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +10
-0+for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + cq = &apc->tx_qp[i].tx_cq; +while (atomic_read (&txq->pending_sends)) { + skb =skb_dequeue (&txq->pending_skbs); +mana_unmap_skb (skb, apc); +napi_consume_skb (skb, cq->budget); +atomic_sub (1 , &txq->pending_sends); + } + }
Can we combine these 2 loops into 1 something like this ?
for (i = 0; i < apc->num_queues; i++) {
txq = &apc->tx_qp[i].txq;
cq = &apc->tx_qp[i].tx_cq;
while (atomic_read(&txq->pending_sends)) {
if (time_before(jiffies, timeout)) {
usleep_range(1000, 2000);
} else {
skb = skb_dequeue(&txq->pending_skbs);
mana_unmap_skb(skb, apc);
napi_consume_skb(skb, cq->budget);
atomic_sub(1, &txq->pending_sends);
}
}
}Quoted 3 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +0
-0/* We're 100% sure the queues can no longer be woken up, because * we're sure now mana_poll_tx_cq() can't be running. */
Quoted 61 lines · Praveen Kumar
Quoted 47 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This patch addresses the VF unload issue, where
mana_dealloc_queues()gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces frommana_dealloc_queues().--- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)drivers/net/ethernet/microsoft/mana/mana_en.c +7
-2@@ -2329,7 +2329,10 @@ static int mana_dealloc_queues(struct net_device *ndev) {struct mana_port_context *apc =netdev_priv (ndev);struct gdma_dev *gd = apc->ac->gdma_dev; +unsigned long timeout;struct mana_txq *txq; +struct sk_buff *skb; +struct mana_cq *cq;int i, err;if (apc->port_is_up)@@ -2348,13 +2351,25 @@ static int mana_dealloc_queues(struct net_device *ndev)* * Drain all the in-flight TX packets */ + + timeout = jiffies +120 *HZ ;for (i =0 ; i < apc->num_queues; i++) { txq = &apc->tx_qp[i].txq; - -while (atomic_read (&txq->pending_sends) >0 ) +while (atomic_read (&txq->pending_sends) >0 && +time_before (jiffies, timeout))usleep_range (1000 ,2000 ); }drivers/net/ethernet/microsoft/mana/mana_en.c static int mana_dealloc_queues(struct net_device *ndev) · line 2372
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +10
-0+for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + cq = &apc->tx_qp[i].tx_cq; +while (atomic_read (&txq->pending_sends)) { + skb =skb_dequeue (&txq->pending_skbs); +mana_unmap_skb (skb, apc); +napi_consume_skb (skb, cq->budget); +atomic_sub (1 , &txq->pending_sends); + } + }Can we combine these 2 loops into 1 something like this ?
Quoted 14 lines
for (i = 0; i < apc->num_queues; i++) {
txq = &apc->tx_qp[i].txq;
cq = &apc->tx_qp[i].tx_cq;
while (atomic_read(&txq->pending_sends)) {
if (time_before(jiffies, timeout)) {
usleep_range(1000, 2000);
} else {
skb = skb_dequeue(&txq->pending_skbs);
mana_unmap_skb(skb, apc);
napi_consume_skb(skb, cq->budget);
atomic_sub(1, &txq->pending_sends);
}
}
}We should free up the skbs only after timeout has happened or after all the queues are looped.
Quoted 3 lines
Quoted 3 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +0
-0/* We're 100% sure the queues can no longer be woken up, because * we're sure now mana_poll_tx_cq() can't be running. */
From: souradeep chakrabarti <[email protected]> Sent: Monday, June 26, 2023 2:19 AM
Quoted 4 lines
This patch addresses the VF unload issue, where
mana_dealloc_queues()gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.
For a patch series, the cover letter (patch 0 of the series) does not get
included in the commit log anywhere. The cover letter can provide
overall motivation and describe how the patches fit together, but the commit message for each patch should be as self-contained as possible.
The commit message here refers to "the VF unload issue", and there's no context for understanding what that issue is, though you do provide some description in the text following "where". Could you provide a commit message that is a bit more self-contained?
Same comment applies to commit message for the 2nd patch of this series.
Also, avoid text like "this patch". See the "Describe your changes"
section in Documentation/process/submitting-patches.rst where the use of imperative mood is mentioned. If you like, I can provide some offline help on writing a good commit message.
Michael
Quoted 58 lines
Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces frommana_dealloc_queues().--- drivers/net/ethernet/microsoft/mana/mana_en.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)drivers/net/ethernet/microsoft/mana/mana_en.c +17
-3@@ -2329,7 +2329,10 @@ static int mana_dealloc_queues(struct net_device *ndev) {struct mana_port_context *apc =netdev_priv (ndev);struct gdma_dev *gd = apc->ac->gdma_dev; +unsigned long timeout;struct mana_txq *txq; +struct sk_buff *skb; +struct mana_cq *cq;int i, err;if (apc->port_is_up)@@ -2348,13 +2351,25 @@ static int mana_dealloc_queues(struct net_device *ndev)* * Drain all the in-flight TX packets */ + + timeout = jiffies +120 *HZ ;for (i =0 ; i < apc->num_queues; i++) { txq = &apc->tx_qp[i].txq; - -while (atomic_read (&txq->pending_sends) >0 ) +while (atomic_read (&txq->pending_sends) >0 && +time_before (jiffies, timeout))usleep_range (1000 ,2000 ); } +for (i =0 ; i < apc->num_queues; i++) { + txq = &apc->tx_qp[i].txq; + cq = &apc->tx_qp[i].tx_cq; +while (atomic_read (&txq->pending_sends)) { + skb =skb_dequeue (&txq->pending_skbs); +mana_unmap_skb (skb, apc); +napi_consume_skb (skb, cq->budget); +atomic_sub (1 , &txq->pending_sends); + } + }/* We're 100% sure the queues can no longer be woken up, because * we're sure now mana_poll_tx_cq() can't be running. */ --2.34.1
kernel test robot
build botFYI: kernel test robot notices the stable kernel rule is not satisfied.
Hi,
Thanks for your patch.
FYI: kernel test robot notices the stable kernel rule is not satisfied.
Rule: 'Cc: [email protected]' or 'commit <sha1> upstream.'
Subject: [PATCH 1/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsive
The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v3 2/2 net: mana: Fix MANA VF unload when host is unresponsive
In reply to souradeep chakrabartiFrom: Souradeep Chakrabarti <[email protected]>
This is the second part of the fix.
Also this patch adds a new attribute in mana_context, which gets set when mana_hwc_send_request() hits a timeout because of host unresponsiveness. This flag then helps to avoid the timeouts in successive calls.
Microsoft Azure Network Adapter)
---
V2 -> V3:
* Removed the initialization of vf_unload_timeout
* Splitted the patch in two.
* Fixed extra space from the commit message.
--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 +++- drivers/net/ethernet/microsoft/mana/hw_channel.c | 12 +++++++++++- include/net/mana/mana.h | 2 ++ 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +3 -1
@@ -946,10 +946,12 @@ int mana_gd_deregister_device(struct gdma_dev *gd)struct gdma_context *gc = gd->gdma_context;struct gdma_general_resp resp = {};struct gdma_general_req req = {}; +struct mana_context *ac;int err;if (gd->pdid ==INVALID_PDID )return -EINVAL ; + ac = gd->driver_data;mana_gd_init_req_hdr (&req.hdr,GDMA_DEREGISTER_DEVICE ,sizeof (req),sizeof (resp));
@@ -957,7 +959,7 @@ int mana_gd_deregister_device(struct gdma_dev *gd) req.hdr.dev_id = gd->dev_id; err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); -if (err || resp.hdr.status) { +if ((err || resp.hdr.status) && !ac->vf_unload_timeout) {dev_err (gc->dev,"Failed to deregister device: %d, 0x%x\n" , err, resp.hdr.status);if (!err)
drivers/net/ethernet/microsoft/mana/hw_channel.c +11 -1
@@ -1,8 +1,10 @@// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* Copyright (c) 2021, Microsoft Corporation. */ +#include "asm-generic/errno.h" #include <net/mana/gdma.h> #include <net/mana/hw_channel.h> +#include <net/mana/mana.h> static int mana_hwc_get_msg_index (struct hw_channel_context *hwc,u16 *msg_id) {
@@ -786,12 +788,19 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,struct hwc_wq *txq = hwc->txq;struct gdma_req_hdr *req_msg;struct hwc_caller_ctx *ctx; +struct mana_context *ac;u32 dest_vrcq =0 ;u32 dest_vrq =0 ;u16 msg_id;int err;mana_hwc_get_msg_index (hwc, &msg_id); + ac = hwc->gdma_dev->driver_data; +if (ac->vf_unload_timeout) { +dev_err (hwc->dev,"HWC: vport is already unloaded.\n" ); + err = -ETIMEDOUT ; + goto out; + } tx_wr = &txq->msg_buf->reqs[msg_id];
@@ -825,9 +834,10 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, goto out; } -if (!wait_for_completion_timeout (&ctx->comp_event,30 *HZ )) { +if (!wait_for_completion_timeout (&ctx->comp_event,5 *HZ )) {dev_err (hwc->dev,"HWC: Request timed out!\n" ); err = -ETIMEDOUT ; + ac->vf_unload_timeout = true; goto out; }
include/net/mana/mana.h +2 -0
@@ -358,6 +358,8 @@ struct mana_context {u16 num_ports; +bool vf_unload_timeout; +struct mana_eq *eqs;struct net_device *ports[MAX_PORTS_IN_MANA_DEV ];
Signature · 1 line
2.34.1
Quoted 40 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This is the second part of the fix.
Also this patch adds a new attribute in
mana_context, which gets set whenmana_hwc_send_request()hits a timeout because of host unresponsiveness. This flag then helps to avoid the timeouts in successive calls.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Removed the initialization ofvf_unload_timeout
* Splitted the patch in two.
* Fixed extra space from the commit message.--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 +++- drivers/net/ethernet/microsoft/mana/hw_channel.c | 12 +++++++++++- include/net/mana/mana.h | 2 ++ 3 files changed, 16 insertions(+), 2 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-0@@ -946,10 +946,12 @@ int mana_gd_deregister_device(struct gdma_dev *gd)struct gdma_context *gc = gd->gdma_context;struct gdma_general_resp resp = {};struct gdma_general_req req = {}; +struct mana_context *ac;int err;if (gd->pdid ==INVALID_PDID )return -EINVAL ; + ac = gd->driver_data;mana_gd_init_req_hdr (&req.hdr,GDMA_DEREGISTER_DEVICE ,sizeof (req),sizeof (resp));
drivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_deregister_device(struct gdma_dev *gd) · line 964
Quoted 7 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +1
-1@@ -957,7 +959,7 @@ int mana_gd_deregister_device(struct gdma_dev *gd) req.hdr.dev_id = gd->dev_id; err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); -if (err || resp.hdr.status) { +if ((err || resp.hdr.status) && !ac->vf_unload_timeout) {dev_err (gc->dev,"Failed to deregister device: %d, 0x%x\n" , err, resp.hdr.status);
With !ac->vf_unload_timeout option, this message may not be correctly showing err, status. Probably you want to add explicit information during timeouts so that it give right information ? Or have the err, status field properly updated.
Quoted 18 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0if (!err)drivers/net/ethernet/microsoft/mana/hw_channel.c +2
-0@@ -1,8 +1,10 @@// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* Copyright (c) 2021, Microsoft Corporation. */ +#include "asm-generic/errno.h" #include <net/mana/gdma.h> #include <net/mana/hw_channel.h> +#include <net/mana/mana.h> static int mana_hwc_get_msg_index (struct hw_channel_context *hwc,u16 *msg_id) {@@ -786,12 +788,19 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,struct hwc_wq *txq = hwc->txq;
drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, · line 798
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +2
-0struct gdma_req_hdr *req_msg;struct hwc_caller_ctx *ctx; +struct mana_context *ac;u32 dest_vrcq =0 ;u32 dest_vrq =0 ;u16 msg_id;int err;mana_hwc_get_msg_index (hwc, &msg_id); + ac = hwc->gdma_dev->driver_data;
Is there a case where gdma_dev be invalid here ? If so, lets check the state and then proceed further ?
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +5
-0+if (ac->vf_unload_timeout) { +dev_err (hwc->dev,"HWC: vport is already unloaded.\n" ); + err = -ETIMEDOUT ; + goto out; + } tx_wr = &txq->msg_buf->reqs[msg_id];
drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, · line 837
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-1@@ -825,9 +834,10 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, goto out; } -if (!wait_for_completion_timeout (&ctx->comp_event,30 *HZ )) { +if (!wait_for_completion_timeout (&ctx->comp_event,5 *HZ )) {
IMHO we should have macros instead of magic numbers (5 , 30 or so). But would like others to comment here.
Quoted 19 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-0dev_err (hwc->dev,"HWC: Request timed out!\n" ); err = -ETIMEDOUT ; + ac->vf_unload_timeout = true; goto out; }include/net/mana/mana.h +2
-0@@ -358,6 +358,8 @@ struct mana_context {u16 num_ports; +bool vf_unload_timeout; +struct mana_eq *eqs;struct net_device *ports[MAX_PORTS_IN_MANA_DEV ];
Haiyang Zhang
maintainerQuoted 106 lines
-----Original Message-----
From: Praveen Kumar <[email protected]>
Sent: Monday, June 26, 2023 10:13 AM
To: souradeep chakrabarti <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>
Subject: Re: [PATCH 2/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsiveQuoted 9 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This is the second part of the fix.
Also this patch adds a new attribute in
mana_context, which gets set whenmana_hwc_send_request()hits a timeout because of host unresponsiveness. This flag then helps to avoid the timeouts in successive calls.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add adriver for
Quoted 7 lines
Microsoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Removed the initialization ofvf_unload_timeout
* Splitted the patch in two.
* Fixed extra space from the commit message.Review
Quoted 7 lines
--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 +++- drivers/net/ethernet/microsoft/mana/hw_channel.c | 12 +++++++++++- include/net/mana/mana.h | 2 ++ 3 files changed, 16 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/net/ethernet/microsoft/mana/gdma_main.cdrivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_deregister_device(struct gdma_dev
Quoted 4 lines
index
8f3f78b68592..6411f01be0d9100644drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0@@ -946,10 +946,12 @@ int mana_gd_deregister_device(struct gdma_dev*gd)
drivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_deregister_device(struct gdma_dev *gd)
Quoted 11 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-0struct gdma_context *gc = gd->gdma_context;struct gdma_general_resp resp = {};struct gdma_general_req req = {}; +struct mana_context *ac;int err;if (gd->pdid ==INVALID_PDID )return -EINVAL ; + ac = gd->driver_data; mana_gd_init_req_hdr(&req.hdr, GDMA_DEREGISTER_DEVICE,sizeof(req),
drivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_deregister_device(struct gdma_dev *gd) · line 964
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +1
-1sizeof(resp));@@ -957,7 +959,7 @@ int mana_gd_deregister_device(struct gdma_dev *gd) req.hdr.dev_id = gd->dev_id; err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); -if (err || resp.hdr.status) { +if ((err || resp.hdr.status) && !ac->vf_unload_timeout) {dev_err (gc->dev,"Failed to deregister device: %d, 0x%x\n" , err, resp.hdr.status);With !ac->vf_unload_timeout option, this message may not be correctly showing err, status. Probably you want to add explicit information during timeouts so that it give right information ? Or have the err, status field properly updated.
Review
Quoted 2 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0if (!err)(quoted diff) +0
-0
b/drivers/net/ethernet/microsoft/mana/hw_channel.cdrivers/net/ethernet/microsoft/mana/hw_channel.c line 9
Quoted 13 lines
index
9d1507eba5b9..492cb2c6e2cb100644drivers/net/ethernet/microsoft/mana/hw_channel.c +2
-0@@ -1,8 +1,10 @@// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* Copyright (c) 2021, Microsoft Corporation. */ +#include "asm-generic/errno.h" #include <net/mana/gdma.h> #include <net/mana/hw_channel.h> +#include <net/mana/mana.h> static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16
*msg_id)drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct
Quoted 2 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +0
-0{@@ -786,12 +788,19 @@ int mana_hwc_send_request(struct
hw_channel_context*hwc, u32req_len,Quoted 9 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-0struct hwc_wq *txq = hwc->txq;struct gdma_req_hdr *req_msg;struct hwc_caller_ctx *ctx; +struct mana_context *ac;u32 dest_vrcq =0 ;u32 dest_vrq =0 ;u16 msg_id;int err;
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, · line 798
Quoted 2 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-0mana_hwc_get_msg_index (hwc, &msg_id); + ac = hwc->gdma_dev->driver_data;Is there a case where
gdma_devbe invalid here ? If so, lets check the state and then proceed further ?
Yes, hwc->gdma_dev is assigned shortly after it's allocated - see the code below. So it's valid.
But hwc->gdma_dev->driver_data is hwc, not "mana_context *ac". There are two gdma_dev in gdma_context: hwc & mana.
You can get ac from: hwc->gdma_dev->gdma_context->mana.driver_data Or, to avoid too many pointer deference, I suggest to put the vf_unload_timeout into gdma_context.
int mana_hwc_create_channel(struct gdma_context *gc)
{
hwc = kzalloc(sizeof(*hwc), GFP_KERNEL);
...
gd->gdma_context = gc;
gd->driver_data = hwc;
hwc->gdma_dev = gd;
hwc->dev = gc->dev;Also, mana_gd_send_request/mana_hwc_send_request() is used in many places, not just unloading.
Should you use timeout value 5 sec, and the vf_unload_timeout flag in unloading path only, and avoid touching other code paths? Please check with hostnet team for suggestions.
If we decide to let the vf_unload_timeout flag affect all code paths, not just unloading, then it should be renamed to hwc_timeout, and submit the second patch separately.
If just use it for unloading, since mana_gd_deregister_device() is used by PF too, name it like: unload_hwc_timeout.
Thanks,
-Haiyang
Quoted 43 lines · Praveen Kumar
Quoted 42 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This is the second part of the fix.
Also this patch adds a new attribute in
mana_context, which gets set whenmana_hwc_send_request()hits a timeout because of host unresponsiveness. This flag then helps to avoid the timeouts in successive calls.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Removed the initialization ofvf_unload_timeout
* Splitted the patch in two.
* Fixed extra space from the commit message.--- drivers/net/ethernet/microsoft/mana/gdma_main.c | 4 +++- drivers/net/ethernet/microsoft/mana/hw_channel.c | 12 +++++++++++- include/net/mana/mana.h | 2 ++ 3 files changed, 16 insertions(+), 2 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +2
-0@@ -946,10 +946,12 @@ int mana_gd_deregister_device(struct gdma_dev *gd)struct gdma_context *gc = gd->gdma_context;struct gdma_general_resp resp = {};struct gdma_general_req req = {}; +struct mana_context *ac;int err;if (gd->pdid ==INVALID_PDID )return -EINVAL ; + ac = gd->driver_data;mana_gd_init_req_hdr (&req.hdr,GDMA_DEREGISTER_DEVICE ,sizeof (req),sizeof (resp));@@ -957,7 +959,7 @@ int mana_gd_deregister_device(struct gdma_dev *gd) req.hdr.dev_id = gd->dev_id;
Quoted 7 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c int mana_gd_deregister_device(struct gdma_dev *gd) · line 964
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +1
-1err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); -if (err || resp.hdr.status) { +if ((err || resp.hdr.status) && !ac->vf_unload_timeout) {dev_err (gc->dev,"Failed to deregister device: %d, 0x%x\n" , err, resp.hdr.status);With !ac->vf_unload_timeout option, this message may not be correctly showing err, status. Probably you want to add explicit information during timeouts so that it give right information ? Or have the err, status field properly updated.
This check !ac->vf_unload_timeout here means if ac->vf_unload_timeout is not yet set, then only consider the err path, else continue the remaining operation.
Quoted 27 lines
Quoted 27 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0if (!err)drivers/net/ethernet/microsoft/mana/hw_channel.c +3
-0@@ -1,8 +1,10 @@// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* Copyright (c) 2021, Microsoft Corporation. */ +#include "asm-generic/errno.h" #include <net/mana/gdma.h> #include <net/mana/hw_channel.h> +#include <net/mana/mana.h> static int mana_hwc_get_msg_index (struct hw_channel_context *hwc,u16 *msg_id) {@@ -786,12 +788,19 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,struct hwc_wq *txq = hwc->txq;struct gdma_req_hdr *req_msg;struct hwc_caller_ctx *ctx; +struct mana_context *ac;u32 dest_vrcq =0 ;u32 dest_vrq =0 ;u16 msg_id;int err;
Quoted 4 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, · line 798
Quoted 2 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-0mana_hwc_get_msg_index (hwc, &msg_id); + ac = hwc->gdma_dev->driver_data;Is there a case where
gdma_devbe invalid here ? If so, lets check the state and then proceed further ?
I can see Haiyang has already in his comment, responded on the same. hwc->gdma_dev will be valid here, but as Haiyang pointed we need to use hwc->gdma_dev->gdma_context->mana.driver_data, or better to relocate the attribute in gdma_context.
Quoted 37 lines
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +5
-0+if (ac->vf_unload_timeout) { +dev_err (hwc->dev,"HWC: vport is already unloaded.\n" ); + err = -ETIMEDOUT ; + goto out; + } tx_wr = &txq->msg_buf->reqs[msg_id];drivers/net/ethernet/microsoft/mana/hw_channel.c int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, · line 837
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-1@@ -825,9 +834,10 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len, goto out; } -if (!wait_for_completion_timeout (&ctx->comp_event,30 *HZ )) { +if (!wait_for_completion_timeout (&ctx->comp_event,5 *HZ )) {IMHO we should have macros instead of magic numbers (5 , 30 or so). But would like others to comment here.
Quoted 19 lines
drivers/net/ethernet/microsoft/mana/hw_channel.c +1
-0dev_err (hwc->dev,"HWC: Request timed out!\n" ); err = -ETIMEDOUT ; + ac->vf_unload_timeout = true; goto out; }include/net/mana/mana.h +2
-0@@ -358,6 +358,8 @@ struct mana_context {u16 num_ports; +bool vf_unload_timeout; +struct mana_eq *eqs;struct net_device *ports[MAX_PORTS_IN_MANA_DEV ];
Haiyang Zhang
maintainerQuoted 18 lines
-----Original Message-----
From: souradeep chakrabarti <[email protected]>
Sent: Monday, June 26, 2023 5:20 AM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>; Souradeep Chakrabarti <[email protected]>
Subject: [PATCH 2/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsive
In general, two patches shouldn't have the same Subject.
For this patch set, the two patches are two steps to fix the unloading issue, and they are not very long. IMHO, they should be in one patch.
Thanks,
- Haiyang
Haiyang Zhang
maintainerQuoted 37 lines
-----Original Message-----
From: Haiyang Zhang <[email protected]>
Sent: Monday, June 26, 2023 11:54 AM
To: souradeep chakrabarti <[email protected]>; KY Srinivasan <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>
Subject: RE: [PATCH 2/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsiveQuoted 8 lines
-----Original Message-----
From: souradeep chakrabarti <[email protected]>
Sent: Monday, June 26, 2023 5:20 AM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected];Quoted 8 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>; Souradeep Chakrabarti <[email protected]>
Subject: [PATCH 2/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsive
Quoted 2 lines
In general, two patches shouldn't have the same Subject.
If two patches are preferred, please use more descriptive subjects like these:
1/2: Fix the infinite waiting on pending_sends during host unresponsiveness 2/2: Avoid extra waits if host not responding in earlier steps
Thanks,
- Haiyang
Quoted 19 lines · Haiyang Zhang
Quoted 18 lines
-----Original Message-----
From: souradeep chakrabarti <[email protected]>
Sent: Monday, June 26, 2023 5:20 AM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; Ajay Sharma <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Souradeep Chakrabarti <[email protected]>; Souradeep Chakrabarti <[email protected]>
Subject: [PATCH 2/2 V3 net] net: mana: Fix MANA VF unload when host is unresponsive
Quoted 7 lines
In general, two patches shouldn't have the same Subject.
For this patch set, the two patches are two steps to fix the unloading issue, and they are not very long. IMHO, they should be in one patch.
Thanks,
- Haiyang
I will create a single patch in next version. As this fixes the unloading issue.
Quoted 8 lines · souradeep chakrabarti
From: Souradeep Chakrabarti <[email protected]>
This patch addresses the VF unload issue, where
mana_dealloc_queues()gets stuck in infinite while loop, because of host unresponsiveness. It adds a timeout in the while loop, to fix it.Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add a driver forMicrosoft Azure Network Adapter)
nit: A correct format of this fixes tag is:
In particular:
* All on lone line
* Description in double quotes.
Quoted 5 lines
Signed-off-by Souradeep Chakrabarti <[email protected]>---
V2 -> V3:
* Splitted the patch in two parts.
* Removed the unnecessary braces frommana_dealloc_queues().
Dexuan Cui
maintainerQuoted 7 lines
From: Simon Horman
Sent: Monday, June 26, 2023 6:05 AMQuoted 4 lines
...
Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add adriver for
Microsoft Azure Network Adapter)
Quoted 8 lines
nit: A correct format of this fixes tag is:
In particular:
* All on lone line
* Description in double quotes.Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure NetworkAdapter (MANA)")
Hi Souradeep, FYI I often refer to:
https://marc.info/?l=linux-pci&m=150905742808166&w=2
The link mentions:
alias gsr='git --no-pager show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"'
"gsr ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f" produces:ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")
Quoted 16 lines · On Mon, 26 Jun 2023 20:06:48 +0000 Dexuan Cui <[email protected]> wrote:
Quoted 7 lines
From: Simon Horman
Sent: Monday, June 26, 2023 6:05 AMQuoted 4 lines
...
Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add adriver for
Microsoft Azure Network Adapter)Quoted 8 lines
nit: A correct format of this fixes tag is:
In particular:
* All on lone line
* Description in double quotes.Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure NetworkAdapter (MANA)")
Quoted 8 lines
Hi Souradeep, FYI I often refer to:
https://marc.info/?l=linux-pci&m=150905742808166&w=2The link mentions:
alias gsr='git --no-pager show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"'"gsr
ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f" produces:ca9c54d2d6a5("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")
You can do same thing without shell alias by using git-config
[alias]
fixes = log -1 --format=fixes gsr = log -1 --format=gsr
[pretty]
fixes = Fixes: %h (\"%s\") gsr = %h (\"%s\")
Then:
$ git gsr 1919b39fc6eabb9a6f9a51706ff6d03865f5df29
1919b39fc6ea ("net: mana: Fix perf regression: remove rx_cqes, tx_cqes counters")Quoted 19 lines · Stephen Hemminger
Quoted 16 lines · On Mon, 26 Jun 2023 20:06:48 +0000 Dexuan Cui <[email protected]> wrote:
Quoted 15 lines
From: Simon Horman
Sent: Monday, June 26, 2023 6:05 AMQuoted 4 lines
...
Fixes ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f (net: mana: Add adriver for
Microsoft Azure Network Adapter)nit: A correct format of this fixes tag is:
In particular:
* All on lone line
* Description in double quotes.Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure NetworkAdapter (MANA)")
Quoted 6 lines
Quoted 6 lines
Hi Souradeep, FYI I often refer to:
https://marc.info/?l=linux-pci&m=150905742808166&w=2The link mentions:
alias gsr='git --no-pager show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"'
Thank you for the advice. Will use it from now onwards.
Quoted 9 lines
Quoted 2 lines
"gsr
ca9c54d2d6a5ab2430c4eda364c77125d62e5e0f" produces:ca9c54d2d6a5("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")You can do same thing without shell alias by using git-config
[alias]
fixes = log -1 --format=fixes gsr = log -1 --format=gsr
Quoted 7 lines
[pretty]
fixes = Fixes: %h (\"%s\") gsr = %h (\"%s\")Then:
$ git gsr 1919b39fc6eabb9a6f9a51706ff6d03865f5df29 1919b39fc6ea ("net: mana: Fix perf regression: remove rx_cqes, tx_cqes counters")
Thank you for the suggestion.