fix bnxt_en queue reset when queue is active
v2 cover fix bnxt_en queue reset when queue is active
The current bnxt_en queue API implementation is buggy when resetting a queue that has active traffic. The problem is that there is no FW involved to stop the flow of packets and relying on napi_disable() isn't enough.
To fix this, call bnxt_hwrm_vnic_update() with MRU set to 0 for both the default and the ntuple vnic to stop the flow of packets. This works for any Rx queue and not only those that have ntuple rules since every Rx queue is either in the default or the ntuple vnic.
The first patch is from Michael Chan and adds the prerequisite vnic functions and definitions.
Tested on BCM957504 while iperf3 is active:
1. Reset a queue that has an ntuple rule steering flow into it
2. Reset all queues in order, one at a time
In both cases the flow is not interrupted.
Sending this to net-next as there is no in-tree kernel consumer of queue API just yet, and there is a patch that changes when the queue_mgmt_ops is registered.
---
v2:
- split setting vnic->mru into a separate patch (Wojciech)
- clarify why napi_enable()/disable() is removed
David Wei (3):
bnxt_en: set vnic->mru in bnxt_hwrm_vnic_cfg()
bnxt_en: stop packet flow during bnxt_queue_stop/start
bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC
Michael Chan (1):
bnxt_en: Add support to call FW to update a VNIC
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 50 ++++++++++++++++--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 3 ++ drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 ++++++++++++++ 3 files changed, 83 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/4 bnxt_en: Add support to call FW to update a VNIC
From: Michael Chan <[email protected]>
Add the HWRM_VNIC_UPDATE message structures and the function to send the message to firmware. This message can be used when disabling and enabling a receive ring within a VNIC. The mru which is the maximum receive size of packets received by the VNIC can be updated.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 20 ++++++++++ drivers/net/ethernet/broadcom/bnxt/bnxt.h | 3 ++ drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 +++++++++++++++++++ 3 files changed, 60 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +20 -0
@@ -10086,6 +10086,26 @@ static int __bnxt_setup_vnic(struct bnxt *bp, struct bnxt_vnic_info *vnic)return rc; } +int bnxt_hwrm_vnic_update (struct bnxt *bp,struct bnxt_vnic_info *vnic, +u8 valid) +{ +struct hwrm_vnic_update_input *req; +int rc; + + rc =hwrm_req_init (bp, req,HWRM_VNIC_UPDATE ); +if (rc) +return rc; + + req->vnic_id =cpu_to_le32 (vnic->fw_vnic_id); + +if (valid &VNIC_UPDATE_REQ_ENABLES_MRU_VALID ) + req->mru =cpu_to_le16 (vnic->mru); + + req->enables =cpu_to_le32 (valid); + +return hwrm_req_send (bp, req); +} +int bnxt_hwrm_vnic_rss_cfg_p5 (struct bnxt *bp,struct bnxt_vnic_info *vnic) {int rc;
drivers/net/ethernet/broadcom/bnxt/bnxt.h +3 -0
@@ -1250,6 +1250,7 @@ struct bnxt_vnic_info {#define BNXT_MAX_CTX_PER_VNIC 8u16 fw_rss_cos_lb_ctx[BNXT_MAX_CTX_PER_VNIC ];u16 fw_l2_ctx_id; +u16 mru;#define BNXT_MAX_UC_ADDRS 4struct bnxt_l2_filter *l2_filters[BNXT_MAX_UC_ADDRS ];/* index 0 always dev_addr */
@@ -2838,6 +2839,8 @@ int bnxt_hwrm_free_wol_fltr(struct bnxt *bp);int bnxt_hwrm_func_resc_qcaps (struct bnxt *bp,bool all);int bnxt_hwrm_func_qcaps (struct bnxt *bp);int bnxt_hwrm_fw_set_time (struct bnxt *); +int bnxt_hwrm_vnic_update (struct bnxt *bp,struct bnxt_vnic_info *vnic, +u8 valid);int bnxt_hwrm_vnic_rss_cfg_p5 (struct bnxt *bp,struct bnxt_vnic_info *vnic);int __bnxt_setup_vnic_p5 (struct bnxt *bp,struct bnxt_vnic_info *vnic); void bnxt_del_one_rss_ctx(struct bnxt *bp, struct bnxt_rss_ctx *rss_ctx,
drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h +37 -0
@@ -6510,6 +6510,43 @@ struct hwrm_vnic_alloc_output {u8 valid; }; +/* hwrm_vnic_update_input (size:256b/32B) */ +struct hwrm_vnic_update_input { +__le16 req_type; +__le16 cmpl_ring; +__le16 seq_id; +__le16 target_id; +__le64 resp_addr; +__le32 vnic_id; +__le32 enables; +#define VNIC_UPDATE_REQ_ENABLES_VNIC_STATE_VALID 0x1UL +#define VNIC_UPDATE_REQ_ENABLES_MRU_VALID 0x2UL +#define VNIC_UPDATE_REQ_ENABLES_METADATA_FORMAT_TYPE_VALID 0x4UL +u8 vnic_state; +#define VNIC_UPDATE_REQ_VNIC_STATE_NORMAL 0x0UL +#define VNIC_UPDATE_REQ_VNIC_STATE_DROP 0x1UL +#define VNIC_UPDATE_REQ_VNIC_STATE_LAST VNIC_UPDATE_REQ_VNIC_STATE_DROP +u8 metadata_format_type; +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_0 0x0UL +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_1 0x1UL +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_2 0x2UL +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_3 0x3UL +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_4 0x4UL +#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_LAST VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_4 +__le16 mru; +u8 unused_1[4 ]; +}; + +/* hwrm_vnic_update_output (size:128b/16B) */ +struct hwrm_vnic_update_output { +__le16 error_code; +__le16 req_type; +__le16 seq_id; +__le16 resp_len; +u8 unused_0[7 ]; +u8 valid; +}; +/* hwrm_vnic_free_input (size:192b/24B) */ struct hwrm_vnic_free_input {__le16 req_type;
Signature · 1 line
2.43.0
PATCH v2 2/4 bnxt_en: set vnic->mru in bnxt_hwrm_vnic_cfg()
In reply to David WeiSet the newly added vnic->mru field in bnxt_hwrm_vnic_cfg().
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -1
@@ -6579,7 +6579,8 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, struct bnxt_vnic_info *vnic) req->dflt_ring_grp =cpu_to_le16 (bp->grp_info[grp_idx].fw_grp_id); req->lb_rule =cpu_to_le16 (0xffff ); vnic_mru: - req->mru =cpu_to_le16 (bp->dev->mtu +ETH_HLEN +VLAN_HLEN ); + vnic->mru = bp->dev->mtu +ETH_HLEN +VLAN_HLEN ; + req->mru =cpu_to_le16 (vnic->mru); req->vnic_id =cpu_to_le16 (vnic->fw_vnic_id);#ifdef CONFIG_BNXT_SRIOV
Signature · 1 line
2.43.0
PATCH v2 3/4 bnxt_en: stop packet flow during bnxt_queue_stop/start
In reply to David WeiThe current implementation when resetting a queue while packets are flowing puts the queue into an inconsistent state.
There needs to be some synchronisation with the FW. Add calls to bnxt_hwrm_vnic_update() to set the MRU for both the default and ntuple vnic during queue start/stop. When the MRU is set to 0, flow is stopped. Each Rx queue belongs to either the default or the ntuple vnic.
With calling bnxt_hwrm_vnic_update() the calls to napi_enable() and napi_disable() must be removed for reset to work on a queue that has active traffic flowing e.g. iperf3.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +18 -4
@@ -15172,7 +15172,8 @@ static int bnxt_queue_start(struct net_device *dev, void *qmem, int idx)struct bnxt *bp =netdev_priv (dev);struct bnxt_rx_ring_info *rxr, *clone;struct bnxt_cp_ring_info *cpr; -int rc; +struct bnxt_vnic_info *vnic; +int i, rc; rxr = &bp->rx_ring[idx]; clone = qmem;
@@ -15197,11 +15198,16 @@ static int bnxt_queue_start(struct net_device *dev, void *qmem, int idx)if (bp->flags &BNXT_FLAG_AGG_RINGS )bnxt_db_write (bp, &rxr->rx_agg_db, rxr->rx_agg_prod); -napi_enable (&rxr->bnapi->napi); - cpr = &rxr->bnapi->cp_ring; cpr->sw_stats->rx.rx_resets++; +for (i =0 ; i <=BNXT_VNIC_NTUPLE ; i++) { + vnic = &bp->vnic_info[i]; + vnic->mru = bp->dev->mtu +ETH_HLEN +VLAN_HLEN ; +bnxt_hwrm_vnic_update (bp, vnic, +VNIC_UPDATE_REQ_ENABLES_MRU_VALID ); + } +return 0 ; err_free_hwrm_rx_ring:
@@ -15213,9 +15219,17 @@ static int bnxt_queue_stop(struct net_device *dev, void *qmem, int idx) {struct bnxt *bp =netdev_priv (dev);struct bnxt_rx_ring_info *rxr; +struct bnxt_vnic_info *vnic; +int i; + +for (i =0 ; i <=BNXT_VNIC_NTUPLE ; i++) { + vnic = &bp->vnic_info[i]; + vnic->mru =0 ; +bnxt_hwrm_vnic_update (bp, vnic, +VNIC_UPDATE_REQ_ENABLES_MRU_VALID ); + } rxr = &bp->rx_ring[idx]; -napi_disable (&rxr->bnapi->napi);bnxt_hwrm_rx_ring_free (bp, rxr, false);bnxt_hwrm_rx_agg_ring_free (bp, rxr, false); rxr->rx_next_cons =0 ;
Signature · 1 line
2.43.0
PATCH v2 4/4 bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC
In reply to David WeiThe queue API calls bnxt_hwrm_vnic_update() to stop/start the flow of packets. It can only be called if BNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only setting queue_mgmt_ops if this is true.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -2
@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) dev->stat_ops = &bnxt_stat_ops; dev->watchdog_timeo =BNXT_TX_TIMEOUT ; dev->ethtool_ops = &bnxt_ethtool_ops; - dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;pci_set_drvdata (pdev, dev); rc =bnxt_alloc_hwrm_resources (bp);
@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)INIT_LIST_HEAD (&bp->usr_fltr_list); -if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) +if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) { bp->rss_cap |=BNXT_RSS_CAP_MULTI_RSS_CTX ; + dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops; + } rc =register_netdev (dev);if (rc)
Signature · 1 line
2.43.0
Michael Chan
maintainerRe: [PATCH net-next v2 4/4] bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC
Quoted 23 lines · David Wei
The queue API calls
bnxt_hwrm_vnic_update()to stop/start the flow of packets. It can only be called ifBNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only settingqueue_mgmt_opsif this is true.Signed-off-by David Wei <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-1@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) dev->stat_ops = &bnxt_stat_ops; dev->watchdog_timeo =BNXT_TX_TIMEOUT ; dev->ethtool_ops = &bnxt_ethtool_ops; - dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;pci_set_drvdata (pdev, dev); rc =bnxt_alloc_hwrm_resources (bp);
drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 15894
Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-1@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)INIT_LIST_HEAD (&bp->usr_fltr_list); -if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) +if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) {
Thanks for the patch. We found out during internal testing that an additional FW fix is required to make the ring restart 100% reliable with traffic. FW needs to add one more step to flush the RX pipeline during HWRM_VNIC_UPDATE. Once we determine which FW versions will have the fix, we can add the conditional check here to make this patch more complete. I think we just need about a week to determine that. Please hold off on this patchset. Thanks.
Re: [PATCH net-next v2 4/4] bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC
Quoted 27 lines · Michael Chan
Quoted 26 lines · David Wei
The queue API calls
bnxt_hwrm_vnic_update()to stop/start the flow of packets. It can only be called ifBNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only settingqueue_mgmt_opsif this is true.Signed-off-by David Wei <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-1@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) dev->stat_ops = &bnxt_stat_ops; dev->watchdog_timeo =BNXT_TX_TIMEOUT ; dev->ethtool_ops = &bnxt_ethtool_ops; - dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;pci_set_drvdata (pdev, dev); rc =bnxt_alloc_hwrm_resources (bp);@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)INIT_LIST_HEAD (&bp->usr_fltr_list);
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 15894
Quoted 2 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-1-if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) +if (BNXT_SUPPORTS_NTUPLE_VNIC (bp)) {Thanks for the patch. We found out during internal testing that an additional FW fix is required to make the ring restart 100% reliable with traffic. FW needs to add one more step to flush the RX pipeline during
HWRM_VNIC_UPDATE. Once we determine which FW versions will have the fix, we can add the conditional check here to make this patch more complete. I think we just need about a week to determine that. Please hold off on this patchset. Thanks.
Got it, no problem. Let me know the FW version and I'll gate the ops behind it. Thanks.
Quoted 20 lines · David Wei
The current
bnxt_enqueue API implementation is buggy when resetting a queue that has active traffic. The problem is that there is no FW involved to stop the flow of packets and relying onnapi_disable()isn't enough.To fix this, call
bnxt_hwrm_vnic_update()with MRU set to 0 for both the default and the ntuple vnic to stop the flow of packets. This works for any Rx queue and not only those that have ntuple rules since every Rx queue is either in the default or the ntuple vnic.The first patch is from Michael Chan and adds the prerequisite vnic functions and definitions.
Tested on BCM957504 while iperf3 is active:
1. Reset a queue that has an ntuple rule steering flow into it
2. Reset all queues in order, one at a timeIn both cases the flow is not interrupted.
Quoted 8 lines
Sending this to net-next as there is no in-tree kernel consumer of queue API just yet, and there is a patch that changes when the
queue_mgmt_opsis registered.---
v2:
- split settingvnic->mruinto a separate patch (Wojciech)
- clarify why napi_enable()/disable() is removed
Thanks!
For the whole series:
Remember to include my tag if next version will be needed :)
Quoted 14 lines
David Wei (3):
bnxt_en: setvnic->mruinbnxt_hwrm_vnic_cfg()
bnxt_en: stop packet flow during bnxt_queue_stop/start
bnxt_en: only setdev->queue_mgmt_opsifBNXT_SUPPORTS_NTUPLE_VNICMichael Chan (1):
bnxt_en: Add support to call FW to update a VNICdrivers/net/ethernet/broadcom/bnxt/bnxt.c | 50 ++++++++++++++++--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 3 ++ drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 ++++++++++++++ 3 files changed, 83 insertions(+), 7 deletions(-)