Skip to contentSkip to conversation

fix bnxt_en queue reset when queue is active

PATCH Superseded David Wei 3 participants 8 messages Jul 31, 2024 – Aug 1, 2024

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

David Wei

author
#0 lore raw

v2 cover fix bnxt_en queue reset when queue is active

The current bnxt_en queue API implementation is buggy when resetting a queue that has active traffic. The problem is that there is no FW involved to stop the flow of packets and relying on napi_disable() isn't enough.

To fix this, call bnxt_hwrm_vnic_update() with MRU set to 0 for both the default and the ntuple vnic to stop the flow of packets. This works for any Rx queue and not only those that have ntuple rules since every Rx queue is either in the default or the ntuple vnic.

The first patch is from Michael Chan and adds the prerequisite vnic functions and definitions.

Tested on BCM957504 while iperf3 is active:

1. Reset a queue that has an ntuple rule steering flow into it
2. Reset all queues in order, one at a time

In both cases the flow is not interrupted.

Sending this to net-next as there is no in-tree kernel consumer of queue API just yet, and there is a patch that changes when the queue_mgmt_ops is registered.

---
v2:
- split setting vnic->mru into a separate patch (Wojciech)
- clarify why napi_enable()/disable() is removed

David Wei (3):
bnxt_en: set vnic->mru in bnxt_hwrm_vnic_cfg()
bnxt_en: stop packet flow during bnxt_queue_stop/start
bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC

Michael Chan (1):
bnxt_en: Add support to call FW to update a VNIC

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 50 ++++++++++++++++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  3 ++
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 ++++++++++++++
 3 files changed, 83 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v2 1/4 bnxt_en: Add support to call FW to update a VNIC

From: Michael Chan <[email protected]>

Add the HWRM_VNIC_UPDATE message structures and the function to send the message to firmware. This message can be used when disabling and enabling a receive ring within a VNIC. The mru which is the maximum receive size of packets received by the VNIC can be updated.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 20 ++++++++++
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  3 ++
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 +++++++++++++++++++
 3 files changed, 60 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +20 -0
index ffa74c26ee53..52998065956e 100644
@@ -10086,6 +10086,26 @@ static int __bnxt_setup_vnic(struct bnxt *bp, struct bnxt_vnic_info *vnic)
 	return rc;
 }
 
+int bnxt_hwrm_vnic_update(struct bnxt *bp, struct bnxt_vnic_info *vnic,
+			  u8 valid)
+{
+	struct hwrm_vnic_update_input *req;
+	int rc;
+
+	rc = hwrm_req_init(bp, req, HWRM_VNIC_UPDATE);
+	if (rc)
+		return rc;
+
+	req->vnic_id = cpu_to_le32(vnic->fw_vnic_id);
+
+	if (valid & VNIC_UPDATE_REQ_ENABLES_MRU_VALID)
+		req->mru = cpu_to_le16(vnic->mru);
+
+	req->enables = cpu_to_le32(valid);
+
+	return hwrm_req_send(bp, req);
+}
+
 int bnxt_hwrm_vnic_rss_cfg_p5(struct bnxt *bp, struct bnxt_vnic_info *vnic)
 {
 	int rc;
drivers/net/ethernet/broadcom/bnxt/bnxt.h +3 -0
index 6bbdc718c3a7..5de67f718993 100644
@@ -1250,6 +1250,7 @@ struct bnxt_vnic_info {
 #define BNXT_MAX_CTX_PER_VNIC	8
 	u16		fw_rss_cos_lb_ctx[BNXT_MAX_CTX_PER_VNIC];
 	u16		fw_l2_ctx_id;
+	u16		mru;
 #define BNXT_MAX_UC_ADDRS	4
 	struct bnxt_l2_filter *l2_filters[BNXT_MAX_UC_ADDRS];
 				/* index 0 always dev_addr */
@@ -2838,6 +2839,8 @@ int bnxt_hwrm_free_wol_fltr(struct bnxt *bp);
 int bnxt_hwrm_func_resc_qcaps(struct bnxt *bp, bool all);
 int bnxt_hwrm_func_qcaps(struct bnxt *bp);
 int bnxt_hwrm_fw_set_time(struct bnxt *);
+int bnxt_hwrm_vnic_update(struct bnxt *bp, struct bnxt_vnic_info *vnic,
+			  u8 valid);
 int bnxt_hwrm_vnic_rss_cfg_p5(struct bnxt *bp, struct bnxt_vnic_info *vnic);
 int __bnxt_setup_vnic_p5(struct bnxt *bp, struct bnxt_vnic_info *vnic);
 void bnxt_del_one_rss_ctx(struct bnxt *bp, struct bnxt_rss_ctx *rss_ctx,
drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h +37 -0
index f219709f9563..933f48a62586 100644
@@ -6510,6 +6510,43 @@ struct hwrm_vnic_alloc_output {
 	u8	valid;
 };
 
+/* hwrm_vnic_update_input (size:256b/32B) */
+struct hwrm_vnic_update_input {
+	__le16	req_type;
+	__le16	cmpl_ring;
+	__le16	seq_id;
+	__le16	target_id;
+	__le64	resp_addr;
+	__le32	vnic_id;
+	__le32	enables;
+	#define VNIC_UPDATE_REQ_ENABLES_VNIC_STATE_VALID               0x1UL
+	#define VNIC_UPDATE_REQ_ENABLES_MRU_VALID                      0x2UL
+	#define VNIC_UPDATE_REQ_ENABLES_METADATA_FORMAT_TYPE_VALID     0x4UL
+	u8	vnic_state;
+	#define VNIC_UPDATE_REQ_VNIC_STATE_NORMAL 0x0UL
+	#define VNIC_UPDATE_REQ_VNIC_STATE_DROP   0x1UL
+	#define VNIC_UPDATE_REQ_VNIC_STATE_LAST  VNIC_UPDATE_REQ_VNIC_STATE_DROP
+	u8	metadata_format_type;
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_0 0x0UL
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_1 0x1UL
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_2 0x2UL
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_3 0x3UL
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_4 0x4UL
+	#define VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_LAST VNIC_UPDATE_REQ_METADATA_FORMAT_TYPE_4
+	__le16	mru;
+	u8	unused_1[4];
+};
+
+/* hwrm_vnic_update_output (size:128b/16B) */
+struct hwrm_vnic_update_output {
+	__le16	error_code;
+	__le16	req_type;
+	__le16	seq_id;
+	__le16	resp_len;
+	u8	unused_0[7];
+	u8	valid;
+};
+
 /* hwrm_vnic_free_input (size:192b/24B) */
 struct hwrm_vnic_free_input {
 	__le16	req_type;
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH v2 2/4 bnxt_en: set vnic->mru in bnxt_hwrm_vnic_cfg()

In reply to David Wei

Set the newly added vnic->mru field in bnxt_hwrm_vnic_cfg().

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -1
index 52998065956e..8822d7a17fbf 100644
@@ -6579,7 +6579,8 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, struct bnxt_vnic_info *vnic)
 	req->dflt_ring_grp = cpu_to_le16(bp->grp_info[grp_idx].fw_grp_id);
 	req->lb_rule = cpu_to_le16(0xffff);
 vnic_mru:
-	req->mru = cpu_to_le16(bp->dev->mtu + ETH_HLEN + VLAN_HLEN);
+	vnic->mru = bp->dev->mtu + ETH_HLEN + VLAN_HLEN;
+	req->mru = cpu_to_le16(vnic->mru);
 
 	req->vnic_id = cpu_to_le16(vnic->fw_vnic_id);
 #ifdef CONFIG_BNXT_SRIOV
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH v2 3/4 bnxt_en: stop packet flow during bnxt_queue_stop/start

In reply to David Wei

The current implementation when resetting a queue while packets are flowing puts the queue into an inconsistent state.

There needs to be some synchronisation with the FW. Add calls to bnxt_hwrm_vnic_update() to set the MRU for both the default and ntuple vnic during queue start/stop. When the MRU is set to 0, flow is stopped. Each Rx queue belongs to either the default or the ntuple vnic.

With calling bnxt_hwrm_vnic_update() the calls to napi_enable() and napi_disable() must be removed for reset to work on a queue that has active traffic flowing e.g. iperf3.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +18 -4
index 8822d7a17fbf..ce60c9322fe6 100644
@@ -15172,7 +15172,8 @@ static int bnxt_queue_start(struct net_device *dev, void *qmem, int idx)
 	struct bnxt *bp = netdev_priv(dev);
 	struct bnxt_rx_ring_info *rxr, *clone;
 	struct bnxt_cp_ring_info *cpr;
-	int rc;
+	struct bnxt_vnic_info *vnic;
+	int i, rc;
 
 	rxr = &bp->rx_ring[idx];
 	clone = qmem;
@@ -15197,11 +15198,16 @@ static int bnxt_queue_start(struct net_device *dev, void *qmem, int idx)
 	if (bp->flags & BNXT_FLAG_AGG_RINGS)
 		bnxt_db_write(bp, &rxr->rx_agg_db, rxr->rx_agg_prod);
 
-	napi_enable(&rxr->bnapi->napi);
-
 	cpr = &rxr->bnapi->cp_ring;
 	cpr->sw_stats->rx.rx_resets++;
 
+	for (i = 0; i <= BNXT_VNIC_NTUPLE; i++) {
+		vnic = &bp->vnic_info[i];
+		vnic->mru = bp->dev->mtu + ETH_HLEN + VLAN_HLEN;
+		bnxt_hwrm_vnic_update(bp, vnic,
+				      VNIC_UPDATE_REQ_ENABLES_MRU_VALID);
+	}
+
 	return 0;
 
 err_free_hwrm_rx_ring:
@@ -15213,9 +15219,17 @@ static int bnxt_queue_stop(struct net_device *dev, void *qmem, int idx)
 {
 	struct bnxt *bp = netdev_priv(dev);
 	struct bnxt_rx_ring_info *rxr;
+	struct bnxt_vnic_info *vnic;
+	int i;
+
+	for (i = 0; i <= BNXT_VNIC_NTUPLE; i++) {
+		vnic = &bp->vnic_info[i];
+		vnic->mru = 0;
+		bnxt_hwrm_vnic_update(bp, vnic,
+				      VNIC_UPDATE_REQ_ENABLES_MRU_VALID);
+	}
 
 	rxr = &bp->rx_ring[idx];
-	napi_disable(&rxr->bnapi->napi);
 	bnxt_hwrm_rx_ring_free(bp, rxr, false);
 	bnxt_hwrm_rx_agg_ring_free(bp, rxr, false);
 	rxr->rx_next_cons = 0;
Signature · 1 line
2.43.0
author
#4 lore raw

PATCH v2 4/4 bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC

In reply to David Wei

The queue API calls bnxt_hwrm_vnic_update() to stop/start the flow of packets. It can only be called if BNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only setting queue_mgmt_ops if this is true.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -2
index ce60c9322fe6..2801ae94d87b 100644
@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	dev->stat_ops = &bnxt_stat_ops;
 	dev->watchdog_timeo = BNXT_TX_TIMEOUT;
 	dev->ethtool_ops = &bnxt_ethtool_ops;
-	dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;
 	pci_set_drvdata(pdev, dev);
 
 	rc = bnxt_alloc_hwrm_resources(bp);
@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	INIT_LIST_HEAD(&bp->usr_fltr_list);
 
-	if (BNXT_SUPPORTS_NTUPLE_VNIC(bp))
+	if (BNXT_SUPPORTS_NTUPLE_VNIC(bp)) {
 		bp->rss_cap |= BNXT_RSS_CAP_MULTI_RSS_CTX;
+		dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;
+	}
 
 	rc = register_netdev(dev);
 	if (rc)
Signature · 1 line
2.43.0
maintainer
#5 lore raw

Re: [PATCH net-next v2 4/4] bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC

Quoted 23 lines · David Wei

The queue API calls bnxt_hwrm_vnic_update() to stop/start the flow of packets. It can only be called if BNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only setting queue_mgmt_ops if this is true.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -1
index ce60c9322fe6..2801ae94d87b 100644
@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
        dev->stat_ops = &bnxt_stat_ops;
        dev->watchdog_timeo = BNXT_TX_TIMEOUT;
        dev->ethtool_ops = &bnxt_ethtool_ops;
-       dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;
        pci_set_drvdata(pdev, dev);
 
        rc = bnxt_alloc_hwrm_resources(bp);

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 15894

Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -1
@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
        INIT_LIST_HEAD(&bp->usr_fltr_list);
 
-       if (BNXT_SUPPORTS_NTUPLE_VNIC(bp))
+       if (BNXT_SUPPORTS_NTUPLE_VNIC(bp)) {

Thanks for the patch. We found out during internal testing that an additional FW fix is required to make the ring restart 100% reliable with traffic. FW needs to add one more step to flush the RX pipeline during HWRM_VNIC_UPDATE. Once we determine which FW versions will have the fix, we can add the conditional check here to make this patch more complete. I think we just need about a week to determine that. Please hold off on this patchset. Thanks.

author
#6 lore raw

Re: [PATCH net-next v2 4/4] bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC

Quoted 27 lines · Michael Chan
Quoted 26 lines · David Wei

The queue API calls bnxt_hwrm_vnic_update() to stop/start the flow of packets. It can only be called if BNXT_SUPPORTS_NTUPLE_VNIC(), so key support for it by only setting queue_mgmt_ops if this is true.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -1
index ce60c9322fe6..2801ae94d87b 100644
@@ -15713,7 +15713,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
        dev->stat_ops = &bnxt_stat_ops;
        dev->watchdog_timeo = BNXT_TX_TIMEOUT;
        dev->ethtool_ops = &bnxt_ethtool_ops;
-       dev->queue_mgmt_ops = &bnxt_queue_mgmt_ops;
        pci_set_drvdata(pdev, dev);
 
        rc = bnxt_alloc_hwrm_resources(bp);
@@ -15892,8 +15891,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
        INIT_LIST_HEAD(&bp->usr_fltr_list);
 
Quoted 10 lines

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 15894

Quoted 2 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -1
-       if (BNXT_SUPPORTS_NTUPLE_VNIC(bp))
+       if (BNXT_SUPPORTS_NTUPLE_VNIC(bp)) {

Thanks for the patch. We found out during internal testing that an additional FW fix is required to make the ring restart 100% reliable with traffic. FW needs to add one more step to flush the RX pipeline during HWRM_VNIC_UPDATE. Once we determine which FW versions will have the fix, we can add the conditional check here to make this patch more complete. I think we just need about a week to determine that. Please hold off on this patchset. Thanks.

Got it, no problem. Let me know the FW version and I'll gate the ops behind it. Thanks.

Reviewed-by on the v2 series
#7 lore raw
In reply to David Wei
Quoted 20 lines · David Wei

The current bnxt_en queue API implementation is buggy when resetting a queue that has active traffic. The problem is that there is no FW involved to stop the flow of packets and relying on napi_disable() isn't enough.

To fix this, call bnxt_hwrm_vnic_update() with MRU set to 0 for both the default and the ntuple vnic to stop the flow of packets. This works for any Rx queue and not only those that have ntuple rules since every Rx queue is either in the default or the ntuple vnic.

The first patch is from Michael Chan and adds the prerequisite vnic functions and definitions.

Tested on BCM957504 while iperf3 is active:

1. Reset a queue that has an ntuple rule steering flow into it
2. Reset all queues in order, one at a time

In both cases the flow is not interrupted.

Quoted 8 lines

Sending this to net-next as there is no in-tree kernel consumer of queue API just yet, and there is a patch that changes when the queue_mgmt_ops is registered.

---
v2:
- split setting vnic->mru into a separate patch (Wojciech)
- clarify why napi_enable()/disable() is removed

Thanks!
For the whole series:

Remember to include my tag if next version will be needed :)

Quoted 14 lines

David Wei (3):
bnxt_en: set vnic->mru in bnxt_hwrm_vnic_cfg()
bnxt_en: stop packet flow during bnxt_queue_stop/start
bnxt_en: only set dev->queue_mgmt_ops if BNXT_SUPPORTS_NTUPLE_VNIC

Michael Chan (1):
bnxt_en: Add support to call FW to update a VNIC

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 50 ++++++++++++++++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  3 ++
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 37 ++++++++++++++
 3 files changed, 83 insertions(+), 7 deletions(-)