Skip to contentSkip to conversation

RDMA/bnxt_re: Bug fixes

PATCH Applied Selvin Xavier 2 participants 12 messages Oct 8, 2024 – Oct 11, 2024

Subsystems (2)
lore
Share & tools
lore

Selvin Xavier

author maintainer
#0 lore raw

cover RDMA/bnxt_re: Bug fixes

Few bugfixes for bnxt_re driver. Please review and apply.

Thanks,
Selvin Xavier

Abhishek Mohapatra (1):
RDMA/bnxt_re: Fix the max CQ WQEs for older adapters

Bhargava Chenna Marreddy (1):
RDMA/bnxt_re: Fix a bug while setting up Level-2 PBL pages

Chandramohan Akula (1):
RDMA/bnxt_re: Change the sequence of updating the CQ toggle value

Kalesh AP (5):
RDMA/bnxt_re: Fix out of bound check
RDMA/ bnxt_re: Return more meaningful error RDMA/bnxt_re: Fix a possible NULL pointer dereference RDMA/bnxt_re: Fix an error path in bnxt_re_add_device RDMA/bnxt_re: Fix the GID table length

Kashyap Desai (1):
RDMA/bnxt_re: Fix incorrect dereference of srq in async event

Selvin Xavier (1):
RDMA/bnxt_re: Avoid CPU lockups due fifo occupancy check loop

 drivers/infiniband/hw/bnxt_re/hw_counters.c |  2 +-
 drivers/infiniband/hw/bnxt_re/main.c        | 42 ++++++++++++++---------------
 drivers/infiniband/hw/bnxt_re/qplib_fp.c    |  5 ++++
 drivers/infiniband/hw/bnxt_re/qplib_rcfw.c  |  2 +-
 drivers/infiniband/hw/bnxt_re/qplib_res.c   | 19 +++----------
 drivers/infiniband/hw/bnxt_re/qplib_sp.c    | 11 +++++++-
 drivers/infiniband/hw/bnxt_re/qplib_sp.h    |  1 +
 7 files changed, 42 insertions(+), 40 deletions(-)
Signature · 1 line
2.5.5
author maintainer
#1 lore raw

PATCH 1/10 RDMA/bnxt_re: Fix the max CQ WQEs for older adapters

From: Abhishek Mohapatra <[email protected]>

Older adapters doesn't support the MAX CQ WQEs reported by older FW. So restrict the value reported to 1M always for older adapters.

Fixes 1ac5a4047975 ("RDMA/bnxt_re: Add bnxt_re RoCE driver")
---
 drivers/infiniband/hw/bnxt_re/qplib_sp.c | 2 ++
 drivers/infiniband/hw/bnxt_re/qplib_sp.h | 1 +
 2 files changed, 3 insertions(+)
drivers/infiniband/hw/bnxt_re/qplib_sp.c +2 -0
index 4f75e7e..32c1cc7 100644
@@ -140,6 +140,8 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
 			    min_t(u32, sb->max_sge_var_wqe, BNXT_VAR_MAX_SGE) : 6;
 	attr->max_cq = le32_to_cpu(sb->max_cq);
 	attr->max_cq_wqes = le32_to_cpu(sb->max_cqe);
+	if (!bnxt_qplib_is_chip_gen_p7(rcfw->res->cctx))
+		attr->max_cq_wqes = min_t(u32, BNXT_QPLIB_MAX_CQ_WQES, attr->max_cq_wqes);
 	attr->max_cq_sges = attr->max_qp_sges;
 	attr->max_mr = le32_to_cpu(sb->max_mr);
 	attr->max_mw = le32_to_cpu(sb->max_mw);
drivers/infiniband/hw/bnxt_re/qplib_sp.h +1 -0
index acd9c14..ecf3f45 100644
@@ -56,6 +56,7 @@ struct bnxt_qplib_dev_attr {
 	u32				max_qp_wqes;
 	u32				max_qp_sges;
 	u32				max_cq;
+#define BNXT_QPLIB_MAX_CQ_WQES          0xfffff
 	u32				max_cq_wqes;
 	u32				max_cq_sges;
 	u32				max_mr;
Signature · 1 line
2.5.5
author maintainer
#2 lore raw

PATCH 2/10 RDMA/bnxt_re: Fix out of bound check

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

Driver exports pacing stats only on GenP5 and P7 adapters. But while parsing the pacing stats, driver has a check for "rdev->dbr_pacing". This caused a trace when KASAN is enabled.

BUG: KASAN: slab-out-of-bounds in bnxt_re_get_hw_stats+0x2b6a/0x2e00 [bnxt_re]
Write of size 8 at addr ffff8885942a6340 by task modprobe/4809
Fixes 8b6573ff3420 ("bnxt_re: Update the debug counters for doorbell pacing")
---
 drivers/infiniband/hw/bnxt_re/hw_counters.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/hw_counters.c +1 -1
index 128651c..1e63f80 100644
@@ -366,7 +366,7 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev,
 				goto done;
 			}
 		}
-		if (rdev->pacing.dbr_pacing)
+		if (rdev->pacing.dbr_pacing && bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx))
 			bnxt_re_copy_db_pacing_stats(rdev, stats);
 	}
 
Signature · 1 line
2.5.5
author maintainer
#3 lore raw

PATCH 3/10 RDMA/bnxt_re: Fix incorrect dereference of srq in async event

In reply to Selvin Xavier

From: Kashyap Desai <[email protected]>

Currently driver is not getting correct srq. Dereference only if qplib has a valid srq.

Fixes b02fd3f79ec3 ("RDMA/bnxt_re: Report async events and errors")
---
 drivers/infiniband/hw/bnxt_re/main.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +5 -2
index 686e405..dd39948 100644
@@ -1028,12 +1028,15 @@ static int bnxt_re_handle_unaffi_async_event(struct creq_func_event
 static int bnxt_re_handle_qp_async_event(struct creq_qp_event *qp_event,
 					 struct bnxt_re_qp *qp)
 {
-	struct bnxt_re_srq *srq = container_of(qp->qplib_qp.srq, struct bnxt_re_srq,
-					       qplib_srq);
 	struct creq_qp_error_notification *err_event;
+	struct bnxt_re_srq *srq = NULL;
 	struct ib_event event = {};
 	unsigned int flags;
 
+	if (qp->qplib_qp.srq)
+		srq =  container_of(qp->qplib_qp.srq, struct bnxt_re_srq,
+				    qplib_srq);
+
 	if (qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_ERR &&
 	    rdma_is_kernel_res(&qp->ib_qp.res)) {
 		flags = bnxt_re_lock_cqs(qp);
Signature · 1 line
2.5.5
author maintainer
#4 lore raw

PATCH 4/10 RDMA/ bnxt_re: Return more meaningful error

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

When the HWRM command fails, driver currently returns -EFAULT(Bad address). This does not look correct.

Modified to return -EIO(I/O error).

Fixes cc1ec769b87c ("RDMA/bnxt_re: Fixing the Control path command and response handling")
Fixes 65288a22ddd8 ("RDMA/bnxt_re: use shadow qd while posting non blocking rcfw command")
---
 drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +1 -1
index 3ffaef0c..7294221 100644
@@ -525,7 +525,7 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw,
 		/* failed with status */
 		dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x status %#x\n",
 			cookie, opcode, evnt->status);
-		rc = -EFAULT;
+		rc = -EIO;
 	}
 
 	return rc;
Signature · 1 line
2.5.5
author maintainer
#5 lore raw

PATCH 5/10 RDMA/bnxt_re: Fix a possible NULL pointer dereference

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

There is a possibility of a NULL pointer dereference in the failure path of bnxt_re_add_device().
To address that, moved the update of "rdev->adev" to bnxt_re_dev_add().

Fixes dee3da3422d5 ("RDMA/bnxt_re: Change aux driver data to en_info to hold more information")
---
 drivers/infiniband/hw/bnxt_re/main.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +3 -3
index dd39948..915b0d3 100644
@@ -960,7 +960,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev)
 	return ib_register_device(ibdev, "bnxt_re%d", &rdev->en_dev->pdev->dev);
 }
 
-static struct bnxt_re_dev *bnxt_re_dev_add(struct bnxt_aux_priv *aux_priv,
+static struct bnxt_re_dev *bnxt_re_dev_add(struct auxiliary_device *adev,
 					   struct bnxt_en_dev *en_dev)
 {
 	struct bnxt_re_dev *rdev;
@@ -976,6 +976,7 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct bnxt_aux_priv *aux_priv,
 	rdev->nb.notifier_call = NULL;
 	rdev->netdev = en_dev->net;
 	rdev->en_dev = en_dev;
+	rdev->adev = adev;
 	rdev->id = rdev->en_dev->pdev->devfn;
 	INIT_LIST_HEAD(&rdev->qp_list);
 	mutex_init(&rdev->qp_lock);
@@ -1829,7 +1830,6 @@ static void bnxt_re_update_en_info_rdev(struct bnxt_re_dev *rdev,
 	 */
 	rtnl_lock();
 	en_info->rdev = rdev;
-	rdev->adev = adev;
 	rtnl_unlock();
 }
 
@@ -1846,7 +1846,7 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type)
 	en_dev = en_info->en_dev;
 
 
-	rdev = bnxt_re_dev_add(aux_priv, en_dev);
+	rdev = bnxt_re_dev_add(adev, en_dev);
 	if (!rdev || !rdev_to_dev(rdev)) {
 		rc = -ENOMEM;
 		goto exit;
Signature · 1 line
2.5.5
author maintainer
#6 lore raw

PATCH 6/10 RDMA/bnxt_re: Avoid CPU lockups due fifo occupancy check loop

In reply to Selvin Xavier

Driver waits indefinitely for the fifo occupancy to go below
a threshold as soon as the pacing interrupt is received. This can cause soft lockup on one of the processors, if the rate of DB is very high.

Add a loop count for FPGA and exit the __wait_for_fifo_occupancy_below_th if the loop is taking more time. Pacing will be continuing until the occupancy is below the threshold. This is ensured by the checks in bnxt_re_pacing_timer_exp and further scheduling the work for pacing based on the fifo occupancy.

Fixes 2ad4e6303a6d ("RDMA/bnxt_re: Implement doorbell pacing algorithm")
---
 drivers/infiniband/hw/bnxt_re/main.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/infiniband/hw/bnxt_re/main.c +9 -0
index 915b0d3..b1dcb6b 100644
@@ -534,6 +534,7 @@ static bool is_dbr_fifo_full(struct bnxt_re_dev *rdev)
 static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev)
 {
 	struct bnxt_qplib_db_pacing_data *pacing_data = rdev->qplib_res.pacing_data;
+	u32 retry_fifo_check = 1000;
 	u32 fifo_occup;
 
 	/* loop shouldn't run infintely as the occupancy usually goes
@@ -547,6 +548,14 @@ static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev)
 
 		if (fifo_occup < pacing_data->pacing_th)
 			break;
+		if (!retry_fifo_check--) {
+			dev_info_once(rdev_to_dev(rdev),
+				      "%s: fifo_occup = 0x%xfifo_max_depth = 0x%x pacing_th = 0x%x\n",
+				      __func__, fifo_occup, pacing_data->fifo_max_depth,
+					pacing_data->pacing_th);
+			break;
+		}
+
 	}
 }
 
Signature · 1 line
2.5.5
author maintainer
#7 lore raw

PATCH 7/10 RDMA/bnxt_re: Fix an error path in bnxt_re_add_device

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

In bnxt_re_add_device(), when register netdev notifier fails, driver is not unregistering the IB device in the error cleanup path. Also, removed the duplicate cleanup in error path of bnxt_re_probe.

Fixes 94a9dc6ac8f7 ("RDMA/bnxt_re: Group all operations under add_device and remove_device")
---
 drivers/infiniband/hw/bnxt_re/main.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +3 -9
index b1dcb6b..63ca600 100644
@@ -1880,12 +1880,14 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type)
 		rdev->nb.notifier_call = NULL;
 		pr_err("%s: Cannot register to netdevice_notifier",
 		       ROCE_DRV_MODULE_NAME);
-		return rc;
+		goto re_dev_unreg;
 	}
 	bnxt_re_setup_cc(rdev, true);
 
 	return 0;
 
+re_dev_unreg:
+	ib_unregister_device(&rdev->ibdev);
 re_dev_uninit:
 	bnxt_re_update_en_info_rdev(NULL, en_info, adev);
 	bnxt_re_dev_uninit(rdev, BNXT_RE_COMPLETE_REMOVE);
@@ -2029,15 +2031,7 @@ static int bnxt_re_probe(struct auxiliary_device *adev,
 	auxiliary_set_drvdata(adev, en_info);
 
 	rc = bnxt_re_add_device(adev, BNXT_RE_COMPLETE_INIT);
-	if (rc)
-		goto err;
 	mutex_unlock(&bnxt_re_mutex);
-	return 0;
-
-err:
-	mutex_unlock(&bnxt_re_mutex);
-	bnxt_re_remove(adev);
-
 	return rc;
 }
 
Signature · 1 line
2.5.5
author maintainer
#8 lore raw

PATCH 8/10 RDMA/bnxt_re: Change the sequence of updating the CQ toggle value

In reply to Selvin Xavier

From: Chandramohan Akula <[email protected]>

Currently the CQ toggle value in the shared page (read by the userlib) is updated as part of the cqn_handler. There is a potential race of application calling the CQ ARM doorbell immediately and using the old toggle value.

Change the sequence of updating CQ toggle value to update in the bnxt_qplib_service_nq function immediately after reading the toggle value to be in sync with the HW updated value.

Fixes e275919d9669 ("RDMA/bnxt_re: Share a page to expose per CQ info with userspace")
---
 drivers/infiniband/hw/bnxt_re/main.c     | 8 +-------
 drivers/infiniband/hw/bnxt_re/qplib_fp.c | 5 +++++
 2 files changed, 6 insertions(+), 7 deletions(-)
drivers/infiniband/hw/bnxt_re/main.c +1 -7
index 63ca600..6715c96 100644
@@ -1274,15 +1274,9 @@ static int bnxt_re_cqn_handler(struct bnxt_qplib_nq *nq,
 {
 	struct bnxt_re_cq *cq = container_of(handle, struct bnxt_re_cq,
 					     qplib_cq);
-	u32 *cq_ptr;
 
-	if (cq->ib_cq.comp_handler) {
-		if (cq->uctx_cq_page) {
-			cq_ptr = (u32 *)cq->uctx_cq_page;
-			*cq_ptr = cq->qplib_cq.toggle;
-		}
+	if (cq->ib_cq.comp_handler)
 		(*cq->ib_cq.comp_handler)(&cq->ib_cq, cq->ib_cq.cq_context);
-	}
 
 	return 0;
 }
drivers/infiniband/hw/bnxt_re/qplib_fp.c +5 -0
index 42e98e5..2ebcb2d 100644
@@ -327,6 +327,7 @@ static void bnxt_qplib_service_nq(struct tasklet_struct *t)
 		case NQ_BASE_TYPE_CQ_NOTIFICATION:
 		{
 			struct nq_cn *nqcne = (struct nq_cn *)nqe;
+			struct bnxt_re_cq *cq_p;
 
 			q_handle = le32_to_cpu(nqcne->cq_handle_low);
 			q_handle |= (u64)le32_to_cpu(nqcne->cq_handle_high)
@@ -337,6 +338,10 @@ static void bnxt_qplib_service_nq(struct tasklet_struct *t)
 			cq->toggle = (le16_to_cpu(nqe->info10_type) &
 					NQ_CN_TOGGLE_MASK) >> NQ_CN_TOGGLE_SFT;
 			cq->dbinfo.toggle = cq->toggle;
+			cq_p = container_of(cq, struct bnxt_re_cq, qplib_cq);
+			if (cq_p->uctx_cq_page)
+				*((u32 *)cq_p->uctx_cq_page) = cq->toggle;
+
 			bnxt_qplib_armen_db(&cq->dbinfo,
 					    DBC_DBC_TYPE_CQ_ARMENA);
 			spin_lock_bh(&cq->compl_lock);
Signature · 1 line
2.5.5
author maintainer
#9 lore raw

PATCH 9/10 RDMA/bnxt_re: Fix a bug while setting up Level-2 PBL pages

In reply to Selvin Xavier

From: Bhargava Chenna Marreddy <[email protected]>

Avoid memory corruption while setting up Level-2 PBL pages for the non MR resources when num_pages > 256K.

There will be a single PDE page address (contiguous pages in the case of > PAGE_SIZE), but, current logic assumes multiple pages, leading to invalid memory access after 256K PBL entries in the PDE.

Fixes 0c4dcd602817 ("RDMA/bnxt_re: Refactor hardware queue memory allocation")
---
 drivers/infiniband/hw/bnxt_re/qplib_res.c | 19 +++----------------
 1 file changed, 3 insertions(+), 16 deletions(-)
drivers/infiniband/hw/bnxt_re/qplib_res.c +3 -16
index 1fdffd6..96ceec1 100644
@@ -257,22 +257,9 @@ int bnxt_qplib_alloc_init_hwq(struct bnxt_qplib_hwq *hwq,
 			dst_virt_ptr =
 				(dma_addr_t **)hwq->pbl[PBL_LVL_0].pg_arr;
 			src_phys_ptr = hwq->pbl[PBL_LVL_1].pg_map_arr;
-			if (hwq_attr->type == HWQ_TYPE_MR) {
-			/* For MR it is expected that we supply only 1 contigous
-			 * page i.e only 1 entry in the PDL that will contain
-			 * all the PBLs for the user supplied memory region
-			 */
-				for (i = 0; i < hwq->pbl[PBL_LVL_1].pg_count;
-				     i++)
-					dst_virt_ptr[0][i] = src_phys_ptr[i] |
-						flag;
-			} else {
-				for (i = 0; i < hwq->pbl[PBL_LVL_1].pg_count;
-				     i++)
-					dst_virt_ptr[PTR_PG(i)][PTR_IDX(i)] =
-						src_phys_ptr[i] |
-						PTU_PDE_VALID;
-			}
+			for (i = 0; i < hwq->pbl[PBL_LVL_1].pg_count; i++)
+				dst_virt_ptr[0][i] = src_phys_ptr[i] | flag;
+
 			/* Alloc or init PTEs */
 			rc = __alloc_pbl(res, &hwq->pbl[PBL_LVL_2],
 					 hwq_attr->sginfo);
Signature · 1 line
2.5.5
author maintainer
#10 lore raw

PATCH 10/10 RDMA/bnxt_re: Fix the GID table length

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

GID table length is reported by FW. The gid index which is passed to the driver during modify_qp/create_ah is restricted by the sgid_index field of struct ib_global_route. sgid_index is u8 and the max sgid possible is 256.

Each GID entry in HW will have 2 GID entries in the kernel gid table. So we can support twice the gid table size reported by FW. Also, restrict the max GID to 256 also.

Fixes 847b97887ed4 ("RDMA/bnxt_re: Restrict the max_gids to 256")
---
 drivers/infiniband/hw/bnxt_re/qplib_sp.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/infiniband/hw/bnxt_re/qplib_sp.c +8 -1
index 32c1cc7..e29fbbd 100644
@@ -159,7 +159,14 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
 	if (!bnxt_qplib_is_chip_gen_p7(rcfw->res->cctx))
 		attr->l2_db_size = (sb->l2_db_space_size + 1) *
 				    (0x01 << RCFW_DBR_BASE_PAGE_SHIFT);
-	attr->max_sgid = BNXT_QPLIB_NUM_GIDS_SUPPORTED;
+	/*
+	 * Read the max gid supported by HW.
+	 * For each entry in HW  GID in HW table, we consume 2
+	 * GID entries in the kernel GID table.  So max_gid reported
+	 * to stack can be up to twice the value reported by the HW, up to 256 gids.
+	 */
+	attr->max_sgid = le32_to_cpu(sb->max_gid);
+	attr->max_sgid = min_t(u32, BNXT_QPLIB_NUM_GIDS_SUPPORTED, 2 * attr->max_sgid);
 	attr->dev_cap_flags = le16_to_cpu(sb->dev_cap_flags);
 	attr->dev_cap_flags2 = le16_to_cpu(sb->dev_cap_ext_flags_2);
 
Signature · 1 line
2.5.5
maintainer Applied
#11 lore raw
In reply to Selvin Xavier
Quoted 21 lines · Selvin Xavier

Few bugfixes for bnxt_re driver. Please review and apply.

Thanks,
Selvin Xavier

Abhishek Mohapatra (1):
RDMA/bnxt_re: Fix the max CQ WQEs for older adapters

Bhargava Chenna Marreddy (1):
RDMA/bnxt_re: Fix a bug while setting up Level-2 PBL pages

Chandramohan Akula (1):
RDMA/bnxt_re: Change the sequence of updating the CQ toggle value

Kalesh AP (5):
RDMA/bnxt_re: Fix out of bound check
RDMA/ bnxt_re: Return more meaningful error RDMA/bnxt_re: Fix a possible NULL pointer dereference RDMA/bnxt_re: Fix an error path in bnxt_re_add_device RDMA/bnxt_re: Fix the GID table length

Quoted 5 lines

Kashyap Desai (1):
RDMA/bnxt_re: Fix incorrect dereference of srq in async event

Selvin Xavier (1):
RDMA/bnxt_re: Avoid CPU lockups due fifo occupancy check loop

Applied to for-rc, thanks

Jason