Skip to contentSkip to conversation

RDMA/bnxt_re: Refactor Notification queue allocation

PATCH Superseded Selvin Xavier 2 participants 10 messages Nov 8, 2024 – Nov 13, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Selvin Xavier

author maintainer
#0 lore raw

cover RDMA/bnxt_re: Refactor Notification queue allocation

Includes some generic improvments and code refactoring for the Notification Queue handling in the driver. Remove the data structures that store the NQ information out of the device structure. Fix few issues in selecting the NQ during CQ create. Also, fail the driver load if NIC driver can not allocate at least two NQs for RoCE.

Please review and apply.

Thanks,
Selvin Xavier

Kalesh AP (5):
RDMA/bnxt_re: Fail probe early when not enough MSI-x vectors are
reserved
RDMA/bnxt_re: Refactor NQ allocation
RDMA/bnxt_re: Refurbish CQ to NQ hash calculation RDMA/bnxt_re: Cache MSIx info to a local structure RDMA/bnxt_re: Add new function to setup NQs

 drivers/infiniband/hw/bnxt_re/bnxt_re.h  |  20 ++-
 drivers/infiniband/hw/bnxt_re/ib_verbs.c |  37 +++--
 drivers/infiniband/hw/bnxt_re/main.c     | 267 +++++++++++++++++++------------
 drivers/infiniband/hw/bnxt_re/qplib_fp.c |   1 +
 drivers/infiniband/hw/bnxt_re/qplib_fp.h |   1 +
 5 files changed, 212 insertions(+), 114 deletions(-)
Signature · 1 line
2.5.5
author maintainer
#1 lore raw

PATCH 1/5 RDMA/bnxt_re: Fail probe early when not enough MSI-x vectors are reserved

From: Kalesh AP <[email protected]>

L2 driver allocates and populates the MSI-x vector details for RoCE in the en_dev structure. RoCE driver requires minimum 2 MSIx vectors. Hence during probe, driver has to check and bail out if there are not enough MSI-x vectors reserved for it before proceeding further initialization.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |  2 ++
 drivers/infiniband/hw/bnxt_re/main.c    | 21 +++++++++++----------
 2 files changed, 13 insertions(+), 10 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
index d1b7c20..7abc37b 100644
@@ -154,6 +154,8 @@ struct bnxt_re_pacing {
 
 #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000
 
+#define BNXT_RE_MIN_MSIX		2
+
 #define MAX_CQ_HASH_BITS		(16)
 #define MAX_SRQ_HASH_BITS		(16)
 
drivers/infiniband/hw/bnxt_re/main.c +11 -10
index cb61941..f9826c4 100644
@@ -1936,6 +1936,17 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
 	}
 	set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags);
 
+	if (rdev->en_dev->ulp_tbl->msix_requested < BNXT_RE_MIN_MSIX) {
+		ibdev_err(&rdev->ibdev,
+			  "RoCE requires minimum 2 MSI-X vectors, but only %d reserved\n",
+			  rdev->en_dev->ulp_tbl->msix_requested);
+		clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags);
+		return -EINVAL;
+	}
+	ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n",
+		  rdev->en_dev->ulp_tbl->msix_requested);
+	rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested;
+
 	rc = bnxt_re_setup_chip_ctx(rdev);
 	if (rc) {
 		bnxt_unregister_dev(rdev->en_dev);
@@ -1947,16 +1958,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
 	/* Check whether VF or PF */
 	bnxt_re_get_sriov_func_type(rdev);
 
-	if (!rdev->en_dev->ulp_tbl->msix_requested) {
-		ibdev_err(&rdev->ibdev,
-			  "Failed to get MSI-X vectors: %#x\n", rc);
-		rc = -EINVAL;
-		goto fail;
-	}
-	ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n",
-		  rdev->en_dev->ulp_tbl->msix_requested);
-	rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested;
-
 	bnxt_re_query_hwrm_intf_version(rdev);
 
 	/* Establish RCFW Communication Channel to initialize the context
Signature · 1 line
2.5.5
author maintainer
#2 lore raw

PATCH 2/5 RDMA/bnxt_re: Refactor NQ allocation

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

Move NQ related data structures from rdev to a new structure named "struct bnxt_re_nq_record" by keeping a pointer to in the rdev structure. Allocate the memory for it dynamically. This change is needed for subsequent patches in the series.

Also, removed the nq_task variable from rdev structure as it is redundant and no longer used.

This change would help to reduce the size of the driver private structure as well.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h  | 13 +++---
 drivers/infiniband/hw/bnxt_re/ib_verbs.c |  6 +--
 drivers/infiniband/hw/bnxt_re/main.c     | 74 +++++++++++++++++++++-----------
 3 files changed, 60 insertions(+), 33 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +7 -6
index 7abc37b..d1c839e 100644
@@ -155,6 +155,11 @@ struct bnxt_re_pacing {
 #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000
 
 #define BNXT_RE_MIN_MSIX		2
+#define BNXT_RE_MAX_MSIX		BNXT_MAX_ROCE_MSIX
+struct bnxt_re_nq_record {
+	struct bnxt_qplib_nq	nq[BNXT_RE_MAX_MSIX];
+	int			num_msix;
+};
 
 #define MAX_CQ_HASH_BITS		(16)
 #define MAX_SRQ_HASH_BITS		(16)
@@ -183,21 +188,17 @@ struct bnxt_re_dev {
 	unsigned int			version, major, minor;
 	struct bnxt_qplib_chip_ctx	*chip_ctx;
 	struct bnxt_en_dev		*en_dev;
-	int				num_msix;
 
 	int				id;
 
 	struct delayed_work		worker;
 	u8				cur_prio_map;
 
-	/* FP Notification Queue (CQ & SRQ) */
-	struct tasklet_struct		nq_task;
-
 	/* RCFW Channel */
 	struct bnxt_qplib_rcfw		rcfw;
 
-	/* NQ */
-	struct bnxt_qplib_nq		nq[BNXT_MAX_ROCE_MSIX];
+	/* NQ record */
+	struct bnxt_re_nq_record	*nqr;
 
 	/* Device Resources */
 	struct bnxt_qplib_dev_attr	dev_attr;
drivers/infiniband/hw/bnxt_re/ib_verbs.c +3 -3
index 9a188cc..a9c32c0 100644
@@ -1872,8 +1872,8 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
 	srq->qplib_srq.wqe_size = bnxt_re_get_rwqe_size(dev_attr->max_srq_sges);
 	srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit;
 	srq->srq_limit = srq_init_attr->attr.srq_limit;
-	srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0].ring_id;
-	nq = &rdev->nq[0];
+	srq->qplib_srq.eventq_hw_ring_id = rdev->nqr->nq[0].ring_id;
+	nq = &rdev->nqr->nq[0];
 
 	if (udata) {
 		rc = bnxt_re_init_user_srq(rdev, pd, srq, udata);
@@ -3122,7 +3122,7 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 	 * used for getting the NQ index.
 	 */
 	nq_alloc_cnt = atomic_inc_return(&rdev->nq_alloc_cnt);
-	nq = &rdev->nq[nq_alloc_cnt % (rdev->num_msix - 1)];
+	nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix - 1)];
 	cq->qplib_cq.max_wqe = entries;
 	cq->qplib_cq.cnq_hw_ring_id = nq->ring_id;
 	cq->qplib_cq.nq	= nq;
drivers/infiniband/hw/bnxt_re/main.c +50 -24

181 lines not on this page · raw message

Signature · 1 line
2.5.5
author maintainer
#3 lore raw

PATCH 3/5 RDMA/bnxt_re: Refurbish CQ to NQ hash calculation

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

There are few use cases where CQ create and destroy is seen before re-creating the CQ, this kind of use case is disturbing the RR distribution and all the active CQ getting mapped to only 2 NQ alternatively.

Fixing the CQ to NQ hash calculation by implementing a quick load sorting mechanism under a mutex.

Using this, if the CQ was allocated and destroyed before using it, the nq selecting algorithm still obtains the least loaded CQ. Thus balancing the load on NQs.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h  |  2 ++
 drivers/infiniband/hw/bnxt_re/ib_verbs.c | 33 +++++++++++++++++++++++++-------
 drivers/infiniband/hw/bnxt_re/main.c     |  2 ++
 drivers/infiniband/hw/bnxt_re/qplib_fp.c |  1 +
 drivers/infiniband/hw/bnxt_re/qplib_fp.h |  1 +
 5 files changed, 32 insertions(+), 7 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
index d1c839e..5f64fc4 100644
@@ -159,6 +159,8 @@ struct bnxt_re_pacing {
 struct bnxt_re_nq_record {
 	struct bnxt_qplib_nq	nq[BNXT_RE_MAX_MSIX];
 	int			num_msix;
+	/* serialize NQ access */
+	struct mutex		load_lock;
 };
 
 #define MAX_CQ_HASH_BITS		(16)
drivers/infiniband/hw/bnxt_re/ib_verbs.c +26 -7
index a9c32c0..9b0a435 100644
@@ -3029,6 +3029,28 @@ int bnxt_re_post_recv(struct ib_qp *ib_qp, const struct ib_recv_wr *wr,
 	return rc;
 }
 
+static struct bnxt_qplib_nq *bnxt_re_get_nq(struct bnxt_re_dev *rdev)
+{
+	int min, indx;
+
+	mutex_lock(&rdev->nqr->load_lock);
+	for (indx = 0, min = 0; indx < (rdev->nqr->num_msix - 1); indx++) {
+		if (rdev->nqr->nq[min].load > rdev->nqr->nq[indx].load)
+			min = indx;
+	}
+	rdev->nqr->nq[min].load++;
+	mutex_unlock(&rdev->nqr->load_lock);
+
+	return &rdev->nqr->nq[min];
+}
+
+static void bnxt_re_put_nq(struct bnxt_re_dev *rdev, struct bnxt_qplib_nq *nq)
+{
+	mutex_lock(&rdev->nqr->load_lock);
+	nq->load--;
+	mutex_unlock(&rdev->nqr->load_lock);
+}
+
 /* Completion Queues */
 int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
 {
@@ -3047,6 +3069,8 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
 		hash_del(&cq->hash_entry);
 	}
 	bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
+
+	bnxt_re_put_nq(rdev, nq);
 	ib_umem_release(cq->umem);
 
 	atomic_dec(&rdev->stats.res.cq_count);
@@ -3065,8 +3089,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 		rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);
 	struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
 	struct bnxt_qplib_chip_ctx *cctx;
-	struct bnxt_qplib_nq *nq = NULL;
-	unsigned int nq_alloc_cnt;
 	int cqe = attr->cqe;
 	int rc, entries;
 	u32 active_cqs;
@@ -3121,12 +3143,10 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 	 * Allocating the NQ in a round robin fashion. nq_alloc_cnt is a
 	 * used for getting the NQ index.
 	 */
-	nq_alloc_cnt = atomic_inc_return(&rdev->nq_alloc_cnt);
-	nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix - 1)];
 	cq->qplib_cq.max_wqe = entries;
-	cq->qplib_cq.cnq_hw_ring_id = nq->ring_id;
-	cq->qplib_cq.nq	= nq;
 	cq->qplib_cq.coalescing = &rdev->cq_coalescing;
+	cq->qplib_cq.nq	= bnxt_re_get_nq(rdev);
+	cq->qplib_cq.cnq_hw_ring_id = cq->qplib_cq.nq->ring_id;
 
 	rc = bnxt_qplib_create_cq(&rdev->qplib_res, &cq->qplib_cq);
 	if (rc) {
@@ -3136,7 +3156,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 
 	cq->ib_cq.cqe = entries;
 	cq->cq_period = cq->qplib_cq.period;
-	nq->budget++;
 
 	active_cqs = atomic_inc_return(&rdev->stats.res.cq_count);
 	if (active_cqs > rdev->stats.res.cq_watermark)
drivers/infiniband/hw/bnxt_re/main.c +2 -0
index 9acc0e2..e556110 100644
@@ -1566,6 +1566,8 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev)
 
 	bnxt_qplib_init_res(&rdev->qplib_res);
 
+	mutex_init(&rdev->nqr->load_lock);
+
 	for (i = 1; i < rdev->nqr->num_msix ; i++) {
 		db_offt = rdev->en_dev->msix_entries[i].db_offset;
 		rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1],
drivers/infiniband/hw/bnxt_re/qplib_fp.c +1 -0
index e2eea71..e56f42f 100644
@@ -551,6 +551,7 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq,
 	nq->pdev = pdev;
 	nq->cqn_handler = cqn_handler;
 	nq->srqn_handler = srqn_handler;
+	nq->load = 0;
 
 	/* Have a task to schedule CQ notifiers in post send case */
 	nq->cqn_wq  = create_singlethread_workqueue("bnxt_qplib_nq");
drivers/infiniband/hw/bnxt_re/qplib_fp.h +1 -0
index b5a90581..8ff56d7 100644
@@ -519,6 +519,7 @@ struct bnxt_qplib_nq {
 	struct tasklet_struct		nq_tasklet;
 	bool				requested;
 	int				budget;
+	u32				load;
 
 	cqn_handler_t			cqn_handler;
 	srqn_handler_t			srqn_handler;
Signature · 1 line
2.5.5
author maintainer
#4 lore raw

PATCH 4/5 RDMA/bnxt_re: Cache MSIx info to a local structure

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

L2 driver allocates the vectors for RoCE and pass it through the en_dev structure to RoCE. During probe, cache the MSIx related info to a local structure.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |  1 +
 drivers/infiniband/hw/bnxt_re/main.c    | 18 ++++++++++--------
 2 files changed, 11 insertions(+), 8 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +1 -0
index 5f64fc4..2975b11 100644
@@ -157,6 +157,7 @@ struct bnxt_re_pacing {
 #define BNXT_RE_MIN_MSIX		2
 #define BNXT_RE_MAX_MSIX		BNXT_MAX_ROCE_MSIX
 struct bnxt_re_nq_record {
+	struct bnxt_msix_entry	msix_entries[BNXT_RE_MAX_MSIX];
 	struct bnxt_qplib_nq	nq[BNXT_RE_MAX_MSIX];
 	int			num_msix;
 	/* serialize NQ access */
drivers/infiniband/hw/bnxt_re/main.c +10 -8
index e556110..1c7171a 100644
@@ -347,7 +347,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
 		return;
 
 	rdev = en_info->rdev;
-	msix_ent = rdev->en_dev->msix_entries;
+	msix_ent = rdev->nqr->msix_entries;
 	rcfw = &rdev->rcfw;
 	if (!ent) {
 		/* Not setting the f/w timeout bit in rcfw.
@@ -363,7 +363,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
 	 * in device sctructure.
 	 */
 	for (indx = 0; indx < rdev->nqr->num_msix; indx++)
-		rdev->en_dev->msix_entries[indx].vector = ent[indx].vector;
+		rdev->nqr->msix_entries[indx].vector = ent[indx].vector;
 
 	rc = bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector,
 				       false);
@@ -1569,9 +1569,9 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev)
 	mutex_init(&rdev->nqr->load_lock);
 
 	for (i = 1; i < rdev->nqr->num_msix ; i++) {
-		db_offt = rdev->en_dev->msix_entries[i].db_offset;
+		db_offt = rdev->nqr->msix_entries[i].db_offset;
 		rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1],
-					  i - 1, rdev->en_dev->msix_entries[i].vector,
+					  i - 1, rdev->nqr->msix_entries[i].vector,
 					  db_offt, &bnxt_re_cqn_handler,
 					  &bnxt_re_srqn_handler);
 		if (rc) {
@@ -1658,7 +1658,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev)
 		rattr.type = type;
 		rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX;
 		rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1;
-		rattr.lrid = rdev->en_dev->msix_entries[i + 1].ring_idx;
+		rattr.lrid = rdev->nqr->msix_entries[i + 1].ring_idx;
 		rc = bnxt_re_net_ring_alloc(rdev, &rattr, &nq->ring_id);
 		if (rc) {
 			ibdev_err(&rdev->ibdev,
@@ -1982,6 +1982,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
 		return rc;
 	}
 	rdev->nqr->num_msix = rdev->en_dev->ulp_tbl->msix_requested;
+	memcpy(rdev->nqr->msix_entries, rdev->en_dev->msix_entries,
+	       sizeof(struct bnxt_msix_entry) * rdev->nqr->num_msix);
 
 	/* Check whether VF or PF */
 	bnxt_re_get_sriov_func_type(rdev);
@@ -2007,14 +2009,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
 	rattr.type = type;
 	rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX;
 	rattr.depth = BNXT_QPLIB_CREQE_MAX_CNT - 1;
-	rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx;
+	rattr.lrid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].ring_idx;
 	rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id);
 	if (rc) {
 		ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc);
 		goto free_rcfw;
 	}
-	db_offt = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].db_offset;
-	vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector;
+	db_offt = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].db_offset;
+	vid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].vector;
 	rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw,
 					    vid, db_offt,
 					    &bnxt_re_aeq_handler);
Signature · 1 line
2.5.5
author maintainer
#5 lore raw

PATCH 5/5 RDMA/bnxt_re: Add new function to setup NQs

In reply to Selvin Xavier

From: Kalesh AP <[email protected]>

Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/main.c    | 204 +++++++++++++++++++-------------
 2 files changed, 123 insertions(+), 83 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
index 2975b11..74a340f 100644
@@ -160,6 +160,7 @@ struct bnxt_re_nq_record {
 	struct bnxt_msix_entry	msix_entries[BNXT_RE_MAX_MSIX];
 	struct bnxt_qplib_nq	nq[BNXT_RE_MAX_MSIX];
 	int			num_msix;
+	int			max_init;
 	/* serialize NQ access */
 	struct mutex		load_lock;
 };
@@ -178,6 +179,7 @@ struct bnxt_re_dev {
 	struct list_head		list;
 	unsigned long			flags;
 #define BNXT_RE_FLAG_NETDEV_REGISTERED		0
+#define BNXT_RE_FLAG_SETUP_NQ			1
 #define BNXT_RE_FLAG_HAVE_L2_REF		3
 #define BNXT_RE_FLAG_RCFW_CHANNEL_EN		4
 #define BNXT_RE_FLAG_QOS_WORK_REG		5
drivers/infiniband/hw/bnxt_re/main.c +121 -83

335 lines not on this page · raw message

Signature · 1 line
2.5.5
#6 lore raw

Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs

Quoted 14 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/main.c    | 204 +++++++++++++++++++-------------
 2 files changed, 123 insertions(+), 83 deletions(-)

<...>

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4 -0
+	rtnl_lock();
+	if (test_and_clear_bit(BNXT_RE_FLAG_SETUP_NQ, &rdev->flags))
+		bnxt_re_clean_nqs(rdev);
+	rtnl_unlock();

<...>

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934

Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
+		rtnl_lock();
 		bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
 		bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
 		type = bnxt_qplib_get_ring_type(rdev->chip_ctx);
 		bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type);
+		rtnl_unlock();

Please don't add rtnl_lock() to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.

Please work to remove existing rtnl_lock() from bnxt_re_update_en_info_rdev() too. IMHO that lock is not needed after your driver conversion to auxbus.

Thanks

author maintainer
#7 lore raw

Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs

+Michael Chan

Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/main.c    | 204 +++++++++++++++++++-------------
 2 files changed, 123 insertions(+), 83 deletions(-)

<...>

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4 -0
+     rtnl_lock();
+     if (test_and_clear_bit(BNXT_RE_FLAG_SETUP_NQ, &rdev->flags))
+             bnxt_re_clean_nqs(rdev);
+     rtnl_unlock();

<...>

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934

Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
+             rtnl_lock();
              bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
              bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
              type = bnxt_qplib_get_ring_type(rdev->chip_ctx);
              bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type);
+             rtnl_unlock();
Quoted 3 lines

Please don't add rtnl_lock() to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.

The main reason for bnxt_re to take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.

Quoted 3 lines

Please work to remove existing rtnl_lock() from bnxt_re_update_en_info_rdev() too. IMHO that lock is not needed after your driver conversion to auxbus.

This check is also to synchronize between the irq_stop and restart implementation between
bnxt_en and bnxt_re driver and roce driver unload.

We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.

Quoted 2 lines

Thanks

#8 lore raw

Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs

Quoted 40 lines · Selvin Xavier

+Michael Chan

Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/main.c    | 204 +++++++++++++++++++-------------
 2 files changed, 123 insertions(+), 83 deletions(-)

<...>

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4 -0
+     rtnl_lock();
+     if (test_and_clear_bit(BNXT_RE_FLAG_SETUP_NQ, &rdev->flags))
+             bnxt_re_clean_nqs(rdev);
+     rtnl_unlock();

<...>

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934

Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
+             rtnl_lock();
              bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
              bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
              type = bnxt_qplib_get_ring_type(rdev->chip_ctx);
              bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type);
+             rtnl_unlock();
Quoted 3 lines

Please don't add rtnl_lock() to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.

Quoted 6 lines

The main reason for bnxt_re to take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.

rtnl_lock is a big kernel lock, which blocks almost everything in the netdev. In your case, you are changing one device configuration and should use your per-device locks. Even in the system with more than one BNXT device, the MSI-X on one device will influence other "innocent" devices.

Quoted 10 lines
Quoted 3 lines

Please work to remove existing rtnl_lock() from bnxt_re_update_en_info_rdev() too. IMHO that lock is not needed after your driver conversion to auxbus.

This check is also to synchronize between the irq_stop and restart implementation between
bnxt_en and bnxt_re driver and roce driver unload.

We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.

You are adding new rtnl_lock and not moving it from one place to another, so this redesign should be done together with this new feature.

Thanks

Quoted 2 lines
Quoted 2 lines

Thanks

author maintainer
#9 lore raw

Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs

Quoted 61 lines · Leon Romanovsky
Quoted 40 lines · Selvin Xavier

+Michael Chan

Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier

From: Kalesh AP <[email protected]>

Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.

---
 drivers/infiniband/hw/bnxt_re/bnxt_re.h |   2 +
 drivers/infiniband/hw/bnxt_re/main.c    | 204 +++++++++++++++++++-------------
 2 files changed, 123 insertions(+), 83 deletions(-)

<...>

Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4 -0
+     rtnl_lock();
+     if (test_and_clear_bit(BNXT_RE_FLAG_SETUP_NQ, &rdev->flags))
+             bnxt_re_clean_nqs(rdev);
+     rtnl_unlock();

<...>

drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934

Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2 -0
+             rtnl_lock();
              bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
              bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
              type = bnxt_qplib_get_ring_type(rdev->chip_ctx);
              bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type);
+             rtnl_unlock();
Quoted 3 lines

Please don't add rtnl_lock() to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.

Quoted 6 lines

The main reason for bnxt_re to take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.

rtnl_lock is a big kernel lock, which blocks almost everything in the netdev. In your case, you are changing one device configuration and should use your per-device locks. Even in the system with more than one BNXT device, the MSI-X on one device will influence other "innocent" devices.

Quoted 7 lines
Quoted 3 lines

Please work to remove existing rtnl_lock() from bnxt_re_update_en_info_rdev() too. IMHO that lock is not needed after your driver conversion to auxbus.

This check is also to synchronize between the irq_stop and restart implementation between
bnxt_en and bnxt_re driver and roce driver unload.

Quoted 7 lines
Quoted 3 lines

We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.

You are adding new rtnl_lock and not moving it from one place to another, so this redesign should be done together with this new feature.

Ok. Will drop this patch and post a v2.

Quoted 7 lines

Thanks

Quoted 2 lines
Quoted 2 lines

Thanks