RDMA/bnxt_re: Refactor Notification queue allocation
Subsystems (2)
Selvin Xavier
maintainercover RDMA/bnxt_re: Refactor Notification queue allocation
Includes some generic improvments and code refactoring for the Notification Queue handling in the driver. Remove the data structures that store the NQ information out of the device structure. Fix few issues in selecting the NQ during CQ create. Also, fail the driver load if NIC driver can not allocate at least two NQs for RoCE.
Please review and apply.
Thanks,
Selvin Xavier
Kalesh AP (5):
RDMA/bnxt_re: Fail probe early when not enough MSI-x vectors are
reserved
RDMA/bnxt_re: Refactor NQ allocation
RDMA/bnxt_re: Refurbish CQ to NQ hash calculation RDMA/bnxt_re: Cache MSIx info to a local structure RDMA/bnxt_re: Add new function to setup NQs
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 20 ++- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 37 +++-- drivers/infiniband/hw/bnxt_re/main.c | 267 +++++++++++++++++++------------ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 1 + drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 5 files changed, 212 insertions(+), 114 deletions(-)
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 1/5 RDMA/bnxt_re: Fail probe early when not enough MSI-x vectors are reserved
From: Kalesh AP <[email protected]>
L2 driver allocates and populates the MSI-x vector details for RoCE in the en_dev structure. RoCE driver requires minimum 2 MSIx vectors. Hence during probe, driver has to check and bail out if there are not enough MSI-x vectors reserved for it before proceeding further initialization.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/main.c | 21 +++++++++++---------- 2 files changed, 13 insertions(+), 10 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
@@ -154,6 +154,8 @@ struct bnxt_re_pacing {#define BNXT_RE_GRC_FIFO_REG_BASE 0x2000 +#define BNXT_RE_MIN_MSIX 2 +#define MAX_CQ_HASH_BITS (16)#define MAX_SRQ_HASH_BITS (16)
drivers/infiniband/hw/bnxt_re/main.c +11 -10
@@ -1936,6 +1936,17 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) }set_bit (BNXT_RE_FLAG_NETDEV_REGISTERED , &rdev->flags); +if (rdev->en_dev->ulp_tbl->msix_requested <BNXT_RE_MIN_MSIX ) { +ibdev_err (&rdev->ibdev, +"RoCE requires minimum 2 MSI-X vectors, but only %d reserved\n" , + rdev->en_dev->ulp_tbl->msix_requested); +clear_bit (BNXT_RE_FLAG_NETDEV_REGISTERED , &rdev->flags); +return -EINVAL ; + } +ibdev_dbg (&rdev->ibdev,"Got %d MSI-X vectors\n" , + rdev->en_dev->ulp_tbl->msix_requested); + rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; + rc =bnxt_re_setup_chip_ctx (rdev);if (rc) {bnxt_unregister_dev (rdev->en_dev);
@@ -1947,16 +1958,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)/* Check whether VF or PF */ bnxt_re_get_sriov_func_type (rdev); -if (!rdev->en_dev->ulp_tbl->msix_requested) { -ibdev_err (&rdev->ibdev, -"Failed to get MSI-X vectors: %#x\n" , rc); - rc = -EINVAL ; - goto fail; - } -ibdev_dbg (&rdev->ibdev,"Got %d MSI-X vectors\n" , - rdev->en_dev->ulp_tbl->msix_requested); - rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; -bnxt_re_query_hwrm_intf_version (rdev);/* Establish RCFW Communication Channel to initialize the context
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 2/5 RDMA/bnxt_re: Refactor NQ allocation
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
Move NQ related data structures from rdev to a new structure named "struct bnxt_re_nq_record" by keeping a pointer to in the rdev structure. Allocate the memory for it dynamically. This change is needed for subsequent patches in the series.
Also, removed the nq_task variable from rdev structure as it is redundant and no longer used.
This change would help to reduce the size of the driver private structure as well.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 13 +++--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 6 +-- drivers/infiniband/hw/bnxt_re/main.c | 74 +++++++++++++++++++++----------- 3 files changed, 60 insertions(+), 33 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +7 -6
@@ -155,6 +155,11 @@ struct bnxt_re_pacing {#define BNXT_RE_GRC_FIFO_REG_BASE 0x2000#define BNXT_RE_MIN_MSIX 2 +#define BNXT_RE_MAX_MSIX BNXT_MAX_ROCE_MSIX +struct bnxt_re_nq_record { +struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX ]; +int num_msix; +};#define MAX_CQ_HASH_BITS (16)#define MAX_SRQ_HASH_BITS (16)
@@ -183,21 +188,17 @@ struct bnxt_re_dev {unsigned int version, major, minor;struct bnxt_qplib_chip_ctx *chip_ctx;struct bnxt_en_dev *en_dev; -int num_msix;int id;struct delayed_work worker;u8 cur_prio_map; -/* FP Notification Queue (CQ & SRQ) */ -struct tasklet_struct nq_task; -/* RCFW Channel */ struct bnxt_qplib_rcfw rcfw; -/* NQ */ -struct bnxt_qplib_nq nq[BNXT_MAX_ROCE_MSIX ]; +/* NQ record */ +struct bnxt_re_nq_record *nqr;/* Device Resources */ struct bnxt_qplib_dev_attr dev_attr;
drivers/infiniband/hw/bnxt_re/ib_verbs.c +3 -3
@@ -1872,8 +1872,8 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq, srq->qplib_srq.wqe_size =bnxt_re_get_rwqe_size (dev_attr->max_srq_sges); srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit; srq->srq_limit = srq_init_attr->attr.srq_limit; - srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0 ].ring_id; - nq = &rdev->nq[0 ]; + srq->qplib_srq.eventq_hw_ring_id = rdev->nqr->nq[0 ].ring_id; + nq = &rdev->nqr->nq[0 ];if (udata) { rc =bnxt_re_init_user_srq (rdev, pd, srq, udata);
@@ -3122,7 +3122,7 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,* used for getting the NQ index. */ nq_alloc_cnt =atomic_inc_return (&rdev->nq_alloc_cnt); - nq = &rdev->nq[nq_alloc_cnt % (rdev->num_msix -1 )]; + nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix -1 )]; cq->qplib_cq.max_wqe = entries; cq->qplib_cq.cnq_hw_ring_id = nq->ring_id; cq->qplib_cq.nq = nq;
drivers/infiniband/hw/bnxt_re/main.c +50 -24
181 lines not on this page · raw message
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 3/5 RDMA/bnxt_re: Refurbish CQ to NQ hash calculation
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
There are few use cases where CQ create and destroy is seen before re-creating the CQ, this kind of use case is disturbing the RR distribution and all the active CQ getting mapped to only 2 NQ alternatively.
Fixing the CQ to NQ hash calculation by implementing a quick load sorting mechanism under a mutex.
Using this, if the CQ was allocated and destroyed before using it, the nq selecting algorithm still obtains the least loaded CQ. Thus balancing the load on NQs.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 33 +++++++++++++++++++++++++------- drivers/infiniband/hw/bnxt_re/main.c | 2 ++ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 1 + drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 5 files changed, 32 insertions(+), 7 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
@@ -159,6 +159,8 @@ struct bnxt_re_pacing {struct bnxt_re_nq_record {struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX ];int num_msix; +/* serialize NQ access */ +struct mutex load_lock; };#define MAX_CQ_HASH_BITS (16)
drivers/infiniband/hw/bnxt_re/ib_verbs.c +26 -7
@@ -3029,6 +3029,28 @@ int bnxt_re_post_recv(struct ib_qp *ib_qp, const struct ib_recv_wr *wr,return rc; } +static struct bnxt_qplib_nq *bnxt_re_get_nq (struct bnxt_re_dev *rdev) +{ +int min, indx; + +mutex_lock (&rdev->nqr->load_lock); +for (indx =0 , min =0 ; indx < (rdev->nqr->num_msix -1 ); indx++) { +if (rdev->nqr->nq[min].load > rdev->nqr->nq[indx].load) + min = indx; + } + rdev->nqr->nq[min].load++; +mutex_unlock (&rdev->nqr->load_lock); + +return &rdev->nqr->nq[min]; +} + +static void bnxt_re_put_nq (struct bnxt_re_dev *rdev,struct bnxt_qplib_nq *nq) +{ +mutex_lock (&rdev->nqr->load_lock); + nq->load--; +mutex_unlock (&rdev->nqr->load_lock); +} +/* Completion Queues */ int bnxt_re_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) {
@@ -3047,6 +3069,8 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)hash_del (&cq->hash_entry); }bnxt_qplib_destroy_cq (&rdev->qplib_res, &cq->qplib_cq); + +bnxt_re_put_nq (rdev, nq);ib_umem_release (cq->umem);atomic_dec (&rdev->stats.res.cq_count);
@@ -3065,8 +3089,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;struct bnxt_qplib_chip_ctx *cctx; -struct bnxt_qplib_nq *nq =NULL ; -unsigned int nq_alloc_cnt;int cqe = attr->cqe;int rc, entries;u32 active_cqs;
@@ -3121,12 +3143,10 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,* Allocating the NQ in a round robin fashion. nq_alloc_cnt is a * used for getting the NQ index. */ - nq_alloc_cnt =atomic_inc_return (&rdev->nq_alloc_cnt); - nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix -1 )]; cq->qplib_cq.max_wqe = entries; - cq->qplib_cq.cnq_hw_ring_id = nq->ring_id; - cq->qplib_cq.nq = nq; cq->qplib_cq.coalescing = &rdev->cq_coalescing; + cq->qplib_cq.nq =bnxt_re_get_nq (rdev); + cq->qplib_cq.cnq_hw_ring_id = cq->qplib_cq.nq->ring_id; rc =bnxt_qplib_create_cq (&rdev->qplib_res, &cq->qplib_cq);if (rc) {
@@ -3136,7 +3156,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, cq->ib_cq.cqe = entries; cq->cq_period = cq->qplib_cq.period; - nq->budget++; active_cqs =atomic_inc_return (&rdev->stats.res.cq_count);if (active_cqs > rdev->stats.res.cq_watermark)
drivers/infiniband/hw/bnxt_re/main.c +2 -0
@@ -1566,6 +1566,8 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev)bnxt_qplib_init_res (&rdev->qplib_res); +mutex_init (&rdev->nqr->load_lock); +for (i =1 ; i < rdev->nqr->num_msix ; i++) { db_offt = rdev->en_dev->msix_entries[i].db_offset; rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1],
drivers/infiniband/hw/bnxt_re/qplib_fp.c +1 -0
@@ -551,6 +551,7 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, nq->pdev = pdev; nq->cqn_handler = cqn_handler; nq->srqn_handler = srqn_handler; + nq->load =0 ;/* Have a task to schedule CQ notifiers in post send case */ nq->cqn_wq =create_singlethread_workqueue ("bnxt_qplib_nq" );
drivers/infiniband/hw/bnxt_re/qplib_fp.h +1 -0
@@ -519,6 +519,7 @@ struct bnxt_qplib_nq {struct tasklet_struct nq_tasklet;bool requested;int budget; +u32 load;cqn_handler_t cqn_handler;srqn_handler_t srqn_handler;
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 4/5 RDMA/bnxt_re: Cache MSIx info to a local structure
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
L2 driver allocates the vectors for RoCE and pass it through the en_dev structure to RoCE. During probe, cache the MSIx related info to a local structure.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 + drivers/infiniband/hw/bnxt_re/main.c | 18 ++++++++++-------- 2 files changed, 11 insertions(+), 8 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +1 -0
@@ -157,6 +157,7 @@ struct bnxt_re_pacing {#define BNXT_RE_MIN_MSIX 2#define BNXT_RE_MAX_MSIX BNXT_MAX_ROCE_MSIXstruct bnxt_re_nq_record { +struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX ];struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX ];int num_msix;/* serialize NQ access */
drivers/infiniband/hw/bnxt_re/main.c +10 -8
@@ -347,7 +347,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)return ; rdev = en_info->rdev; - msix_ent = rdev->en_dev->msix_entries; + msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw;if (!ent) {/* Not setting the f/w timeout bit in rcfw.
@@ -363,7 +363,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)* in device sctructure. */ for (indx =0 ; indx < rdev->nqr->num_msix; indx++) - rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; + rdev->nqr->msix_entries[indx].vector = ent[indx].vector; rc =bnxt_qplib_rcfw_start_irq (rcfw, msix_ent[BNXT_RE_AEQ_IDX ].vector, false);
@@ -1569,9 +1569,9 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev)mutex_init (&rdev->nqr->load_lock);for (i =1 ; i < rdev->nqr->num_msix ; i++) { - db_offt = rdev->en_dev->msix_entries[i].db_offset; + db_offt = rdev->nqr->msix_entries[i].db_offset; rc =bnxt_qplib_enable_nq (rdev->en_dev->pdev, &rdev->nqr->nq[i -1 ], - i -1 , rdev->en_dev->msix_entries[i].vector, + i -1 , rdev->nqr->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler);if (rc) {
@@ -1658,7 +1658,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) rattr.type = type; rattr.mode =RING_ALLOC_REQ_INT_MODE_MSIX ; rattr.depth =BNXT_QPLIB_NQE_MAX_CNT -1 ; - rattr.lrid = rdev->en_dev->msix_entries[i +1 ].ring_idx; + rattr.lrid = rdev->nqr->msix_entries[i +1 ].ring_idx; rc =bnxt_re_net_ring_alloc (rdev, &rattr, &nq->ring_id);if (rc) { ibdev_err(&rdev->ibdev,
@@ -1982,6 +1982,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)return rc; } rdev->nqr->num_msix = rdev->en_dev->ulp_tbl->msix_requested; +memcpy (rdev->nqr->msix_entries, rdev->en_dev->msix_entries, +sizeof (struct bnxt_msix_entry ) * rdev->nqr->num_msix);/* Check whether VF or PF */ bnxt_re_get_sriov_func_type (rdev);
@@ -2007,14 +2009,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) rattr.type = type; rattr.mode =RING_ALLOC_REQ_INT_MODE_MSIX ; rattr.depth =BNXT_QPLIB_CREQE_MAX_CNT -1 ; - rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX ].ring_idx; + rattr.lrid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX ].ring_idx; rc =bnxt_re_net_ring_alloc (rdev, &rattr, &creq->ring_id);if (rc) {ibdev_err (&rdev->ibdev,"Failed to allocate CREQ: %#x\n" , rc); goto free_rcfw; } - db_offt = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX ].db_offset; - vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX ].vector; + db_offt = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX ].db_offset; + vid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX ].vector; rc =bnxt_qplib_enable_rcfw_channel (&rdev->rcfw, vid, db_offt, &bnxt_re_aeq_handler);
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 5/5 RDMA/bnxt_re: Add new function to setup NQs
In reply to Selvin XavierFrom: Kalesh AP <[email protected]>
Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside bnxt_re_stop_irq() and bnxt_re_start_irq() to avoid possible race conditions.
--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 + drivers/infiniband/hw/bnxt_re/main.c | 204 +++++++++++++++++++------------- 2 files changed, 123 insertions(+), 83 deletions(-)
drivers/infiniband/hw/bnxt_re/bnxt_re.h +2 -0
@@ -160,6 +160,7 @@ struct bnxt_re_nq_record {struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX ];struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX ];int num_msix; +int max_init;/* serialize NQ access */ struct mutex load_lock; };
@@ -178,6 +179,7 @@ struct bnxt_re_dev {struct list_head list;unsigned long flags;#define BNXT_RE_FLAG_NETDEV_REGISTERED 0 +#define BNXT_RE_FLAG_SETUP_NQ 1#define BNXT_RE_FLAG_HAVE_L2_REF 3#define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4#define BNXT_RE_FLAG_QOS_WORK_REG 5
drivers/infiniband/hw/bnxt_re/main.c +121 -83
335 lines not on this page · raw message
Signature · 1 line
2.5.5
Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs
Quoted 14 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside
bnxt_re_stop_irq()andbnxt_re_start_irq()to avoid possible race conditions.Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 + drivers/infiniband/hw/bnxt_re/main.c | 204 +++++++++++++++++++------------- 2 files changed, 123 insertions(+), 83 deletions(-)
<...>
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4
-0+rtnl_lock (); +if (test_and_clear_bit (BNXT_RE_FLAG_SETUP_NQ , &rdev->flags)) +bnxt_re_clean_nqs (rdev); +rtnl_unlock ();
<...>
drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934
Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0+rtnl_lock ();bnxt_qplib_free_ctx (&rdev->qplib_res, &rdev->qplib_ctx);bnxt_qplib_disable_rcfw_channel (&rdev->rcfw); type =bnxt_qplib_get_ring_type (rdev->chip_ctx);bnxt_re_net_ring_free (rdev, rdev->rcfw.creq.ring_id, type); +rtnl_unlock ();
Please don't add rtnl_lock() to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.
Please work to remove existing rtnl_lock() from bnxt_re_update_en_info_rdev() too. IMHO that lock is not needed after your driver conversion to auxbus.
Thanks
Selvin Xavier
maintainerRe: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs
+Michael Chan
Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside
bnxt_re_stop_irq()andbnxt_re_start_irq()to avoid possible race conditions.Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 + drivers/infiniband/hw/bnxt_re/main.c | 204 +++++++++++++++++++------------- 2 files changed, 123 insertions(+), 83 deletions(-)<...>
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4
-0+rtnl_lock (); +if (test_and_clear_bit (BNXT_RE_FLAG_SETUP_NQ , &rdev->flags)) +bnxt_re_clean_nqs (rdev); +rtnl_unlock ();<...>
drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934
Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0+rtnl_lock ();bnxt_qplib_free_ctx (&rdev->qplib_res, &rdev->qplib_ctx);bnxt_qplib_disable_rcfw_channel (&rdev->rcfw); type =bnxt_qplib_get_ring_type (rdev->chip_ctx);bnxt_re_net_ring_free (rdev, rdev->rcfw.creq.ring_id, type); +rtnl_unlock ();
Quoted 3 lines
Please don't add
rtnl_lock()to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.
The main reason for bnxt_re to take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.
Quoted 3 lines
Please work to remove existing
rtnl_lock()frombnxt_re_update_en_info_rdev()too. IMHO that lock is not needed after your driver conversion to auxbus.
This check is also to synchronize between the irq_stop and restart implementation betweenbnxt_en and bnxt_re driver and roce driver unload.
We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.
Quoted 2 lines
Thanks
Re: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs
Quoted 40 lines · Selvin Xavier
+Michael Chan
Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside
bnxt_re_stop_irq()andbnxt_re_start_irq()to avoid possible race conditions.Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 + drivers/infiniband/hw/bnxt_re/main.c | 204 +++++++++++++++++++------------- 2 files changed, 123 insertions(+), 83 deletions(-)<...>
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4
-0+rtnl_lock (); +if (test_and_clear_bit (BNXT_RE_FLAG_SETUP_NQ , &rdev->flags)) +bnxt_re_clean_nqs (rdev); +rtnl_unlock ();<...>
drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934
Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0+rtnl_lock ();bnxt_qplib_free_ctx (&rdev->qplib_res, &rdev->qplib_ctx);bnxt_qplib_disable_rcfw_channel (&rdev->rcfw); type =bnxt_qplib_get_ring_type (rdev->chip_ctx);bnxt_re_net_ring_free (rdev, rdev->rcfw.creq.ring_id, type); +rtnl_unlock ();Quoted 3 lines
Please don't add
rtnl_lock()to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.
Quoted 6 lines
The main reason for
bnxt_reto take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.
rtnl_lock is a big kernel lock, which blocks almost everything in the netdev. In your case, you are changing one device configuration and should use your per-device locks. Even in the system with more than one BNXT device, the MSI-X on one device will influence other "innocent" devices.
Quoted 10 lines
Quoted 3 lines
Please work to remove existing
rtnl_lock()frombnxt_re_update_en_info_rdev()too. IMHO that lock is not needed after your driver conversion to auxbus.This check is also to synchronize between the
irq_stopand restart implementation betweenbnxt_enandbnxt_redriver and roce driver unload.We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.
You are adding new rtnl_lock and not moving it from one place to another, so this redesign should be done together with this new feature.
Thanks
Quoted 2 lines
Quoted 2 lines
Thanks
Selvin Xavier
maintainerRe: [rdma-next 5/5] RDMA/bnxt_re: Add new function to setup NQs
Quoted 61 lines · Leon Romanovsky
Quoted 40 lines · Selvin Xavier
+Michael Chan
Quoted 34 lines · Leon Romanovsky
Quoted 14 lines · Selvin Xavier
From: Kalesh AP <[email protected]>
Move the logic to setup and enable NQs to a new function. Similarly moved the NQ cleanup logic to a common function. Introdued a flag to keep track of NQ allocation status and added sanity checks inside
bnxt_re_stop_irq()andbnxt_re_start_irq()to avoid possible race conditions.Signed-off-by Kalesh AP <[email protected]>Signed-off-by Selvin Xavier <[email protected]>--- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 + drivers/infiniband/hw/bnxt_re/main.c | 204 +++++++++++++++++++------------- 2 files changed, 123 insertions(+), 83 deletions(-)<...>
Quoted 5 lines
drivers/infiniband/hw/bnxt_re/main.c +4
-0+rtnl_lock (); +if (test_and_clear_bit (BNXT_RE_FLAG_SETUP_NQ , &rdev->flags)) +bnxt_re_clean_nqs (rdev); +rtnl_unlock ();<...>
drivers/infiniband/hw/bnxt_re/main.c static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · line 1934
Quoted 6 lines
drivers/infiniband/hw/bnxt_re/main.c +2
-0+rtnl_lock ();bnxt_qplib_free_ctx (&rdev->qplib_res, &rdev->qplib_ctx);bnxt_qplib_disable_rcfw_channel (&rdev->rcfw); type =bnxt_qplib_get_ring_type (rdev->chip_ctx);bnxt_re_net_ring_free (rdev, rdev->rcfw.creq.ring_id, type); +rtnl_unlock ();Quoted 3 lines
Please don't add
rtnl_lock()to drivers in RDMA subsystem. BNXT driver is managed through netdev and it is there all proper locking should be done.Quoted 6 lines
The main reason for
bnxt_reto take the rtnl is because of the MSIx resource configuration.
This is because the NIC driver is dynamically modifying the MSIx table when the number
of ring change or ndo->open/close is invoked. So we stop and restart the interrupts of RoCE also with rtnl held.
rtnl_lockis a big kernel lock, which blocks almost everything in the netdev. In your case, you are changing one device configuration and should use your per-device locks. Even in the system with more than one BNXT device, the MSI-X on one device will influence other "innocent" devices.Quoted 7 lines
Quoted 3 lines
Please work to remove existing
rtnl_lock()frombnxt_re_update_en_info_rdev()too. IMHO that lock is not needed after your driver conversion to auxbus.This check is also to synchronize between the
irq_stopand restart implementation betweenbnxt_enandbnxt_redriver and roce driver unload.
Quoted 7 lines
Quoted 3 lines
We will review this locking and see if we can handle it. But it is a
major design change in both L2
and roce drivers.You are adding new
rtnl_lockand not moving it from one place to another, so this redesign should be done together with this new feature.
Ok. Will drop this patch and post a v2.
Quoted 7 lines
Thanks
Quoted 2 lines
Quoted 2 lines
Thanks