RDMA/bnxt_re: Bug fixes
Subsystems (2)
Selvin Xavier
maintainercover RDMA/bnxt_re: Bug fixes
Includes two fixes for issues seen during load/unload stress tests.
Kashyap Desai (1):
RDMA/bnxt_re: Prevent handling any completions after qp destroy
Selvin Xavier (1):
bnxt_re: Fix hang during driver unload
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 12 ++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 28 +++++++++++++++++++++++----- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 9 ++++----- 4 files changed, 40 insertions(+), 10 deletions(-)
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 1/2 RDMA/bnxt_re: Prevent handling any completions after qp destroy
From: Kashyap Desai <[email protected]>
HW may generate completions that indicates QP is destroyed. Driver should not be scheduling any more completion handlers for this QP, after the QP is destroyed. Since CQs are active during the QP destroy, driver may still schedule completion handlers. This can cause a race where the destroy_cq and poll_cq running simultaneously.
Snippet of kernel panic while doing bnxt_re driver load unload in loop. This indicates a poll after the CQ is freed.
[77786.481636] Call Trace: [77786.481640] <TASK> [77786.481644] bnxt_re_poll_cq+0x14a/0x620 [bnxt_re] [77786.481658] ? kvm_clock_read+0x14/0x30 [77786.481693] __ib_process_cq+0x57/0x190 [ib_core] [77786.481728] ib_cq_poll_work+0x26/0x80 [ib_core] [77786.481761] process_one_work+0x1e5/0x3f0 [77786.481768] worker_thread+0x50/0x3a0 [77786.481785] ? __pfx_worker_thread+0x10/0x10 [77786.481790] kthread+0xe2/0x110 [77786.481794] ? __pfx_kthread+0x10/0x10 [77786.481797] ret_from_fork+0x2c/0x50
To avoid this, complete all completion handlers before returning the destroy QP. If free_cq is called soon after destroy_qp, IB stack will cancel the CQ work before invoking the destroy_cq verb and this will prevent any race mentioned.
--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 12 ++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 18 ++++++++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 3 files changed, 31 insertions(+)
drivers/infiniband/hw/bnxt_re/ib_verbs.c +12 -0
@@ -869,7 +869,10 @@ static int bnxt_re_destroy_gsi_sqp(struct bnxt_re_qp *qp)int bnxt_re_destroy_qp (struct ib_qp *ib_qp,struct ib_udata *udata) { struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); +struct bnxt_qplib_qp *qplib_qp = &qp->qplib_qp;struct bnxt_re_dev *rdev = qp->rdev; +struct bnxt_qplib_nq *scq_nq =NULL ; +struct bnxt_qplib_nq *rcq_nq =NULL ;unsigned int flags;int rc;
@@ -903,6 +906,15 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)ib_umem_release (qp->rumem);ib_umem_release (qp->sumem); +/* Flush all the entries of notification queue associated with +* given qp. +*/ + scq_nq = qplib_qp->scq->nq; + rcq_nq = qplib_qp->rcq->nq; +bnxt_re_synchronize_nq (scq_nq); +if (scq_nq != rcq_nq) +bnxt_re_synchronize_nq (rcq_nq); +return 0 ; }
drivers/infiniband/hw/bnxt_re/qplib_fp.c +18 -0
@@ -381,6 +381,24 @@ static void bnxt_qplib_service_nq(struct tasklet_struct *t)spin_unlock_bh (&hwq->lock); } +/* bnxt_re_synchronize_nq - self polling notification queue. +* @nq - notification queue pointer +* +* This function will start polling entries of a given notification queue +* for all pending entries. +* This function is useful to synchronize notification entries while resources +* are going away. +*/ + +void bnxt_re_synchronize_nq (struct bnxt_qplib_nq *nq) +{ +int budget = nq->budget; + + nq->budget = nq->hwq.max_elements; +bnxt_qplib_service_nq (&nq->nq_tasklet); + nq->budget = budget; +} +static irqreturn_t bnxt_qplib_nq_irq (int irq,void *dev_instance) {struct bnxt_qplib_nq *nq = dev_instance;
drivers/infiniband/hw/bnxt_re/qplib_fp.h +1 -0
@@ -553,6 +553,7 @@ int bnxt_qplib_process_flush_list(struct bnxt_qplib_cq *cq,struct bnxt_qplib_cqe *cqe, int num_cqes);void bnxt_qplib_flush_cqn_wq (struct bnxt_qplib_qp *qp); +void bnxt_re_synchronize_nq (struct bnxt_qplib_nq *nq);static inline void *bnxt_qplib_get_swqe (struct bnxt_qplib_q *que,u32 *swq_idx) {
Signature · 1 line
2.5.5
Selvin Xavier
maintainerPATCH 2/2 bnxt_re: Fix hang during driver unload
In reply to Selvin XavierDriver unload hits a hang during stress testing of load/unload.
stack trace snippet -
tasklet_kill at ffffffff9aabb8b2 bnxt_qplib_nq_stop_irq at ffffffffc0a805fb [bnxt_re] bnxt_qplib_disable_nq at ffffffffc0a80c5b [bnxt_re] bnxt_re_dev_uninit at ffffffffc0a67d15 [bnxt_re] bnxt_re_remove_device at ffffffffc0a6af1d [bnxt_re]
tasklet_kill can hang if the tasklet is scheduled after it is disabled.
Modified the sequences to disable the interrupt first and synchronize irq before disabling the tasklet.
--- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 10 +++++----- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 9 ++++----- 2 files changed, 9 insertions(+), 10 deletions(-)
drivers/infiniband/hw/bnxt_re/qplib_fp.c +5 -5
@@ -420,19 +420,19 @@ void bnxt_qplib_nq_stop_irq(struct bnxt_qplib_nq *nq, bool kill)if (!nq->requested)return ; -tasklet_disable (&nq->nq_tasklet); + nq->requested = false;/* Mask h/w interrupt */ bnxt_qplib_ring_nq_db (&nq->nq_db.dbinfo, nq->res->cctx, false);/* Sync with last running IRQ handler */ synchronize_irq (nq->msix_vec); -if (kill) -tasklet_kill (&nq->nq_tasklet); -irq_set_affinity_hint (nq->msix_vec,NULL );free_irq (nq->msix_vec, nq);kfree (nq->name); nq->name =NULL ; - nq->requested = false; + +if (kill) +tasklet_kill (&nq->nq_tasklet); +tasklet_disable (&nq->nq_tasklet); }void bnxt_qplib_disable_nq (struct bnxt_qplib_nq *nq)
drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +4 -5
@@ -989,19 +989,18 @@ void bnxt_qplib_rcfw_stop_irq(struct bnxt_qplib_rcfw *rcfw, bool kill)if (!creq->requested)return ; -tasklet_disable (&creq->creq_tasklet); + creq->requested = false;/* Mask h/w interrupts */ bnxt_qplib_ring_nq_db (&creq->creq_db.dbinfo, rcfw->res->cctx, false);/* Sync with last running IRQ-handler */ synchronize_irq (creq->msix_vec); -if (kill) -tasklet_kill (&creq->creq_tasklet); -free_irq (creq->msix_vec, rcfw);kfree (creq->irq_name); creq->irq_name =NULL ; - creq->requested = false;atomic_set (&rcfw->rcfw_intr_enabled,0 ); +if (kill) +tasklet_kill (&creq->creq_tasklet); +tasklet_disable (&creq->creq_tasklet); }void bnxt_qplib_disable_rcfw_channel (struct bnxt_qplib_rcfw *rcfw)
Signature · 1 line
2.5.5
Leon Romanovsky
AppliedQuoted 9 lines · Selvin Xavier
Includes two fixes for issues seen during load/unload stress tests.
Kashyap Desai (1):
RDMA/bnxt_re: Prevent handling any completions after qp destroySelvin Xavier (1):
bnxt_re: Fix hang during driver unload[...]
Applied, thanks!
[1/2] RDMA/bnxt_re: Prevent handling any completions after qp destroy
https://git.kernel.org/rdma/rdma/c/ab939c8a3a1675 [2/2] bnxt_re: Fix hang during driver unload https://git.kernel.org/rdma/rdma/c/2c03afae5ff9b3
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>