Skip to contentSkip to conversation

RDMA/mlx5: Cancel software CQ notifications before teardown

PATCH Applied Prathamesh Deshpande 3 participants 4 messages Oct 8 – Oct 12

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/mlx5: Cancel software CQ notifications before teardown

GSI completions queue notify_work to invoke the CQ completion handler. CQ destruction drains hardware events and core polling, but does not cancel this software notifier. Queued or running work can therefore access a freed CQ, or restart polling after ib_free_cq() has drained it.

Cancel notify_work in the pre-destroy callback, before core polling is stopped and CQ memory is released. Skip user CQs, which do not initialize this work item.

Fixes 25361e02c448 ("IB/mlx5: Generate completions in software")
---
 drivers/infiniband/hw/mlx5/cq.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/hw/mlx5/cq.c +3 -0
index 49b4bf148a4a..1653b2747f5f 100644
@@ -1117,6 +1117,9 @@ int mlx5_ib_pre_destroy_cq(struct ib_cq *cq)
 	struct mlx5_ib_dev *dev = to_mdev(cq->device);
 	struct mlx5_ib_cq *mcq = to_mcq(cq);
 
+	if (!cq->uobject)
+		cancel_work_sync(&mcq->notify_work);
+
 	return mlx5_core_destroy_cq(dev->mdev, &mcq->mcq);
 }
 
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] rdma/mlx5: missing wc_list drain in mlx5_ib_pre_destroy_cq() causes leak --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] rdma/mlx5: missing wc_list drain in mlx5_ib_pre_destroy_cq() causes leak --

commit bb2bb35d84abe8b8d9f71ac658dfcd1b34acddcb
Author: Prathamesh Deshpande <[email protected]>

RDMA/mlx5: Cancel software CQ notifications before teardown

This commit cancels notify_work in the pre-destroy callback to prevent queued or running work from accessing a freed CQ or restarting polling after the CQ is drained. User CQs are skipped because they do not initialize this work item.

drivers/infiniband/hw/mlx5/cq.c int mlx5_ib_pre_destroy_cq(struct ib_cq *cq) · line 1122

Quoted 11 lines
drivers/infiniband/hw/mlx5/cq.c +3 -0
index 072b77a2c8a6c..7495e20c3dec9 100644
@@ -1117,6 +1117,9 @@ int mlx5_ib_pre_destroy_cq(struct ib_cq *cq)
 	struct mlx5_ib_dev *dev = to_mdev(cq->device);
 	struct mlx5_ib_cq *mcq = to_mcq(cq);
 
+	if (!cq->uobject)
+		cancel_work_sync(&mcq->notify_work);
+

[Severity: Medium]
Does this code cause a memory leak regression by leaving dynamically allocated elements in mcq->wc_list undrained?

When mlx5_ib_generate_wc() generates software completions, it dynamically allocates memory for them and queues them to the wc_list:

mlx5_ib_generate_wc() {
	...
	soft_wc = kmalloc_obj(*soft_wc, GFP_ATOMIC);
	...
	list_add_tail(&soft_wc->list, &cq->wc_list);
	...
}

If the CQ is destroyed before these completions are polled, mlx5_ib_pre_destroy_cq() correctly stops the workqueue to prevent use-after-free, but it fails to drain mcq->wc_list.

Since the work queue is no longer running to consume these elements, they will be leaked when the CQ is freed.

Quoted 2 lines
drivers/infiniband/hw/mlx5/cq.c +0 -0
 	return mlx5_core_destroy_cq(dev->mdev, &mcq->mcq);
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Prathamesh Deshpande
Quoted 9 lines · Prathamesh Deshpande

GSI completions queue notify_work to invoke the CQ completion handler. CQ destruction drains hardware events and core polling, but does not cancel this software notifier. Queued or running work can therefore access a freed CQ, or restart polling after ib_free_cq() has drained it.

Cancel notify_work in the pre-destroy callback, before core polling is stopped and CQ memory is released. Skip user CQs, which do not initialize this work item.

Quoted 5 lines
Fixes 25361e02c448 ("IB/mlx5: Generate completions in software")
---
 drivers/infiniband/hw/mlx5/cq.c | 3 +++
 1 file changed, 3 insertions(+)

I applied the patch because the change is worthwhile, but the description is incorrect. GSI is handled internally by mlx5 CQs and QPs and is stopped before the CQ is destroyed.

Thanks

#3 lore raw
In reply to Prathamesh Deshpande
Quoted 10 lines · Prathamesh Deshpande

GSI completions queue notify_work to invoke the CQ completion handler. CQ destruction drains hardware events and core polling, but does not cancel this software notifier. Queued or running work can therefore access a freed CQ, or restart polling after ib_free_cq() has drained it.

Cancel notify_work in the pre-destroy callback, before core polling is stopped and CQ memory is released. Skip user CQs, which do not initialize this work item.

[...]

Applied, thanks!

[1/1] RDMA/mlx5: Cancel software CQ notifications before teardown

      https://git.kernel.org/rdma/rdma/c/a0ab2bb269e7fd

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>