RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
Subsystems (3)
PATCH 1/1 RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
rxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and reallocates qp->resp.resources[] (the rd_atomic resource array used by the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests) completely outside of the IB_QP_STATE handling above it. Unlike every other place that tears this array down -- rxe_qp_reset(), reached only under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) / rxe_disable_task(&qp->send_task) to drain the responder and requester tasks before touching per-QP state, then re-enables them -- this branch runs with the responder task (rxe_receiver(), scheduled as recv_task on the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp() that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so __qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is never taken here) can therefore race the responder in two ways:
1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and alloc_rd_atomic_resources() kzalloc_objs()'s a new array while rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently walking &qp->resp.resources[i] with no lock held -- a straight free-vs-read race on the array itself.
2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never clears qp->resp.res, the raw pointer *into* that array that rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared only on the normal completion paths). If a modify_qp() races a resource still referenced by qp->resp.res, the array is freed out from under the cached pointer and the next reply packet dereferences it -- independent of the kfree/kzalloc_objs() window in (1).
Reproduced with KASAN: a single process driving one RC QP pair in rxe loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs against qpB while a second thread hammers ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]
with the freed kmalloc-1k object being the rd_atomic resource array freed by the modify_qp() thread while the recv_task kworker reads it. An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free) is clean.
Fix both races the same way rxe_qp_reset() already handles tearing down this exact array: quiesce the responder task around the free/realloc by calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources() and rxe_enable_task(&qp->recv_task) after alloc_rd_atomic_resources(), so rxe_receiver() cannot observe the array mid-free/mid-realloc. And close the still-open window for (2) at the source: have free_rd_atomic_resources() clear qp->resp.res along with qp->resp.resources, exactly like the existing completion paths in rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do when a resource's lifetime ends, so a drained-and-resumed responder restarts at RESPST_CHK_PSN against the fresh array instead of replaying a stale reference into the old one.
Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are touched exclusively by the responder (rxe_resp.c); the requester (send_task / rxe_sender()) never reads them, so there is no need to widen this beyond what rxe_qp_reset() would drain for the equivalent state.
Verified on the same v6.19 KASAN stand: with this fix applied, the identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC storms against qpB well past the ~11s pre-fix time-to-first-splat with zero KASAN reports, versus reliably tripping the slab-use-after-free in rxe_receiver() described above before the fix.
--- drivers/infiniband/sw/rxe/rxe_qp.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_qp.c +8 -1
@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) }kfree (qp->resp.resources); qp->resp.resources =NULL ; + qp->resp.res =NULL ; } }
@@ -709,9 +710,15 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, qp->attr.max_dest_rd_atomic = max_dest_rd_atomic; +/* +* Not gated by IB_QP_STATE above: quiesce the responder task +* the same way rxe_qp_reset() does before touching this +* array, so rxe_receiver() can't race the free/realloc. +*/ +rxe_disable_task (&qp->recv_task);free_rd_atomic_resources (qp); - err =alloc_rd_atomic_resources (qp, max_dest_rd_atomic); +rxe_enable_task (&qp->recv_task);if (err)return err; }
Signature · 1 line
2.50.1 (Apple Git-155)
Quoted 29 lines · Ibrahim Hashimov
rxe_qp_from_attr()'s
IB_QP_MAX_DEST_RD_ATOMICbranch frees and reallocates qp->resp.resources[] (therd_atomicresource array used by the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests) completely outside of theIB_QP_STATEhandling above it. Unlike every other place that tears this array down --rxe_qp_reset(), reached only underIB_QP_STATE, always callsrxe_disable_task(&qp->recv_task)/rxe_disable_task(&qp->send_task)to drain the responder and requester tasks before touching per-QP state, then re-enables them -- this branch runs with the responder task (rxe_receiver(), scheduled asrecv_taskon therxe_wqworkqueue) fully live and unlocked. A userspacemodify_qp()that sets onlyIB_QP_MAX_DEST_RD_ATOMIC(no state change, so __qp_chk_state()/ib_modify_qp_is_ok() never runs andqp->state_lockis never taken here) can therefore race the responder in two ways:1.
free_rd_atomic_resources()callskfree(qp->resp.resources)andalloc_rd_atomic_resources()kzalloc_objs()'s a new array while rxe_prepare_res()/find_resource() inrxe_resp.care concurrently walking &qp->resp.resources[i] with no lock held -- a straight free-vs-read race on the array itself.2.
free_rd_atomic_resources()only NULLsqp->resp.resources; it never clearsqp->resp.res, the raw pointer *into* that array thatrxe_resp.ccaches across a multi-packet RDMA READ/ATOMIC/FLUSH reply (set atrxe_resp.cread/atomic/flush-reply sites, cleared only on the normal completion paths). If amodify_qp()races a resource still referenced byqp->resp.res, the array is freed out from under the cached pointer and the next reply packet dereferences it -- independent of the kfree/kzalloc_objs() window in (1).
Quoted 7 lines
Reproduced with KASAN: a single process driving one RC QP pair in rxe loopback, one thread pumping large multi-packet
IBV_WR_RDMA_READsagainst qpB while a second thread hammersibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) producesBUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]
Can you share all the stack trace with us? Thanks a lot.
Quoted 55 lines
with the freed kmalloc-1k object being the
rd_atomicresource array freed by themodify_qp()thread while therecv_taskkworker reads it. An identical run modifying onlyIB_QP_MIN_RNR_TIMER(no resource free) is clean.Fix both races the same way
rxe_qp_reset()already handles tearing down this exact array: quiesce the responder task around the free/realloc by callingrxe_disable_task(&qp->recv_task)beforefree_rd_atomic_resources()andrxe_enable_task(&qp->recv_task)afteralloc_rd_atomic_resources(), sorxe_receiver()cannot observe the array mid-free/mid-realloc. And close the still-open window for (2) at the source: havefree_rd_atomic_resources()clearqp->resp.resalong withqp->resp.resources, exactly like the existing completion paths inrxe_resp.c(check_rkey()/duplicate_request()/RESPST_CLEANUP) already do when a resource's lifetime ends, so a drained-and-resumed responder restarts atRESPST_CHK_PSNagainst the fresh array instead of replaying a stale reference into the old one.Only
qp->recv_taskis drained:qp->resp.resources/qp->resp.resare touched exclusively by the responder (rxe_resp.c); the requester (send_task/rxe_sender()) never reads them, so there is no need to widen this beyond whatrxe_qp_reset()would drain for the equivalent state.Verified on the same v6.19 KASAN stand: with this fix applied, the identical differential reproducer drives sustained
MAX_DEST_RD_ATOMICstorms against qpB well past the ~11s pre-fix time-to-first-splat with zero KASAN reports, versus reliably tripping the slab-use-after-free inrxe_receiver()described above before the fix.Fixes 8700e3e7c485 ("Soft RoCE driver")Signed-off-by Ibrahim Hashimov <[email protected]>Assisted-by AuditCode-AI:2026.07--- drivers/infiniband/sw/rxe/rxe_qp.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_qp.c +1
-0@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) }kfree (qp->resp.resources); qp->resp.resources =NULL ; + qp->resp.res =NULL ; } }@@ -709,9 +710,15 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
drivers/infiniband/sw/rxe/rxe_qp.c int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, · line 721
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_qp.c +7
-1+/* +* Not gated by IB_QP_STATE above: quiesce the responder task +* the same way rxe_qp_reset() does before touching this +* array, so rxe_receiver() can't race the free/realloc. +*/ +rxe_disable_task (&qp->recv_task);free_rd_atomic_resources (qp); - err =alloc_rd_atomic_resources (qp, max_dest_rd_atomic); +rxe_enable_task (&qp->recv_task);
If alloc_rd_atomic_resources fails, that is, qp->resp.resources is NULL.
After rxe_enable_task(&qp->recv_task); qp->resp.resources(NULL) will be used in resp. This will cause problems.
drivers/infiniband/sw/rxe/rxe_resp.c:656: res = &qp->resp.resources[qp->resp.res_head];drivers/infiniband/sw/rxe/rxe_resp.c:1325: struct resp_res *res = &qp->resp.resources[i];
Zhu Yanjun
Quoted 3 lines
drivers/infiniband/sw/rxe/rxe_qp.c +0
-0if (err)return err; }
Hi Zhu Yanjun, thanks for the review.
Quoted 3 lines
Quoted 2 lines
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]Can you share all the stack trace with us? Thanks a lot.
Sure. The read side -- the recv_task kworker running rxe_receiver() on the freed rd_atomic resource array -- from the pre-fix v6.19 kernel (CONFIG_KASAN generic, rxe loopback):
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Read of size 4 at addr ffff88800b79f84c by task kworker/u8:2/51 CPU: 0 UID: 0 PID: 51 Comm: kworker/u8:2 Not tainted 6.19.0 #1 Workqueue: rxe_wq do_work [rdma_rxe] Call Trace: <TASK> dump_stack_lvl+0x4d/0x70 print_report+0x170/0x4f3 kasan_report+0xda/0x110 rxe_receiver+0x4f78/0x89e0 [rdma_rxe] do_work+0x144/0x470 [rdma_rxe] process_one_work+0x611/0xe80 worker_thread+0x52e/0xdc0 kthread+0x30c/0x630 ret_from_fork+0x2fd/0x3e0 ret_from_fork_asm+0x1a/0x30 </TASK> The buggy address belongs to the object at ffff88800b79f800 which belongs to the cache kmalloc-1k of size 1024 The buggy address is located 76 bytes inside of freed 1024-byte region [ffff88800b79f800, ffff88800b79fc00)
The freed kmalloc-1k object is qp->resp.resources[] (the rd_atomic array); it is freed by the concurrent ib_modify_qp(IB_QP_MAX_DEST_RD_ATOMIC) thread in free_rd_atomic_resources() <- rxe_qp_from_attr(), while this recv_task kworker reads it. This was a kasan_multi_shot run and the per-object Allocated-by/Freed-by backtraces were lost in the splat storm -- I can send a single-shot capture that includes those two stacks if it would help.
Quoted 5 lines
If
alloc_rd_atomic_resourcesfails, that is,qp->resp.resourcesis NULL. Afterrxe_enable_task(&qp->recv_task);qp->resp.resources(NULL)will be used in resp. This will cause problems.drivers/infiniband/sw/rxe/rxe_resp.c:656: res = &qp->resp.resources[qp->resp.res_head];drivers/infiniband/sw/rxe/rxe_resp.c:1325: structresp_res*res = &qp->resp.resources[i];
Good catch -- you're right. Re-enabling recv_task before the error check resumes the responder against a NULL qp->resp.resources on the ENOMEM path. v2 moves rxe_enable_task() below the error check, so the responder is re-enabled only once a fresh array has been installed and stays quiesced on failure:
rxe_disable_task(&qp->recv_task); free_rd_atomic_resources(qp); err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic); if (err) return err; rxe_enable_task(&qp->recv_task);
rxe_disable_task()/rxe_enable_task() are state-based (TASK_STATE_DRAINED /IDLE), not refcounted, so leaving recv_task drained on the failed modify is safe and recoverable: a subsequent successful modify_qp or a QP destroy both handle the DRAINED state.
I'll send this as [PATCH v2].
Thanks,
Ibrahim
PATCH v2 1/1 RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
In reply to Ibrahim Hashimovrxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and reallocates qp->resp.resources[] (the rd_atomic resource array used by the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests) completely outside of the IB_QP_STATE handling above it. Unlike every other place that tears this array down -- rxe_qp_reset(), reached only under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) / rxe_disable_task(&qp->send_task) to drain the responder and requester tasks before touching per-QP state, then re-enables them -- this branch runs with the responder task (rxe_receiver(), scheduled as recv_task on the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp() that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so __qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is never taken here) can therefore race the responder in two ways:
1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and alloc_rd_atomic_resources() kzalloc_objs()'s a new array while rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently walking &qp->resp.resources[i] with no lock held -- a straight free-vs-read race on the array itself.
2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never clears qp->resp.res, the raw pointer *into* that array that rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared only on the normal completion paths). If a modify_qp() races a resource still referenced by qp->resp.res, the array is freed out from under the cached pointer and the next reply packet dereferences it -- independent of the kfree/kzalloc_objs() window in (1).
Reproduced with KASAN: a single process driving one RC QP pair in rxe loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs against qpB while a second thread hammers ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]
with the freed kmalloc-1k object being the rd_atomic resource array freed by the modify_qp() thread while the recv_task kworker reads it. An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free) is clean.
Fix both races the same way rxe_qp_reset() already handles tearing down this exact array: quiesce the responder task around the free/realloc by calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources() and rxe_enable_task(&qp->recv_task) only after alloc_rd_atomic_resources() has succeeded, so rxe_receiver() cannot observe the array mid-free/ mid-realloc. On the alloc-failure path the responder is deliberately left quiesced: qp->resp.resources is NULL at that point and rxe_prepare_res()/find_resource() would dereference it, so recv_task must not be re-enabled until a fresh array has been installed. And close the still-open window for (2) at the source: have free_rd_atomic_resources() clear qp->resp.res along with qp->resp.resources, exactly like the existing completion paths in rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do when a resource's lifetime ends, so a drained-and-resumed responder restarts at RESPST_CHK_PSN against the fresh array instead of replaying a stale reference into the old one.
Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are touched exclusively by the responder (rxe_resp.c); the requester (send_task / rxe_sender()) never reads them, so there is no need to widen this beyond what rxe_qp_reset() would drain for the equivalent state.
Verified on the same v6.19 KASAN stand: with this fix applied, the identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC storms against qpB well past the ~11s pre-fix time-to-first-splat with zero KASAN reports, versus reliably tripping the slab-use-after-free in rxe_receiver() described above before the fix.
---
v2: address Zhu Yanjun's review of v1
(https://lore.kernel.org/linux-rdma/[email protected]/):
only re-enable recv_task after alloc_rd_atomic_resources() succeeds, so the responder is not resumed against a NULL qp->resp.resources on the ENOMEM path (rxe_prepare_res()/find_resource() would dereference it). No change to the successful path; fix description updated accordingly.
drivers/infiniband/sw/rxe/rxe_qp.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_qp.c +15 -1
@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) }kfree (qp->resp.resources); qp->resp.resources =NULL ; + qp->resp.res =NULL ; } }
@@ -709,11 +710,24 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, qp->attr.max_dest_rd_atomic = max_dest_rd_atomic; +/* +* This branch is not gated by IB_QP_STATE, so the responder +* task is live here. Quiesce it the way rxe_qp_reset() does +* before swapping the rd_atomic resource array, so +* rxe_receiver() cannot race the free/realloc. +*/ +rxe_disable_task (&qp->recv_task);free_rd_atomic_resources (qp); - err =alloc_rd_atomic_resources (qp, max_dest_rd_atomic); +/* +* On failure the responder stays quiesced: qp->resp.resources +* is NULL now, and rxe_prepare_res()/find_resource() would +* dereference it, so do not re-enable recv_task until a fresh +* array has been installed. +*/ if (err)return err; +rxe_enable_task (&qp->recv_task); }if (mask &IB_QP_EN_SQD_ASYNC_NOTIFY )
Signature · 1 line
2.50.1 (Apple Git-155)
Quoted 111 lines · Ibrahim Hashimov
rxe_qp_from_attr()'s
IB_QP_MAX_DEST_RD_ATOMICbranch frees and reallocates qp->resp.resources[] (therd_atomicresource array used by the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests) completely outside of theIB_QP_STATEhandling above it. Unlike every other place that tears this array down --rxe_qp_reset(), reached only underIB_QP_STATE, always callsrxe_disable_task(&qp->recv_task)/rxe_disable_task(&qp->send_task)to drain the responder and requester tasks before touching per-QP state, then re-enables them -- this branch runs with the responder task (rxe_receiver(), scheduled asrecv_taskon therxe_wqworkqueue) fully live and unlocked. A userspacemodify_qp()that sets onlyIB_QP_MAX_DEST_RD_ATOMIC(no state change, so __qp_chk_state()/ib_modify_qp_is_ok() never runs andqp->state_lockis never taken here) can therefore race the responder in two ways:1.
free_rd_atomic_resources()callskfree(qp->resp.resources)andalloc_rd_atomic_resources()kzalloc_objs()'s a new array while rxe_prepare_res()/find_resource() inrxe_resp.care concurrently walking &qp->resp.resources[i] with no lock held -- a straight free-vs-read race on the array itself.2.
free_rd_atomic_resources()only NULLsqp->resp.resources; it never clearsqp->resp.res, the raw pointer *into* that array thatrxe_resp.ccaches across a multi-packet RDMA READ/ATOMIC/FLUSH reply (set atrxe_resp.cread/atomic/flush-reply sites, cleared only on the normal completion paths). If amodify_qp()races a resource still referenced byqp->resp.res, the array is freed out from under the cached pointer and the next reply packet dereferences it -- independent of the kfree/kzalloc_objs() window in (1).Reproduced with KASAN: a single process driving one RC QP pair in rxe loopback, one thread pumping large multi-packet
IBV_WR_RDMA_READsagainst qpB while a second thread hammersibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) producesBUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]with the freed kmalloc-1k object being the
rd_atomicresource array freed by themodify_qp()thread while therecv_taskkworker reads it. An identical run modifying onlyIB_QP_MIN_RNR_TIMER(no resource free) is clean.Fix both races the same way
rxe_qp_reset()already handles tearing down this exact array: quiesce the responder task around the free/realloc by callingrxe_disable_task(&qp->recv_task)beforefree_rd_atomic_resources()andrxe_enable_task(&qp->recv_task)only afteralloc_rd_atomic_resources()has succeeded, sorxe_receiver()cannot observe the array mid-free/ mid-realloc. On the alloc-failure path the responder is deliberately left quiesced:qp->resp.resourcesis NULL at that point and rxe_prepare_res()/find_resource() would dereference it, sorecv_taskmust not be re-enabled until a fresh array has been installed. And close the still-open window for (2) at the source: havefree_rd_atomic_resources()clearqp->resp.resalong withqp->resp.resources, exactly like the existing completion paths inrxe_resp.c(check_rkey()/duplicate_request()/RESPST_CLEANUP) already do when a resource's lifetime ends, so a drained-and-resumed responder restarts atRESPST_CHK_PSNagainst the fresh array instead of replaying a stale reference into the old one.Only
qp->recv_taskis drained:qp->resp.resources/qp->resp.resare touched exclusively by the responder (rxe_resp.c); the requester (send_task/rxe_sender()) never reads them, so there is no need to widen this beyond whatrxe_qp_reset()would drain for the equivalent state.Verified on the same v6.19 KASAN stand: with this fix applied, the identical differential reproducer drives sustained
MAX_DEST_RD_ATOMICstorms against qpB well past the ~11s pre-fix time-to-first-splat with zero KASAN reports, versus reliably tripping the slab-use-after-free inrxe_receiver()described above before the fix.Fixes 8700e3e7c485 ("Soft RoCE driver")Signed-off-by Ibrahim Hashimov <[email protected]>Assisted-by AuditCode-AI:2026.07---
v2: address Zhu Yanjun's review of v1
(https://lore.kernel.org/linux-rdma/[email protected]/):
only re-enablerecv_taskafteralloc_rd_atomic_resources()succeeds, so the responder is not resumed against a NULLqp->resp.resourceson the ENOMEM path (rxe_prepare_res()/find_resource() would dereference it). No change to the successful path; fix description updated accordingly.drivers/infiniband/sw/rxe/rxe_qp.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_qp.c +8
-1@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) }kfree (qp->resp.resources); qp->resp.resources =NULL ; + qp->resp.res =NULL ; } }@@ -709,11 +710,24 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, qp->attr.max_dest_rd_atomic = max_dest_rd_atomic; +/* +* This branch is not gated by IB_QP_STATE, so the responder +* task is live here. Quiesce it the way rxe_qp_reset() does +* before swapping the rd_atomic resource array, so +* rxe_receiver() cannot race the free/realloc. +*/ +rxe_disable_task (&qp->recv_task);free_rd_atomic_resources (qp); -
drivers/infiniband/sw/rxe/rxe_qp.c int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, · line 730
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_qp.c +7
-0err =alloc_rd_atomic_resources (qp, max_dest_rd_atomic); +/* +* On failure the responder stays quiesced: qp->resp.resources +* is NULL now, and rxe_prepare_res()/find_resource() would +* dereference it, so do not re-enable recv_task until a fresh +* array has been installed. +*/ if (err)return err; +rxe_enable_task (&qp->recv_task);
Thanks a lot. I am fine with this commit. Please Leon and Jason comment on this commit.
Zhu Yanjun
Quoted 3 lines
drivers/infiniband/sw/rxe/rxe_qp.c +0
-0}if (mask &IB_QP_EN_SQD_ASYNC_NOTIFY )
Quoted 114 lines · yanjun.zhu
Quoted 113 lines · Ibrahim Hashimov
rxe_qp_from_attr()'s
IB_QP_MAX_DEST_RD_ATOMICbranch frees and reallocates qp->resp.resources[] (therd_atomicresource array used by the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests) completely outside of theIB_QP_STATEhandling above it. Unlike every other place that tears this array down --rxe_qp_reset(), reached only underIB_QP_STATE, always callsrxe_disable_task(&qp->recv_task)/rxe_disable_task(&qp->send_task)to drain the responder and requester tasks before touching per-QP state, then re-enables them -- this branch runs with the responder task (rxe_receiver(), scheduled asrecv_taskon therxe_wqworkqueue) fully live and unlocked. A userspacemodify_qp()that sets onlyIB_QP_MAX_DEST_RD_ATOMIC(no state change, so __qp_chk_state()/ib_modify_qp_is_ok() never runs andqp->state_lockis never taken here) can therefore race the responder in two ways:1.
free_rd_atomic_resources()callskfree(qp->resp.resources)andalloc_rd_atomic_resources()kzalloc_objs()'s a new array while rxe_prepare_res()/find_resource() inrxe_resp.care concurrently walking &qp->resp.resources[i] with no lock held -- a straight free-vs-read race on the array itself.2.
free_rd_atomic_resources()only NULLsqp->resp.resources; it never clearsqp->resp.res, the raw pointer *into* that array thatrxe_resp.ccaches across a multi-packet RDMA READ/ATOMIC/FLUSH reply (set atrxe_resp.cread/atomic/flush-reply sites, cleared only on the normal completion paths). If amodify_qp()races a resource still referenced byqp->resp.res, the array is freed out from under the cached pointer and the next reply packet dereferences it -- independent of the kfree/kzalloc_objs() window in (1).Reproduced with KASAN: a single process driving one RC QP pair in rxe loopback, one thread pumping large multi-packet
IBV_WR_RDMA_READsagainst qpB while a second thread hammersibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) producesBUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe] Workqueue: rxe_wq do_work [rdma_rxe]with the freed kmalloc-1k object being the
rd_atomicresource array freed by themodify_qp()thread while therecv_taskkworker reads it. An identical run modifying onlyIB_QP_MIN_RNR_TIMER(no resource free) is clean.Fix both races the same way
rxe_qp_reset()already handles tearing down this exact array: quiesce the responder task around the free/realloc by callingrxe_disable_task(&qp->recv_task)beforefree_rd_atomic_resources()andrxe_enable_task(&qp->recv_task)only afteralloc_rd_atomic_resources()has succeeded, sorxe_receiver()cannot observe the array mid-free/ mid-realloc. On the alloc-failure path the responder is deliberately left quiesced:qp->resp.resourcesis NULL at that point and rxe_prepare_res()/find_resource() would dereference it, sorecv_taskmust not be re-enabled until a fresh array has been installed. And close the still-open window for (2) at the source: havefree_rd_atomic_resources()clearqp->resp.resalong withqp->resp.resources, exactly like the existing completion paths inrxe_resp.c(check_rkey()/duplicate_request()/RESPST_CLEANUP) already do when a resource's lifetime ends, so a drained-and-resumed responder restarts atRESPST_CHK_PSNagainst the fresh array instead of replaying a stale reference into the old one.Only
qp->recv_taskis drained:qp->resp.resources/qp->resp.resare touched exclusively by the responder (rxe_resp.c); the requester (send_task/rxe_sender()) never reads them, so there is no need to widen this beyond whatrxe_qp_reset()would drain for the equivalent state.Verified on the same v6.19 KASAN stand: with this fix applied, the identical differential reproducer drives sustained
MAX_DEST_RD_ATOMICstorms against qpB well past the ~11s pre-fix time-to-first-splat with zero KASAN reports, versus reliably tripping the slab-use-after-free inrxe_receiver()described above before the fix.Fixes 8700e3e7c485 ("Soft RoCE driver")Signed-off-by Ibrahim Hashimov <[email protected]>Assisted-by AuditCode-AI:2026.07---
v2: address Zhu Yanjun's review of v1
(https://lore.kernel.org/linux-rdma/[email protected]/):
only re-enablerecv_taskafteralloc_rd_atomic_resources()succeeds, so the responder is not resumed against a NULLqp->resp.resourceson the ENOMEM path (rxe_prepare_res()/find_resource() would dereference it). No change to the successful path; fix description updated accordingly.drivers/infiniband/sw/rxe/rxe_qp.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_qp.c +12
-1@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) }kfree (qp->resp.resources); qp->resp.resources =NULL ; + qp->resp.res =NULL ; } }@@ -709,11 +710,24 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, qp->attr.max_dest_rd_atomic = max_dest_rd_atomic; +/* +* This branch is not gated by IB_QP_STATE, so the responder +* task is live here. Quiesce it the way rxe_qp_reset() does +* before swapping the rd_atomic resource array, so +* rxe_receiver() cannot race the free/realloc. +*/ +rxe_disable_task (&qp->recv_task);free_rd_atomic_resources (qp); - err =alloc_rd_atomic_resources (qp, max_dest_rd_atomic); +/* +* On failure the responder stays quiesced: qp->resp.resources +* is NULL now, and rxe_prepare_res()/find_resource() would +* dereference it, so do not re-enable recv_task until a fresh
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_qp.c int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, · line 730
Quoted 5 lines
drivers/infiniband/sw/rxe/rxe_qp.c +3
-0+* array has been installed. +*/ if (err)return err; +rxe_enable_task (&qp->recv_task);Thanks a lot. I am fine with this commit. Please Leon and Jason comment on this commit.
1. Please send patches as a standalone series, not as a reply to the previous one.
2. Write commit messages in a clear and sensible way so they are easy for humans to understand. Keep them short and direct, and describe the function call flow instead of producing AI prose.
Thanks
Quoted 8 lines
Reviewed-by Zhu Yanjun <[email protected]>Zhu Yanjun
Quoted 2 lines
drivers/infiniband/sw/rxe/rxe_qp.c +0
-0}if (mask &IB_QP_EN_SQD_ASYNC_NOTIFY )