Skip to contentSkip to conversation

RDMA/rxe: don't re-execute the current packet after the QP moved to error state

PATCH Applied Allison Henderson 3 participants 3 messages Jul 11 – Jul 12

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/rxe: don't re-execute the current packet after the QP moved to error state

When do_complete() finds the QP in the error state it returns RESPST_CHK_RESOURCE. Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c") this was the flush loop:
check_resource() had an error-state branch that fetched each remaining recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching the current packet. That commit removed the error-state branch from check_resource() (draining is now done at rxe_receiver() entry) but kept the do_complete() error-state return.

As a result, when a QP moves to the error state while a packet is being completed - e.g. an rdma_cm disconnect racing with receive processing - the responder state machine loops back into the request processing chain with the already-completed packet still in hand:
check_resource() fetches a fresh recv WQE, execute()/send_data_in() copies the same packet payload again, do_complete() posts another IB_WC_SUCCESS CQE (qp->resp.status is still 0), and control returns to the error-state check. The loop re-executes the same packet once per posted recv WQE (observed: ~1000 duplicate IB_WC_SUCCESS completions of one SEND, one per ~8us, matching the RQ occupancy) until the RQ is exhausted, after which qp->resp.wqe is NULL and send_data_in() dereferences it:

  BUG: kernel NULL pointer dereference, address: 0000000000000014
  Workqueue: rxe_wq do_work
  RIP: copy_data+0x29/0x1f0
  Call Trace:
   send_data_in+0x25/0x50
   rxe_receiver+0xf36/0x1dd0

The duplicate completions are indistinguishable from real receives to the ULP. During an rds stress test, the message was accepted as new and delivered the same datagram to user space hundreds of times, corrupting the stream; any ULP that relies on RC exactly-once delivery is affected.

A live packet reaching the error-state check in do_complete() has been executed and completed exactly once and must be consumed, not re-processed. Return RESPST_CLEANUP for it (dequeue and free); keep returning RESPST_CHK_RESOURCE for the pkt == NULL case.

Fixes 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")
Assisted-by Claude-Code:claude-fable-5
---
 drivers/infiniband/sw/rxe/rxe_resp.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_resp.c +8 -1
index d8cbdfa70cdbd..02b16e2b49b8f 100644
@@ -1217,7 +1217,14 @@ static enum resp_states do_complete(struct rxe_qp *qp,
 	spin_lock_irqsave(&qp->state_lock, flags);
 	if (unlikely(qp_state(qp) == IB_QPS_ERR)) {
 		spin_unlock_irqrestore(&qp->state_lock, flags);
-		return RESPST_CHK_RESOURCE;
+		/* The packet was executed and completed before the QP
+		 * moved to ERROR; it must be consumed exactly once.
+		 * Re-entering the request chain with the stale packet
+		 * would copy it into every remaining recv WQE as a new
+		 * completion.  Remaining WQEs are flushed by the drain
+		 * path at rxe_receiver() entry.
+		 */
+		return pkt ? RESPST_CLEANUP : RESPST_CHK_RESOURCE;
 	}
 	spin_unlock_irqrestore(&qp->state_lock, flags);
 
Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 34 lines · Allison Henderson

When do_complete() finds the QP in the error state it returns RESPST_CHK_RESOURCE. Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c") this was the flush loop:
check_resource() had an error-state branch that fetched each remaining recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching the current packet. That commit removed the error-state branch from check_resource() (draining is now done at rxe_receiver() entry) but kept the do_complete() error-state return.

As a result, when a QP moves to the error state while a packet is being completed - e.g. an rdma_cm disconnect racing with receive processing - the responder state machine loops back into the request processing chain with the already-completed packet still in hand:
check_resource() fetches a fresh recv WQE, execute()/send_data_in() copies the same packet payload again, do_complete() posts another IB_WC_SUCCESS CQE (qp->resp.status is still 0), and control returns to the error-state check. The loop re-executes the same packet once per posted recv WQE (observed: ~1000 duplicate IB_WC_SUCCESS completions of one SEND, one per ~8us, matching the RQ occupancy) until the RQ is exhausted, after which qp->resp.wqe is NULL and send_data_in() dereferences it:

   BUG: kernel NULL pointer dereference, address: 0000000000000014
   Workqueue: rxe_wq do_work
   RIP: copy_data+0x29/0x1f0
   Call Trace:
    send_data_in+0x25/0x50
    rxe_receiver+0xf36/0x1dd0

The duplicate completions are indistinguishable from real receives to the ULP. During an rds stress test, the message was accepted as new and delivered the same datagram to user space hundreds of times, corrupting the stream; any ULP that relies on RC exactly-once delivery is affected.

Quoted 8 lines

A live packet reaching the error-state check in do_complete() has been executed and completed exactly once and must be consumed, not re-processed. Return RESPST_CLEANUP for it (dequeue and free); keep returning RESPST_CHK_RESOURCE for the pkt == NULL case.

Fixes 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")
Assisted-by Claude-Code:claude-fable-5

The fix is correct and strictly adheres to the InfiniBand RC service guarantees. When a QP transitions to the error state while processing a live packet, returning RESPST_CLEANUP ensures the executed packet is properly consumed rather than improperly re-injected into the request chain. This accurately prevents both data stream corruption from duplicate completions and the kernel panic caused by RQ exhaustion.

Thanks a lot.

Zhu Yanjun

Quoted 24 lines
---
  drivers/infiniband/sw/rxe/rxe_resp.c | 9 ++++++++-
  1 file changed, 8 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_resp.c +8 -1
index d8cbdfa70cdbd..02b16e2b49b8f 100644
@@ -1217,7 +1217,14 @@ static enum resp_states do_complete(struct rxe_qp *qp,
  	spin_lock_irqsave(&qp->state_lock, flags);
  	if (unlikely(qp_state(qp) == IB_QPS_ERR)) {
  		spin_unlock_irqrestore(&qp->state_lock, flags);
-		return RESPST_CHK_RESOURCE;
+		/* The packet was executed and completed before the QP
+		 * moved to ERROR; it must be consumed exactly once.
+		 * Re-entering the request chain with the stale packet
+		 * would copy it into every remaining recv WQE as a new
+		 * completion.  Remaining WQEs are flushed by the drain
+		 * path at rxe_receiver() entry.
+		 */
+		return pkt ? RESPST_CLEANUP : RESPST_CHK_RESOURCE;
  	}
  	spin_unlock_irqrestore(&qp->state_lock, flags);
 
#2 lore raw
In reply to Allison Henderson
Quoted 10 lines · Allison Henderson

When do_complete() finds the QP in the error state it returns RESPST_CHK_RESOURCE. Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c") this was the flush loop:
check_resource() had an error-state branch that fetched each remaining recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching the current packet. That commit removed the error-state branch from check_resource() (draining is now done at rxe_receiver() entry) but kept the do_complete() error-state return.

[...]

Applied, thanks!

[1/1] RDMA/rxe: don't re-execute the current packet after the QP moved to error state

      https://git.kernel.org/rdma/rdma/c/15ae32c4a3551c

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>