Skip to content

net/smc: transition to RDMA core CQ pooling

D. Wythe linux-rdma linux-s390 netdev 6 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest3 patches 10 repliesIn review
  2. v3 posting 2/23 patches 1 replySuperseded
  3. v2 posting 2/22 patches 3 repliesSuperseded
Earlier versions · 3
VersionPostedPatchesRepliesStatus
v3 posting 1/21 patch3 repliesSuperseded
v2 posting 1/21 patch2 repliesSuperseded
v11 patch2 repliesSuperseded

Changes in v4 from the cover letter

https://lore.kernel.org/netdev/[email protected]/
1. Reorder: move the exclusive-wait patch before the CQ pooling patch.
2. Fix teardown stall: reintroduce smc_ib_modify_qp_error() and flush the QP in smc_wr_stop_link() before waiting for the refs.
3. smc_wr_reg_send(): take the ref before ib_post_send().
4. smc_wr_tx_process_cqe(): wake_up_all() on a flush completion.
5. Merge the three per-link percpu_refs into a single wr_refs.

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3net/smc: clear the correct v2 slot and buffer in smc_wr_tx_put_slot()net/smc/smc_wr.c+4 -41 reply
2/3net/smc: reduce TX slot contention with exclusive waitnet/smc/smc_wr.c+26 -101 reply
3/3net/smc: transition to RDMA core CQ poolingnet/smc/smc_core.c net/smc/smc_core.h net/smc/smc_ib.c +4 more+257 -3536 replies

Reviews trailers per patch

All 3 patches have a Reviewed-by or Acked-by.

PatchDust LiLeon RomanovskyMahanta JambigiWen Gu
1/3R-b R-b
2/3R-b
3/3R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v4 thread

Cover letter [PATCH net-next v4 0/3] net/smc: transition to RDMA core CQ pooling

This series transitions SMC-R completion handling to RDMA core CQ pooling via the ib_cqe API. The new completion model improves scalability by allowing per-link completion processing across multiple cores and enables DIM-based interrupt moderation.

The increased concurrency can amplify contention for TX slots on the shared wait queue, so the exclusive-wait change comes first as a standalone improvement that the CQ pooling teardown then builds on.

Patch 1 fixes smc_wr_tx_put_slot() to clear the v2 pending slot and buffer structures instead of the pointer variables.
Patch 2 reduces TX slot contention by switching TX slot allocation from non-exclusive wait_event() to prepare_to_wait_exclusive(). Patch 3 replaces the global per-device CQ and manual tasklet polling model with RDMA core CQ pooling.

---
Changes v1 -> v2:
https://lore.kernel.org/netdev/[email protected]/
1. remove unnecessary inline from static CQE init helpers.
2. Use ib_drain_qp() with +1 max_send_wr;
3. Fix v2 state clearing.
4. Add re-check after schedule_timeout() to fix timeout/signal races.

Changes v2 -> v3:
https://lore.kernel.org/netdev/[email protected]/
1. Reserve +3 instead of +1 for the SQ to cover the drain, FastReg and SMC-Rv2 SEND WRs, avoiding SQ exhaustion that breaks ib_drain_sq().
2. Guard the recv WR repost with a per-link percpu_ref so no WR is reposted after ib_drain_qp(), fixing the RX repost/drain use-after-free.
3. Split the smc_wr_tx_put_slot() v2 clearing fix into a separate patch (1/3) with a Fixes: tag and reworded to the verifiable root cause.

Changes v3 -> v4:
https://lore.kernel.org/netdev/[email protected]/
1. Reorder: move the exclusive-wait patch before the CQ pooling patch.
2. Fix teardown stall: reintroduce smc_ib_modify_qp_error() and flush the QP in smc_wr_stop_link() before waiting for the refs.
3. smc_wr_reg_send(): take the ref before ib_post_send().
4. smc_wr_tx_process_cqe(): wake_up_all() on a flush completion.
5. Merge the three per-link percpu_refs into a single wr_refs.

D. Wythe (3):
net/smc: clear the correct v2 slot and buffer in smc_wr_tx_put_slot() net/smc: reduce TX slot contention with exclusive wait net/smc: transition to RDMA core CQ pooling

 net/smc/smc_core.c |  10 +-
 net/smc/smc_core.h |  36 ++--
 net/smc/smc_ib.c   | 106 ++++-------
 net/smc/smc_ib.h   |   6 -
 net/smc/smc_tx.c   |   1 -
 net/smc/smc_wr.c   | 445 +++++++++++++++++++++------------------------
 net/smc/smc_wr.h   |  50 ++---
 7 files changed, 287 insertions(+), 367 deletions(-)
Signature · 1 line
2.45.0