Skip to content

RDMA/mana_ib: Add kernel RC and fast registration support

Konstantin Taranov linux-rdma lkml sashiko-reviews 11 versions – Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v11 Latest9 patches 2 repliesApplied
  2. v109 patches 2 repliesSuperseded
  3. v99 patches 3 repliesSuperseded
Earlier versions · 8
VersionPostedPatchesRepliesStatus
v89 patches0 repliesSuperseded
v79 patches0 repliesSuperseded
v610 patches0 repliesSuperseded
v510 patches1 replySuperseded
v410 patches0 repliesSuperseded
v310 patches0 repliesSuperseded
v210 patches0 repliesSuperseded
v110 patches0 repliesSuperseded

Changes in v11 from the cover letter

- Patch 09: Reset kernel QPs before destruction unless already in RESET or INIT, and always clean CQEs. Keep shared-CQ cleanup under one uninterrupted CQ lock.

Patches in v11 Applied

#Subject and filesSizeRepliesApplied
1/9RDMA/mana_ib: Allocate and map fast-registration MRsdrivers/infiniband/hw/mana/device.c drivers/infiniband/hw/mana/main.c drivers/infiniband/hw/mana/mana_ib.h +2 more+150 -20 replies
2/9RDMA/mana: Create and destroy kernel RC QPsdrivers/infiniband/hw/mana/main.c drivers/infiniband/hw/mana/mana_ib.h drivers/infiniband/hw/mana/qp.c+167 -100 replies
3/9net/mana: Extend GDMA encoding for new RDMA WQEsdrivers/net/ethernet/microsoft/mana/gdma_main.c include/net/mana/gdma.h+63 -130 replies
4/9RDMA/mana_ib: Maintain kernel RC QP statedrivers/infiniband/hw/mana/mana_ib.h drivers/infiniband/hw/mana/qp.c include/net/mana/gdma.h+27 -20 replies
5/9RDMA/mana_ib: Post receive WRs on kernel RC QPsdrivers/infiniband/hw/mana/wr.c+37 -00 replies
6/9RDMA/mana_ib: Post send and memory-management WRs on RC QPsdrivers/infiniband/hw/mana/main.c drivers/infiniband/hw/mana/mana_ib.h drivers/infiniband/hw/mana/wr.c+350 -20 replies
7/9RDMA/mana_ib: Poll RC completions using PSN and FSN progressdrivers/infiniband/hw/mana/cq.c drivers/infiniband/hw/mana/mana_ib.h drivers/infiniband/hw/mana/shadow_queue.h +1 more+203 -110 replies
8/9RDMA/mana_ib: Flush and notify CQs when kernel QPs enter ERRdrivers/infiniband/hw/mana/cq.c drivers/infiniband/hw/mana/qp.c drivers/infiniband/hw/mana/wr.c+105 -170 replies
9/9RDMA/mana_ib: Handle error CQEs for kernel QPsdrivers/infiniband/hw/mana/cq.c drivers/infiniband/hw/mana/mana_ib.h drivers/infiniband/hw/mana/qp.c+311 -461 reply

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v11 yet.

Read the v11 thread

Cover letter [PATCH rdma-next v11 0/9] RDMA/mana_ib: Add kernel RC and fast registration support

From: Konstantin Taranov <[email protected]>

Add kernel RC QPs and fast-registration MRs to the MANA RDMA driver for kernel consumers such as NVMe/RDMA and NFS/RDMA.

Implement QP creation and state management, work request posting, completion processing, and error handling. Use the RDMA core's generic queue drains for kernel RC QPs, with software-flush notification in the posting paths. Kernel RC QPs require extended-WQE and power-of-two-SQ hardware support.

v11:
- Patch 09: Reset kernel QPs before destruction unless already in RESET or INIT, and always clean CQEs. Keep shared-CQ cleanup under one uninterrupted CQ lock.
v10:
- Patch 01: Remove the FMR page-count limit check
- Patch 02: Remove the minimum firmware send-SGE check, kernel QP queue prevalidation
- Patch 06: Remove send-SGE and READ-SGE limit checks and message- length overflow validation

v9:
- Patch 07 and 09: Retire acknowledged successful unsignaled WQEs with a full WC array to avoid deferring rearming.

v8:
- Patch 06: Annotate rc_send.immediate and immediate_ext as __le32

v7:
- Patch 09: Include GSI QPs in deferred ERR handling.
- Patch 09: Complete software-flush setup even if the deferred firmware ERR request fails. An error CQE guarantees whole-QP DMA shutdown for all QP types.
- Drop patch 10 to use the generic SQ/RQ drains for RC.

v6:
- Patch 08: Notify software flushes after posting to error-listed QPs. Add reporting pending software flush for IB_CQ_REPORT_MISSED_EVENTS.
- Patch 09: Check fenced WQEs on error cqes to prevent flushing completed WQEs. Add mana_ib_cq_clean on QP reset to discard stale CQEs. QID 0 is an invalid value for RNIC and safe dor discarding. Add error_pending flag to prevent a deffered ERR request to error reset QPs.

v5:
- Patch 01: Reject max_num_sg above MANA_MAX_FMR_PAGES.
- Patch 02: Validate kernel QP queue sizes and SGE limits to prevent overflows. Ensure that queue size cannnot be 0.
- Patch 06: Preserve fence flags on unsignaled WQEs. Check message length and read sges before posting a wqe.
- Patch 08: Remove the unused QP state field.

v4:
- Patch 01: Use an explicit 64-byte FMR MST alignment
- Patch 02: Validate that the RNIC firmware reports at least two send SGEs and subtract the reserved SGE only for RNIC-based mana_ib.
- Patch 07: Add missing wqe_size to shadow wqe for SEND/Write, and return a missed-event hint for cached CQEs when CQ is armed

v3:
- Patch 07: Mask arm_psn to 24 bits before ringing the doorbell.
- Patch 08: Exclude userspace QPs from kernel software-state handling and CQ error-list insertion, and unlink UC QPs during destruction.

v2:
- Patch 03: Validate fixed-WQE capacity for the header, OOB and SGEs.
- Patch 04: Preserve UC QP modification support.
- Patch 08: Skip software CQ notifications for direct-polling CQs.
- Patch 09: Flush that ERR work during QP teardown.
- Patch 10: Limit drain callbacks to RC QPs and avoid completion callbacks for direct-polling CQs.

Konstantin Taranov (9):
RDMA/mana_ib: Allocate and map fast-registration MRs RDMA/mana: Create and destroy kernel RC QPs net/mana: Extend GDMA encoding for new RDMA WQEs RDMA/mana_ib: Maintain kernel RC QP state RDMA/mana_ib: Post receive WRs on kernel RC QPs RDMA/mana_ib: Post send and memory-management WRs on RC QPs RDMA/mana_ib: Poll RC completions using PSN and FSN progress RDMA/mana_ib: Flush and notify CQs when kernel QPs enter ERR RDMA/mana_ib: Handle error CQEs for kernel QPs

 drivers/infiniband/hw/mana/cq.c                 | 356 ++++++++++++++++++--
 drivers/infiniband/hw/mana/device.c             |   2 +
 drivers/infiniband/hw/mana/main.c               |   8 +-
 drivers/infiniband/hw/mana/mana_ib.h            | 142 +++++++-
 drivers/infiniband/hw/mana/mr.c                 |  84 ++++-
 drivers/infiniband/hw/mana/qp.c                 | 332 +++++++++++++++++--
 drivers/infiniband/hw/mana/shadow_queue.h       |  22 ++
 drivers/infiniband/hw/mana/wr.c                 | 423 +++++++++++++++++++++++-
 drivers/net/ethernet/microsoft/mana/gdma_main.c |  55 ++-
 include/net/mana/gdma.h                         |  38 ++-
 10 files changed, 1386 insertions(+), 76 deletions(-)
Signature · 1 line
2.43.0