Skip to content

RFC RDMA/rxe: add local implicit ODP MR support

Liibaan Egal linux-rdma lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest2 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2RDMA/rxe: add local implicit ODP MR supportdrivers/infiniband/sw/rxe/rxe_mr.c drivers/infiniband/sw/rxe/rxe_odp.c drivers/infiniband/sw/rxe/rxe_verbs.h+269 -561 reply
2/2RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local accessdrivers/infiniband/sw/rxe/rxe.c+6 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH rdma-next 0/2] RDMA/rxe: add local implicit ODP MR support

This RFC adds local-access implicit On-Demand Paging memory regions to RXE (Soft-RoCE).

RXE already supports explicit ODP MRs. The implicit registration form (addr == 0, length == U64_MAX, IB_ACCESS_ON_DEMAND) is recognized but not implemented: the implicit branch in rxe_odp_mr_init_user() returns -EINVAL through a placeholder block, and no path creates child umems for SGE accesses on an implicit MR.

This series wires the implicit registration case through ib_umem_odp_alloc_implicit() and routes the local SGE walker through per-chunk child umems. The chunk size is fixed at 2 MiB (RXE_ODP_CHILD_SHIFT = 21) and children are allocated lazily on first access via ib_umem_odp_alloc_child(), stored in a per-MR xarray.

Patches
-------

1/2 RDMA/rxe: add local implicit ODP MR support

Adds rxe_odp_mr_init_implicit() (rejects remote access bits with -EOPNOTSUPP, allocates the parent umem). Adds rxe_odp_get_child() and the per-chunk loop in rxe_odp_mr_copy() and the prefetch path. Atomic, flush and atomic-write paths reject implicit MRs at the top because those helpers walk mr->umem->pfn_list directly which is empty for an implicit parent. rxe_mr_cleanup walks the child xarray and releases each child before the parent.

This patch leaves IB_ODP_SUPPORT_IMPLICIT unadvertised, so rxe_odp_mr_init_user() still returns -EINVAL on the implicit form. No user-visible behavior change yet.

2/2 RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

Flip the cap bit so userspace can probe support via ibv_query_device. Kept as its own patch so the policy question is separable from the implementation.

Question for reviewers
----------------------

Patch 2/2 advertises IB_ODP_SUPPORT_IMPLICIT for a local-access-only operation matrix. Local SGE access on implicit MRs works; remote rkey access, atomic, flush, and atomic-write on implicit MRs do not. Is this an acceptable use of the capability bit, or should capability exposure wait for a broader operation matrix? Splitting the cap flip out is meant to keep that decision separable from the implementation.

Scope and limitations
---------------------

Out of scope in this series:

- Remote rkey access on implicit MRs. Rejected at registration time with -EOPNOTSUPP.
- Atomic, flush, atomic-write paths. These return -EOPNOTSUPP / RESPST_ERR_RKEY_VIOLATION on implicit MRs.
- Child reclaim. The xarray grows monotonically per MR; a child is not freed until MR destroy. Long-lived implicit MRs that touch a sparse address space accumulate children. A reclaim mechanism is the natural follow-up.

Tested
------

Verified on rdma/for-next at commit 7fd2df204f34 (Linux 7.1-rc2), arm64, Soft-RoCE over loopback:

- Registration accept/reject matrix (5 cases).
- Single-chunk 64 KiB RDMA WRITE through an implicit lkey.
- Two-chunk multi-range test: two 1 MiB WRITEs from buffers in different 2 MiB chunks of one implicit MR.
- Cross-chunk single-SGE test: one 128 KiB WRITE whose SGE spans a 2 MiB chunk boundary.

Each patch builds cleanly standalone (M=drivers/infiniband/sw/rxe).

Registration latency was measured for 4 KiB to 1 GiB across explicit and implicit forms. Explicit grows with size and fails ENOMEM at 1 GiB on a 6 GiB host. Implicit median latency stays in the low microseconds across all sizes; peak RSS during an implicit registration stays at the baseline, while explicit RSS climbs with the registered size. The benchmark measures registration-time work only; it does not characterize first-touch or steady-state data path cost. Tests, bench and raw numbers are in the companion repository:
https://github.com/Liibon/rxe-implicit-odp

scripts/checkpatch.pl --strict on each patch: 0 errors, 0 warnings, 0 checks.

---

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

 drivers/infiniband/sw/rxe/rxe.c       |   7 +-
 drivers/infiniband/sw/rxe/rxe_mr.c    |  19 +++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++++++++-------
 drivers/infiniband/sw/rxe/rxe_verbs.h |  18 +++
 4 files changed, 275 insertions(+), 57 deletions(-)

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

 drivers/infiniband/sw/rxe/rxe.c       |   7 +-
 drivers/infiniband/sw/rxe/rxe_mr.c    |  19 ++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_verbs.h |  18 ++
 4 files changed, 275 insertions(+), 57 deletions(-)
Signature · 1 line
2.43.0