Skip to contentSkip to conversation

RDMA/rxe: add local implicit ODP MR support

RFC PATCH In review Liibaan Egal 2 participants 5 messages May 12

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover RDMA/rxe: add local implicit ODP MR support

This RFC adds local-access implicit On-Demand Paging memory regions to RXE (Soft-RoCE).

RXE already supports explicit ODP MRs. The implicit registration form (addr == 0, length == U64_MAX, IB_ACCESS_ON_DEMAND) is recognized but not implemented: the implicit branch in rxe_odp_mr_init_user() returns -EINVAL through a placeholder block, and no path creates child umems for SGE accesses on an implicit MR.

This series wires the implicit registration case through ib_umem_odp_alloc_implicit() and routes the local SGE walker through per-chunk child umems. The chunk size is fixed at 2 MiB (RXE_ODP_CHILD_SHIFT = 21) and children are allocated lazily on first access via ib_umem_odp_alloc_child(), stored in a per-MR xarray.

Patches
-------

1/2 RDMA/rxe: add local implicit ODP MR support

Adds rxe_odp_mr_init_implicit() (rejects remote access bits with -EOPNOTSUPP, allocates the parent umem). Adds rxe_odp_get_child() and the per-chunk loop in rxe_odp_mr_copy() and the prefetch path. Atomic, flush and atomic-write paths reject implicit MRs at the top because those helpers walk mr->umem->pfn_list directly which is empty for an implicit parent. rxe_mr_cleanup walks the child xarray and releases each child before the parent.

This patch leaves IB_ODP_SUPPORT_IMPLICIT unadvertised, so rxe_odp_mr_init_user() still returns -EINVAL on the implicit form. No user-visible behavior change yet.

2/2 RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

Flip the cap bit so userspace can probe support via ibv_query_device. Kept as its own patch so the policy question is separable from the implementation.

Question for reviewers
----------------------

Patch 2/2 advertises IB_ODP_SUPPORT_IMPLICIT for a local-access-only operation matrix. Local SGE access on implicit MRs works; remote rkey access, atomic, flush, and atomic-write on implicit MRs do not. Is this an acceptable use of the capability bit, or should capability exposure wait for a broader operation matrix? Splitting the cap flip out is meant to keep that decision separable from the implementation.

Scope and limitations
---------------------

Out of scope in this series:

- Remote rkey access on implicit MRs. Rejected at registration time with -EOPNOTSUPP.
- Atomic, flush, atomic-write paths. These return -EOPNOTSUPP / RESPST_ERR_RKEY_VIOLATION on implicit MRs.
- Child reclaim. The xarray grows monotonically per MR; a child is not freed until MR destroy. Long-lived implicit MRs that touch a sparse address space accumulate children. A reclaim mechanism is the natural follow-up.

Tested
------

Verified on rdma/for-next at commit 7fd2df204f34 (Linux 7.1-rc2), arm64, Soft-RoCE over loopback:

- Registration accept/reject matrix (5 cases).
- Single-chunk 64 KiB RDMA WRITE through an implicit lkey.
- Two-chunk multi-range test: two 1 MiB WRITEs from buffers in different 2 MiB chunks of one implicit MR.
- Cross-chunk single-SGE test: one 128 KiB WRITE whose SGE spans a 2 MiB chunk boundary.

Each patch builds cleanly standalone (M=drivers/infiniband/sw/rxe).

Registration latency was measured for 4 KiB to 1 GiB across explicit and implicit forms. Explicit grows with size and fails ENOMEM at 1 GiB on a 6 GiB host. Implicit median latency stays in the low microseconds across all sizes; peak RSS during an implicit registration stays at the baseline, while explicit RSS climbs with the registered size. The benchmark measures registration-time work only; it does not characterize first-touch or steady-state data path cost. Tests, bench and raw numbers are in the companion repository:
https://github.com/Liibon/rxe-implicit-odp

scripts/checkpatch.pl --strict on each patch: 0 errors, 0 warnings, 0 checks.

---

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

 drivers/infiniband/sw/rxe/rxe.c       |   7 +-
 drivers/infiniband/sw/rxe/rxe_mr.c    |  19 +++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++++++++-------
 drivers/infiniband/sw/rxe/rxe_verbs.h |  18 +++
 4 files changed, 275 insertions(+), 57 deletions(-)

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

 drivers/infiniband/sw/rxe/rxe.c       |   7 +-
 drivers/infiniband/sw/rxe/rxe_mr.c    |  19 ++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_verbs.h |  18 ++
 4 files changed, 275 insertions(+), 57 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/2 RDMA/rxe: add local implicit ODP MR support

RXE already supports explicit ODP MRs. The implicit registration form (addr == 0, length == U64_MAX, IB_ACCESS_ON_DEMAND) is recognized but not implemented: the implicit branch in rxe_odp_mr_init_user() returns -EINVAL through a placeholder block, and no path creates child umems for SGE accesses on an implicit MR.

Wire the implicit registration case through ib_umem_odp_alloc_implicit() and route the local SGE walker through per-chunk child umems.

Registration. rxe_odp_mr_init_implicit() rejects remote access bits (-EOPNOTSUPP), allocates the empty parent umem via ib_umem_odp_alloc_implicit(), and initializes mr->implicit_children via xa_init(). rxe_odp_init_pages() is skipped because there are no pages to fault at registration time.

Chunking. Implicit MRs split the address space into fixed-size chunks defined by RXE_ODP_CHILD_SHIFT (21, 2 MiB). Each chunk is backed by at most one child ib_umem_odp allocated on demand. The chunk size keeps the child count bounded while limiting the amount of VA covered by each child; whether the size should be fixed, derived, or configurable is an open design question.

SGE fault path. rxe_odp_umem_for_iova() returns the parent for explicit MRs and rxe_odp_get_child() for implicit MRs. The child lookup is xa_load -> ib_umem_odp_alloc_child -> xa_cmpxchg(GFP_KERNEL); a racing insertion drops the loser. rxe_odp_chunk_len_at() reports how many bytes of an access can be served by one umem; for explicit MRs that is the full request, for implicit it is the bytes remaining in the current chunk. rxe_odp_mr_copy() loops across chunks, resolving, locking, copying, and unlocking each child independently. Explicit MRs run the loop exactly once with identical behavior to the pre-patch path.

Prefetch. rxe_odp_prefetch_one() uses the same chunk loop. Async prefetch walks per chunk under short-held mutexes so a long range does not stall concurrent invalidators.

Atomic, flush, and atomic-write paths reject implicit MRs at the top of each helper. These walk mr->umem->pfn_list directly which is empty for an implicit parent; extending them is not in this series.

Lifetime. rxe_mr_cleanup walks mr->implicit_children with xa_for_each and releases each child via ib_umem_odp_release() before releasing the parent via ib_umem_release(), so each child's mmu_interval_notifier tears down while the parent's per_mm is alive. The xarray is xa_destroy()ed afterwards.

Per-transport ODP caps are unchanged: they describe RC/UD behavior on explicit ODP MRs. Advertising IB_ODP_SUPPORT_IMPLICIT to userspace is a separate patch, since whether the existing capability bit is the right surface for a local-access-only operation matrix is an open question for review.

Limitations. The xarray grows monotonically per MR: a child is not reclaimed until MR destroy. Long-lived MRs that touch a sparse address space accumulate children. A reclaim mechanism is the natural follow-up.

Tested on Linux 7.1-rc2 (arm64, Soft-RoCE over loopback):
- five-case registration accept/reject matrix passes
- single-chunk 64 KiB RDMA WRITE through an implicit lkey delivers
- two-chunk multi test (two 1 MiB WRITEs from buffers in different 2 MiB chunks of one implicit MR) delivers
- cross-chunk single-SGE test (128 KiB WRITE spanning a 2 MiB boundary) delivers

Benchmark measures registration latency and RSS only; first-touch and steady-state data path costs are not characterized in this series.

---
 drivers/infiniband/sw/rxe/rxe_mr.c    |  19 ++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_verbs.h |  18 ++
 3 files changed, 269 insertions(+), 56 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +19 -0
index c696ff8749..c429bf0e6f 100644
@@ -6,6 +6,8 @@
 
 #include <linux/libnvdimm.h>
 
+#include <rdma/ib_umem_odp.h>
+
 #include "rxe.h"
 #include "rxe_loc.h"
 
@@ -809,6 +811,23 @@ void rxe_mr_cleanup(struct rxe_pool_elem *elem)
 	struct rxe_mr *mr = container_of(elem, typeof(*mr), elem);
 
 	rxe_put(mr_pd(mr));
+
+	/* Implicit ODP MRs may have created child umems on demand for each
+	 * accessed 2 MiB chunk. Release them before the parent so each
+	 * child's mmu_interval_notifier tears down while the parent's
+	 * per_mm is still alive. The xarray is empty for explicit MRs, so
+	 * walking it is a no-op there.
+	 */
+	if (mr->umem && mr->umem->is_odp &&
+	    to_ib_umem_odp(mr->umem)->is_implicit_odp) {
+		struct ib_umem_odp *child;
+		unsigned long key;
+
+		xa_for_each(&mr->implicit_children, key, child)
+			ib_umem_odp_release(child);
+		xa_destroy(&mr->implicit_children);
+	}
+
 	ib_umem_release(mr->umem);
 
 	if (mr->ibmr.type != IB_MR_TYPE_DMA)
drivers/infiniband/sw/rxe/rxe_odp.c +232 -56

475 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_verbs.h +18 -0
index d92f80d16f..a783dee95d 100644
@@ -341,12 +341,30 @@ struct rxe_mr_page {
 	unsigned int		offset; /* offset in system page */
 };
 
+/* For implicit ODP MRs the virtual address space is split into fixed-size
+ * chunks. Each chunk is backed by at most one child umem allocated on
+ * first access. The 2 MiB chunk size keeps the child count bounded while
+ * limiting the amount of VA covered by each child. Whether the chunk
+ * size should be fixed, derived from page_shift, or configurable is an
+ * open design question for review.
+ */
+#define RXE_ODP_CHILD_SHIFT 21
+#define RXE_ODP_CHILD_SIZE  (BIT(RXE_ODP_CHILD_SHIFT))
+#define RXE_ODP_CHILD_MASK  (RXE_ODP_CHILD_SIZE - 1)
+
 struct rxe_mr {
 	struct rxe_pool_elem	elem;
 	struct ib_mr		ibmr;
 
 	struct ib_umem		*umem;
 
+	/* For implicit ODP MRs only: xarray of child umems keyed by
+	 * (aligned_start >> RXE_ODP_CHILD_SHIFT). Each entry covers one
+	 * RXE_ODP_CHILD_SIZE-aligned chunk and is created lazily on first
+	 * access. Unused (xa_empty) for explicit MRs.
+	 */
+	struct xarray		implicit_children;
+
 	u32			lkey;
 	u32			rkey;
 	enum rxe_mr_state	state;
Signature · 1 line
2.43.0
#2 lore raw

One clarification on the wording in the cover letter and patch 1/2 commit message: when I said the per-transport ODP caps describe explicit ODP MR semantics, that was too strong. What I meant is that this series leaves the existing per-transport caps unchanged and implements only the local lkey implicit-MR access path, while rejecting remote rkey, atomic, flush, and atomic-write uses of implicit MRs.

The intended review question remains whether advertising IB_ODP_SUPPORT_IMPLICIT is acceptable for that local-access-only implicit operation matrix, or whether the cap should wait for broader implicit coverage. I will reword this in a v2 if the series moves forward.

#3 lore raw

RFC PATCH 2/2 RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

In reply to Liibaan Egal

Now that the implicit ODP registration and local SGE fault paths are in place, advertise IB_ODP_SUPPORT_IMPLICIT in general_odp_caps so userspace can probe the support via ibv_query_device.

The advertised support is intentionally scoped to local access:
remote rkey access on implicit MRs is rejected at registration time, and the atomic, flush, and atomic-write paths reject implicit MRs at the top of each helper.

Question for reviewers: is IB_ODP_SUPPORT_IMPLICIT the right capability bit to advertise for this local-access-only operation matrix, or should capability exposure wait for broader operation coverage? The cap-flip is kept in its own patch so the policy decision is separable from the implementation.

---
 drivers/infiniband/sw/rxe/rxe.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe.c +6 -1
index b0714f9abe..581313591d 100644
@@ -94,8 +94,13 @@ static void rxe_init_device_param(struct rxe_dev *rxe, struct net_device *ndev)
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		rxe->attr.kernel_cap_flags |= IBK_ON_DEMAND_PAGING;
 
-		/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */
 		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT;
+		/* IMPLICIT is gated to the local-access subset. The fault path
+		 * in rxe_odp.c rejects remote-access implicit forms at
+		 * registration time. Per-transport caps below stay unchanged:
+		 * they describe explicit ODP MR semantics and remain accurate.
+		 */
+		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT_IMPLICIT;
 
 		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_SEND;
 		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_RECV;
Signature · 1 line
2.43.0
#4 lore raw
In reply to Liibaan Egal
Quoted 69 lines · Liibaan Egal

This RFC adds local-access implicit On-Demand Paging memory regions to RXE (Soft-RoCE).

RXE already supports explicit ODP MRs. The implicit registration form (addr == 0, length == U64_MAX, IB_ACCESS_ON_DEMAND) is recognized but not implemented: the implicit branch in rxe_odp_mr_init_user() returns -EINVAL through a placeholder block, and no path creates child umems for SGE accesses on an implicit MR.

This series wires the implicit registration case through ib_umem_odp_alloc_implicit() and routes the local SGE walker through per-chunk child umems. The chunk size is fixed at 2 MiB (RXE_ODP_CHILD_SHIFT = 21) and children are allocated lazily on first access via ib_umem_odp_alloc_child(), stored in a per-MR xarray.

Patches
-------

1/2 RDMA/rxe: add local implicit ODP MR support

Adds rxe_odp_mr_init_implicit() (rejects remote access bits with -EOPNOTSUPP, allocates the parent umem). Adds rxe_odp_get_child() and the per-chunk loop in rxe_odp_mr_copy() and the prefetch path. Atomic, flush and atomic-write paths reject implicit MRs at the top because those helpers walk mr->umem->pfn_list directly which is empty for an implicit parent. rxe_mr_cleanup walks the child xarray and releases each child before the parent.

This patch leaves IB_ODP_SUPPORT_IMPLICIT unadvertised, so rxe_odp_mr_init_user() still returns -EINVAL on the implicit form. No user-visible behavior change yet.

2/2 RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

Flip the cap bit so userspace can probe support via ibv_query_device. Kept as its own patch so the policy question is separable from the implementation.

Question for reviewers
----------------------

Patch 2/2 advertises IB_ODP_SUPPORT_IMPLICIT for a local-access-only operation matrix. Local SGE access on implicit MRs works; remote rkey access, atomic, flush, and atomic-write on implicit MRs do not. Is this an acceptable use of the capability bit, or should capability exposure wait for a broader operation matrix? Splitting the cap flip out is meant to keep that decision separable from the implementation.

Scope and limitations
---------------------

Out of scope in this series:

- Remote rkey access on implicit MRs. Rejected at registration time with -EOPNOTSUPP.
- Atomic, flush, atomic-write paths. These return -EOPNOTSUPP / RESPST_ERR_RKEY_VIOLATION on implicit MRs.
- Child reclaim. The xarray grows monotonically per MR; a child is not freed until MR destroy. Long-lived implicit MRs that touch a sparse address space accumulate children. A reclaim mechanism is the natural follow-up.

Tested
------

Verified on rdma/for-next at commit 7fd2df204f34 (Linux 7.1-rc2), arm64, Soft-RoCE over loopback:

Quoted 8 lines

- Registration accept/reject matrix (5 cases).
- Single-chunk 64 KiB RDMA WRITE through an implicit lkey.
- Two-chunk multi-range test: two 1 MiB WRITEs from buffers in different 2 MiB chunks of one implicit MR.
- Cross-chunk single-SGE test: one 128 KiB WRITE whose SGE spans a 2 MiB chunk boundary.

Each patch builds cleanly standalone (M=drivers/infiniband/sw/rxe).

IMO, please use a shell script like the following to act as selftest. Please put the following script in tools/testing/selftests/rdma/

Or you can add more testcases to prove your features.

"
#!/bin/bash
# Enable exit on error for better debugging
set -e

# 1. Cleanup old environment
echo "Cleaning up..."
ip netns delete ns0 2>/dev/null || true
ip link delete nk1 2>/dev/null || true

# 2. Setup Network Namespaces and Netkit interfaces echo "Setting up network..."
ip netns add ns0

# Create netkit pair: nk1 (host) and nk0 (to be moved to ns0) ip link add nk1 type netkit mode l2 peer name nk0

# Set host side up
ip link set nk1 up
ip addr add 10.0.0.2/24 dev nk1

# Move nk0 to namespace ns0
ip link set nk0 netns ns0
ip netns exec ns0 ip addr add 10.0.0.1/24 dev nk0
ip netns exec ns0 ip link set nk0 up
ip netns exec ns0 ip link set lo up

# Verify connectivity
echo "Verifying IP connectivity..."
ping -c 2 10.0.0.1 -I nk1

# 3. Setup Soft-RoCE (RXE) links
echo "Configuring RXE..."
# In namespace ns0
ip netns exec ns0 rdma link add rxe0 type rxe netdev nk0 # In host namespace
rdma link add rxe1 type rxe netdev nk1

# Wait for RDMA devices to initialize
sleep 1
rdma link

# 4. Run ibv_rc_pingpong with Implicit ODP (-O) echo "Starting ibv_rc_pingpong with Implicit ODP..."

# Start Server in ns0
# -g 1: GID index (usually 1 for RoCE v2)
# -O: Use Implicit ODP
ip netns exec ns0 ibv_rc_pingpong -g 1 -O &
SERVER_PID=$!

# Give the server a moment to bind
sleep 2

# Start Client in host
# -O: Use Implicit ODP
ibv_rc_pingpong -g 1 -O 10.0.0.1

# 5. Collect Statistics
echo "--- Post-test Statistics ---"
echo "Host Stats:"
ip -s link show nk1
echo "Namespace ns0 Stats:"
ip netns exec ns0 ip -s link show nk0

# 6. Cleanup
echo "Cleaning up..."
kill $SERVER_PID 2>/dev/null || true

rdma link del rxe0 2>/dev/null || true
rdma link del rxe1 2>/dev/null || true

ip link del nk1
ip netns delete ns0

echo "Test Complete."
"

The output should be the following

"

# ./implicit_odp.sh
Cleaning up...
Setting up network...
Verifying IP connectivity...
PING 10.0.0.1 (10.0.0.1) from 10.0.0.2 nk1: 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.040 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1013ms
rtt min/avg/max/mdev = 0.040/0.055/0.071/0.015 ms
Configuring RXE...
link rxe0/1 state ACTIVE physical_state LINK_UP
link rxe1/1 state ACTIVE physical_state LINK_UP netdev nk1
Starting ibv_rc_pingpong with Implicit ODP...
local address: LID 0x0000, QPN 0x000011, PSN 0x51486a, GID
::ffff:10.0.0.1
local address: LID 0x0000, QPN 0x000012, PSN 0xc14439, GID
::ffff:10.0.0.1
remote address: LID 0x0000, QPN 0x000011, PSN 0x51486a, GID
::ffff:10.0.0.1
remote address: LID 0x0000, QPN 0x000012, PSN 0xc14439, GID
::ffff:10.0.0.1
8192000 bytes in 0.03 seconds = 2341.91 Mbit/sec
8192000 bytes in 0.03 seconds = 2354.70 Mbit/sec
1000 iters in 0.03 seconds = 27.83 usec/iter
1000 iters in 0.03 seconds = 27.98 usec/iter
--- Post-test Statistics ---
Host Stats:
8: nk1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ba:48:69:41:c7:71 brd ff:ff:ff:ff:ff:ff link-netns ns0
RX: bytes packets errors dropped missed mcast
1078 13 0 0 0 0
TX: bytes packets errors dropped carrier collsns
4326 35 0 1 0 0
Namespace ns0 Stats:
7: nk0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3a:46:ee:e9:12:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0
RX: bytes packets errors dropped missed mcast
4326 35 0 0 0 0
TX: bytes packets errors dropped carrier collsns
1078 13 0 0 0 0
Cleaning up...
Test Complete.
"

If you think that rdma-core is better, I am fine with it.

Anyway, some testcases are needed to prove your feature.

Zhu Yanjun

Quoted 36 lines

Registration latency was measured for 4 KiB to 1 GiB across explicit and implicit forms. Explicit grows with size and fails ENOMEM at 1 GiB on a 6 GiB host. Implicit median latency stays in the low microseconds across all sizes; peak RSS during an implicit registration stays at the baseline, while explicit RSS climbs with the registered size. The benchmark measures registration-time work only; it does not characterize first-touch or steady-state data path cost. Tests, bench and raw numbers are in the companion repository:
https://github.com/Liibon/rxe-implicit-odp

scripts/checkpatch.pl --strict on each patch: 0 errors, 0 warnings, 0 checks.

---

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

  drivers/infiniband/sw/rxe/rxe.c       |   7 +-
  drivers/infiniband/sw/rxe/rxe_mr.c    |  19 +++
  drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++++++++-------
  drivers/infiniband/sw/rxe/rxe_verbs.h |  18 +++
  4 files changed, 275 insertions(+), 57 deletions(-)

Liibaan Egal (2):
RDMA/rxe: add local implicit ODP MR support RDMA/rxe: advertise IB_ODP_SUPPORT_IMPLICIT for local access

  drivers/infiniband/sw/rxe/rxe.c       |   7 +-
  drivers/infiniband/sw/rxe/rxe_mr.c    |  19 ++
  drivers/infiniband/sw/rxe/rxe_odp.c   | 288 +++++++++++++++++++++-----
  drivers/infiniband/sw/rxe/rxe_verbs.h |  18 ++
  4 files changed, 275 insertions(+), 57 deletions(-)