Skip to contentSkip to conversation

On-Demand Paging on SoftRoCE

PATCH Applied Daisuke Matsuda 4 participants 17 messages Dec 20, 2024 – Feb 21, 2025 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v9 cover On-Demand Paging on SoftRoCE

This patch series implements the On-Demand Paging feature on SoftRoCE(rxe) driver, which has been available only in mlx5 driver[1] so far.

This series has been blocked because of the hang issue of srp 002 test[2], which was believed to be caused after applying the commit 9b4b7c1f9f54 ("RDMA/rxe: Add workqueue support for rxe tasks"). My patches are dependent on the commit because the ODP feature requires sleeping in kernel space, and it is impossible with the former tasklet implementation.

According to the original reporter[3], the hang issue is already gone in v6.10. Additionally, tasklet is marked deprecated[4]. I think the rxe driver is ready to accept this series since there is no longer any reason to consider reverting back to the old tasklet.

I omitted some contents like the motive behind this series from the cover- letter. Please see the cover letter of v3 for more details[5].

[Overview]
When applications register a memory region(MR), RDMA drivers normally pin pages in the MR so that physical addresses are never changed during RDMA communication. This requires the MR to fit in physical memory and inevitably leads to memory pressure. On the other hand, On-Demand Paging (ODP) allows applications to register MRs without pinning pages. They are paged-in when the driver requires and paged-out when the OS reclaims. As a result, it is possible to register a large MR that does not fit in physical memory without taking up so much physical memory.

[How does ODP work?]
"struct ib_umem_odp" is used to manage pages. It is created for each ODP-enabled MR on its registration. This struct holds a pair of arrays (dma_list/pfn_list) that serve as a driver page table. DMA addresses and PFNs are stored in the driver page table. They are updated on page-in and page-out, both of which use the common interfaces in the ib_uverbs layer.

Page-in can occur when requester, responder or completer access an MR in order to process RDMA operations. If they find that the pages being accessed are not present on physical memory or requisite permissions are not set on the pages, they provoke page fault to make the pages present with proper permissions and at the same time update the driver page table. After confirming the presence of the pages, they execute memory access such as read, write or atomic operations.

Page-out is triggered by page reclaim or filesystem events (e.g. metadata update of a file that is being used as an MR). When creating an ODP-enabled MR, the driver registers an MMU notifier callback. When the kernel issues a page invalidation notification, the callback is provoked to unmap DMA addresses and update the driver page table. After that, the kernel releases the pages.

[Supported operations]
All traditional operations are supported on RC connection. The new Atomic write[6] and RDMA Flush[7] operations are not included in this patchset. I will post them later after this patchset is merged. On UD connection, Send, Recv, and SRQ-Recv are supported.

[How to test ODP?]
There are only a few resources available for testing. pyverbs testcases in rdma-core and perftest[8] are recommendable ones. Other than them, the ibv_rc_pingpong command can also be used for testing. Note that you may have to build perftest from upstream because old versions do not handle ODP capabilities correctly.

The latest ODP tree is available from github:
https://github.com/ddmatsu/linux/tree/odp_v9

[Future work]
My next work is to enable the new Atomic write[6] and RDMA Flush[7] operations with ODP. After that, I am going to implement the prefetch feature. It allows applications to trigger page fault using ibv_advise_mr(3) to optimize performance. Some existing software like librpma[9] use this feature. Additionally, I think we can also add the implicit ODP feature in the future.

[1] Understanding On Demand Paging (ODP)
https://enterprise-support.nvidia.com/s/article/understanding-on-demand-paging--odp-x

[2] [bug report] blktests srp/002 hang
https://lore.kernel.org/linux-rdma/dsg6rd66tyiei32zaxs6ddv5ebefr5vtxjwz6d2ewqrcwisogl@ge7jzan7dg5u/T/

[3] blktests failures with v6.10-rc1 kernel
https://lore.kernel.org/linux-block/wnucs5oboi4flje5yvtea7puvn6zzztcnlrfz3lpzlwgblrxgw@7wvqdzioejgl/

[4] [00/15] ethernet: Convert from tasklet to BH workqueue
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/

[5] [PATCH for-next v3 0/7] On-Demand Paging on SoftRoCE
https://lore.kernel.org/lkml/[email protected]/

[6] [PATCH v7 0/8] RDMA/rxe: Add atomic write operation
https://lore.kernel.org/linux-rdma/[email protected]/

[7] [for-next PATCH 00/10] RDMA/rxe: Add RDMA FLUSH operation
https://lore.kernel.org/lkml/[email protected]/

[8] linux-rdma/perftest: Infiniband Verbs Performance Tests
https://github.com/linux-rdma/perftest

[9] librpma: Remote Persistent Memory Access Library
https://github.com/pmem/rpma

v8->v9:
1) Addressed comments from Jason Gunthorpe
cf. https://lore.kernel.org/lkml/[email protected]/
cf. https://lore.kernel.org/lkml/[email protected]/
2) rebased to 6.13.0-rc1

v7->v8:
1) Dropped the first patch because the same change was made by Bob Pearson. cf. https://github.com/torvalds/linux/commit/23bc06af547f2ca3b7d345e09fd8d04575406274
2) Rebased to 6.12.1-rc2

v6->v7:
1) Rebased to 6.6.0
2) Disabled using hugepages with ODP
3) Addressed comments on v6 from Jason and Zhu
cf. https://lore.kernel.org/lkml/[email protected]/

v5->v6:
Fixed the implementation according to Jason's suggestions
cf. https://lore.kernel.org/all/[email protected]/
cf. https://lore.kernel.org/all/[email protected]/

v4->v5:
1) Rebased to 6.4.0-rc2+
2) Changed to schedule all works on responder and completer to workqueue

v3->v4:
1) Re-designed functions that access MRs to use the MR xarray.
2) Rebased onto the latest jgg-for-next tree.

v2->v3:
1) Removed a patch that changes the common ib_uverbs layer.
2) Re-implemented patches for conversion to workqueue.
3) Fixed compile errors (happened when CONFIG_INFINIBAND_ON_DEMAND_PAGING=n).
4) Fixed some functions that returned incorrect errors.
5) Temporarily disabled ODP for RDMA Flush and Atomic Write.

v1->v2:
1) Fixed a crash issue reported by Haris Iqbal.
2) Tried to make lock patters clearer as pointed out by Romanovsky.
3) Minor clean ups and fixes.

Daisuke Matsuda (5):
RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP RDMA/rxe: Add page invalidation support
RDMA/rxe: Allow registering MRs for On-Demand Paging RDMA/rxe: Add support for Send/Recv/Write/Read with ODP RDMA/rxe: Add support for the traditional Atomic operations with ODP

 drivers/infiniband/sw/rxe/Makefile    |   2 +
 drivers/infiniband/sw/rxe/rxe.c       |  18 ++
 drivers/infiniband/sw/rxe/rxe.h       |  37 ---
 drivers/infiniband/sw/rxe/rxe_loc.h   |  34 +++
 drivers/infiniband/sw/rxe/rxe_mr.c    |  13 +-
 drivers/infiniband/sw/rxe/rxe_odp.c   | 324 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c  |  18 +-
 drivers/infiniband/sw/rxe/rxe_verbs.c |   5 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h |  37 +++
 9 files changed, 443 insertions(+), 45 deletions(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
Signature · 1 line
2.43.0
#1 lore raw

PATCH v9 1/5 RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP

rxe_mr_init() and resp_states are going to be used in rxe_odp.c, which is to be created in the subsequent patch.

---
 drivers/infiniband/sw/rxe/rxe.h       | 37 ---------------------------
 drivers/infiniband/sw/rxe/rxe_loc.h   |  1 +
 drivers/infiniband/sw/rxe/rxe_mr.c    |  2 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h | 37 +++++++++++++++++++++++++++
 4 files changed, 39 insertions(+), 38 deletions(-)
drivers/infiniband/sw/rxe/rxe.h +0 -37
index d8fb2c7af30a..193f7caffaf2 100644
@@ -100,43 +100,6 @@
 #define rxe_info_mw(mw, fmt, ...) ibdev_info_ratelimited((mw)->ibmw.device, \
 		"mw#%d %s:  " fmt, (mw)->elem.index, __func__, ##__VA_ARGS__)
 
-/* responder states */
-enum resp_states {
-	RESPST_NONE,
-	RESPST_GET_REQ,
-	RESPST_CHK_PSN,
-	RESPST_CHK_OP_SEQ,
-	RESPST_CHK_OP_VALID,
-	RESPST_CHK_RESOURCE,
-	RESPST_CHK_LENGTH,
-	RESPST_CHK_RKEY,
-	RESPST_EXECUTE,
-	RESPST_READ_REPLY,
-	RESPST_ATOMIC_REPLY,
-	RESPST_ATOMIC_WRITE_REPLY,
-	RESPST_PROCESS_FLUSH,
-	RESPST_COMPLETE,
-	RESPST_ACKNOWLEDGE,
-	RESPST_CLEANUP,
-	RESPST_DUPLICATE_REQUEST,
-	RESPST_ERR_MALFORMED_WQE,
-	RESPST_ERR_UNSUPPORTED_OPCODE,
-	RESPST_ERR_MISALIGNED_ATOMIC,
-	RESPST_ERR_PSN_OUT_OF_SEQ,
-	RESPST_ERR_MISSING_OPCODE_FIRST,
-	RESPST_ERR_MISSING_OPCODE_LAST_C,
-	RESPST_ERR_MISSING_OPCODE_LAST_D1E,
-	RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
-	RESPST_ERR_RNR,
-	RESPST_ERR_RKEY_VIOLATION,
-	RESPST_ERR_INVALIDATE_RKEY,
-	RESPST_ERR_LENGTH,
-	RESPST_ERR_CQ_OVERFLOW,
-	RESPST_ERROR,
-	RESPST_DONE,
-	RESPST_EXIT,
-};
-
 void rxe_set_mtu(struct rxe_dev *rxe, unsigned int dev_mtu);
 
 int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name);
drivers/infiniband/sw/rxe/rxe_loc.h +1 -0
index ded46119151b..1c78b9d36da4 100644
@@ -58,6 +58,7 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
 
 /* rxe_mr.c */
 u8 rxe_get_next_key(u32 last_key);
+void rxe_mr_init(int access, struct rxe_mr *mr);
 void rxe_mr_init_dma(int access, struct rxe_mr *mr);
 int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 		     int access, struct rxe_mr *mr);
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
index da3dee520876..12cacc4f60f2 100644
@@ -45,7 +45,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length)
 	}
 }
 
-static void rxe_mr_init(int access, struct rxe_mr *mr)
+void rxe_mr_init(int access, struct rxe_mr *mr)
 {
 	u32 key = mr->elem.index << 8 | rxe_get_next_key(-1);
 
drivers/infiniband/sw/rxe/rxe_verbs.h +37 -0
index 3c1354f82283..e4656c7640f0 100644
@@ -126,6 +126,43 @@ struct rxe_comp_info {
 	u32			rnr_retry;
 };
 
+/* responder states */
+enum resp_states {
+	RESPST_NONE,
+	RESPST_GET_REQ,
+	RESPST_CHK_PSN,
+	RESPST_CHK_OP_SEQ,
+	RESPST_CHK_OP_VALID,
+	RESPST_CHK_RESOURCE,
+	RESPST_CHK_LENGTH,
+	RESPST_CHK_RKEY,
+	RESPST_EXECUTE,
+	RESPST_READ_REPLY,
+	RESPST_ATOMIC_REPLY,
+	RESPST_ATOMIC_WRITE_REPLY,
+	RESPST_PROCESS_FLUSH,
+	RESPST_COMPLETE,
+	RESPST_ACKNOWLEDGE,
+	RESPST_CLEANUP,
+	RESPST_DUPLICATE_REQUEST,
+	RESPST_ERR_MALFORMED_WQE,
+	RESPST_ERR_UNSUPPORTED_OPCODE,
+	RESPST_ERR_MISALIGNED_ATOMIC,
+	RESPST_ERR_PSN_OUT_OF_SEQ,
+	RESPST_ERR_MISSING_OPCODE_FIRST,
+	RESPST_ERR_MISSING_OPCODE_LAST_C,
+	RESPST_ERR_MISSING_OPCODE_LAST_D1E,
+	RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
+	RESPST_ERR_RNR,
+	RESPST_ERR_RKEY_VIOLATION,
+	RESPST_ERR_INVALIDATE_RKEY,
+	RESPST_ERR_LENGTH,
+	RESPST_ERR_CQ_OVERFLOW,
+	RESPST_ERROR,
+	RESPST_DONE,
+	RESPST_EXIT,
+};
+
 enum rdatm_res_state {
 	rdatm_res_state_next,
 	rdatm_res_state_new,
Signature · 1 line
2.43.0
#2 lore raw

PATCH v9 2/5 RDMA/rxe: Add page invalidation support

In reply to Daisuke Matsuda

On page invalidation, an MMU notifier callback is invoked to unmap DMA addresses and update the driver page table(umem_odp->dma_list). The callback is registered when an ODP-enabled MR is created.

---
 drivers/infiniband/sw/rxe/Makefile  |  2 ++
 drivers/infiniband/sw/rxe/rxe_loc.h |  3 +++
 drivers/infiniband/sw/rxe/rxe_odp.c | 38 +++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
drivers/infiniband/sw/rxe/Makefile +2 -0
index 5395a581f4bb..93134f1d1d0c 100644
@@ -23,3 +23,5 @@ rdma_rxe-y := \
 	rxe_task.o \
 	rxe_net.o \
 	rxe_hw_counters.o
+
+rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
drivers/infiniband/sw/rxe/rxe_loc.h +3 -0
index 1c78b9d36da4..0162ac9431c1 100644
@@ -182,4 +182,7 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp)
 	return rxe_wr_opcode_info[opcode].mask[qp->ibqp.qp_type];
 }
 
+/* rxe_odp.c */
+extern const struct mmu_interval_notifier_ops rxe_mn_ops;
+
 #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_odp.c new file +38 -0
new file mode 100644
index 000000000000..2be8066db012
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright (c) 2022-2023 Fujitsu Ltd. All rights reserved.
+ */
+
+#include <linux/hmm.h>
+
+#include <rdma/ib_umem_odp.h>
+
+#include "rxe.h"
+
+static bool rxe_ib_invalidate_range(struct mmu_interval_notifier *mni,
+				    const struct mmu_notifier_range *range,
+				    unsigned long cur_seq)
+{
+	struct ib_umem_odp *umem_odp =
+		container_of(mni, struct ib_umem_odp, notifier);
+	unsigned long start, end;
+
+	if (!mmu_notifier_range_blockable(range))
+		return false;
+
+	mutex_lock(&umem_odp->umem_mutex);
+	mmu_interval_set_seq(mni, cur_seq);
+
+	start = max_t(u64, ib_umem_start(umem_odp), range->start);
+	end = min_t(u64, ib_umem_end(umem_odp), range->end);
+
+	/* update umem_odp->dma_list */
+	ib_umem_odp_unmap_dma_pages(umem_odp, start, end);
+
+	mutex_unlock(&umem_odp->umem_mutex);
+	return true;
+}
+
+const struct mmu_interval_notifier_ops rxe_mn_ops = {
+	.invalidate = rxe_ib_invalidate_range,
+};
Signature · 1 line
2.43.0
#3 lore raw

PATCH v9 3/5 RDMA/rxe: Allow registering MRs for On-Demand Paging

In reply to Daisuke Matsuda

Allow userspace to register an ODP-enabled MR, in which case the flag IB_ACCESS_ON_DEMAND is passed to rxe_reg_user_mr(). However, there is no RDMA operation enabled right now. They will be supported later in the subsequent two patches.

rxe_odp_do_pagefault() is called to initialize an ODP-enabled MR. It syncs process address space from the CPU page table to the driver page table (dma_list/pfn_list in umem_odp) when called with RXE_PAGEFAULT_SNAPSHOT flag. Additionally, It can be used to trigger page fault when pages being accessed are not present or do not have proper read/write permissions, and possibly to prefetch pages in the future.

---
 drivers/infiniband/sw/rxe/rxe.c       |  7 +++
 drivers/infiniband/sw/rxe/rxe_loc.h   | 12 ++++
 drivers/infiniband/sw/rxe/rxe_mr.c    |  9 ++-
 drivers/infiniband/sw/rxe/rxe_odp.c   | 86 +++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c  | 15 +++--
 drivers/infiniband/sw/rxe/rxe_verbs.c |  5 +-
 6 files changed, 128 insertions(+), 6 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +7 -0
index 255677bc12b2..3ca73f8d96cc 100644
@@ -75,6 +75,13 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 			rxe->ndev->dev_addr);
 
 	rxe->max_ucontext			= RXE_MAX_UCONTEXT;
+
+	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
+		rxe->attr.kernel_cap_flags |= IBK_ON_DEMAND_PAGING;
+
+		/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */
+		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT;
+	}
 }
 
 /* initialize port attributes */
drivers/infiniband/sw/rxe/rxe_loc.h +12 -0
index 0162ac9431c1..1043240daa97 100644
@@ -185,4 +185,16 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp)
 /* rxe_odp.c */
 extern const struct mmu_interval_notifier_ops rxe_mn_ops;
 
+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
+int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
+			 u64 iova, int access_flags, struct rxe_mr *mr);
+#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
+static inline int
+rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
+		     int access_flags, struct rxe_mr *mr)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
+
 #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +8 -1
index 12cacc4f60f2..190225a34139 100644
@@ -323,7 +323,10 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 		return err;
 	}
 
-	return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
+	if (mr->umem->is_odp)
+		return -EOPNOTSUPP;
+	else
+		return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
 }
 
 /* copy data in or out of a wqe, i.e. sg list
@@ -532,6 +535,10 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value)
 	struct page *page;
 	u64 *va;
 
+	/* ODP is not supported right now. WIP. */
+	if (mr->umem->is_odp)
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+
 	/* See IBA oA19-28 */
 	if (unlikely(mr->state != RXE_MR_STATE_VALID)) {
 		rxe_dbg_mr(mr, "mr not in valid state\n");
drivers/infiniband/sw/rxe/rxe_odp.c +86 -0

89 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_resp.c +11 -4
index c11ab280551a..e703a3ab82d4 100644
@@ -649,6 +649,10 @@ static enum resp_states process_flush(struct rxe_qp *qp,
 	struct rxe_mr *mr = qp->resp.mr;
 	struct resp_res *res = qp->resp.res;
 
+	/* ODP is not supported right now. WIP. */
+	if (mr->umem->is_odp)
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+
 	/* oA19-14, oA19-15 */
 	if (res && res->replay)
 		return RESPST_ACKNOWLEDGE;
@@ -702,10 +706,13 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 	if (!res->replay) {
 		u64 iova = qp->resp.va + qp->resp.offset;
 
-		err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
-					  atmeth_comp(pkt),
-					  atmeth_swap_add(pkt),
-					  &res->atomic.orig_val);
+		if (mr->umem->is_odp)
+			err = RESPST_ERR_UNSUPPORTED_OPCODE;
+		else
+			err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
+						  atmeth_comp(pkt),
+						  atmeth_swap_add(pkt),
+						  &res->atomic.orig_val);
 		if (err)
 			return err;
 
drivers/infiniband/sw/rxe/rxe_verbs.c +4 -1
index 5c18f7e342f2..13064302d766 100644
@@ -1278,7 +1278,10 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, u64 start,
 	mr->ibmr.pd = ibpd;
 	mr->ibmr.device = ibpd->device;
 
-	err = rxe_mr_init_user(rxe, start, length, access, mr);
+	if (access & IB_ACCESS_ON_DEMAND)
+		err = rxe_odp_mr_init_user(rxe, start, length, iova, access, mr);
+	else
+		err = rxe_mr_init_user(rxe, start, length, access, mr);
 	if (err) {
 		rxe_dbg_mr(mr, "reg_user_mr failed, err = %d\n", err);
 		goto err_cleanup;
Signature · 1 line
2.43.0
#4 lore raw

PATCH v9 4/5 RDMA/rxe: Add support for Send/Recv/Write/Read with ODP

In reply to Daisuke Matsuda

rxe_mr_copy() is used widely to copy data to/from a user MR. requester uses it to load payloads of requesting packets; responder uses it to process Send, Write, and Read operaetions; completer uses it to copy data from response packets of Read and Atomic operations to a user MR.

Allow these operations to be used with ODP by adding a subordinate function rxe_odp_mr_copy(). It is comprised of the following steps:
1. Check the driver page table(umem_odp->dma_list) to see if pages being accessed are present with appropriate permission.
2. If necessary, trigger page fault to map the pages.
3. Convert their user space addresses to kernel logical addresses using

    PFNs in the driver page table(umem_odp->pfn_list).

4. Execute data copy to/from the pages.

---
 drivers/infiniband/sw/rxe/rxe.c     |  10 +++
 drivers/infiniband/sw/rxe/rxe_loc.h |   7 ++
 drivers/infiniband/sw/rxe/rxe_mr.c  |   2 +-
 drivers/infiniband/sw/rxe/rxe_odp.c | 131 ++++++++++++++++++++++++++++
 4 files changed, 149 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe.c +10 -0
index 3ca73f8d96cc..ea643ebf9667 100644
@@ -81,6 +81,16 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 
 		/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */
 		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT;
+
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_SEND;
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_RECV;
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
+
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SEND;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_RECV;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_WRITE;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
 	}
 }
 
drivers/infiniband/sw/rxe/rxe_loc.h +7 -0
index 1043240daa97..7a735108d475 100644
@@ -188,6 +188,8 @@ extern const struct mmu_interval_notifier_ops rxe_mn_ops;
 #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
 int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 			 u64 iova, int access_flags, struct rxe_mr *mr);
+int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
+		    enum rxe_mr_copy_dir dir);
 #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 static inline int
 rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -195,6 +197,11 @@ rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 {
 	return -EOPNOTSUPP;
 }
+static inline int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
+				  int length, enum rxe_mr_copy_dir dir)
+{
+	return -EOPNOTSUPP;
+}
 #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 
 #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
index 190225a34139..6cd668a8dfb2 100644
@@ -324,7 +324,7 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 	}
 
 	if (mr->umem->is_odp)
-		return -EOPNOTSUPP;
+		return rxe_odp_mr_copy(mr, iova, addr, length, dir);
 	else
 		return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
 }
drivers/infiniband/sw/rxe/rxe_odp.c +131 -0

134 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw

PATCH v9 5/5 RDMA/rxe: Add support for the traditional Atomic operations with ODP

In reply to Daisuke Matsuda

Enable 'fetch and add' and 'compare and swap' operations to be used with ODP. This is comprised of the following steps:
1. Check the driver page table(umem_odp->dma_list) to see if the target page is both readable and writable.
2. If not, then trigger page fault to map the page.
3. Convert its user space address to a kernel logical address using PFNs

    in the driver page table(umem_odp->pfn_list).

4. Execute the operation.

---
 drivers/infiniband/sw/rxe/rxe.c      |  1 +
 drivers/infiniband/sw/rxe/rxe_loc.h  | 11 +++++
 drivers/infiniband/sw/rxe/rxe_mr.c   |  2 +-
 drivers/infiniband/sw/rxe/rxe_odp.c  | 69 ++++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c |  5 +-
 5 files changed, 86 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +1 -0
index ea643ebf9667..08c69c637663 100644
@@ -90,6 +90,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_RECV;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_WRITE;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
 	}
 }
drivers/infiniband/sw/rxe/rxe_loc.h +11 -0
index 7a735108d475..a45ff6236613 100644
@@ -81,6 +81,9 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key);
 int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe);
 void rxe_mr_cleanup(struct rxe_pool_elem *elem);
 
+/* defined in rxe_mr.c; used in rxe_mr.c and rxe_odp.c */
+extern spinlock_t atomic_ops_lock;
+
 /* rxe_mw.c */
 int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata);
 int rxe_dealloc_mw(struct ib_mw *ibmw);
@@ -190,6 +193,8 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 			 u64 iova, int access_flags, struct rxe_mr *mr);
 int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 		    enum rxe_mr_copy_dir dir);
+int rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+			 u64 compare, u64 swap_add, u64 *orig_val);
 #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 static inline int
 rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -202,6 +207,12 @@ static inline int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 {
 	return -EOPNOTSUPP;
 }
+static inline int
+rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+		     u64 compare, u64 swap_add, u64 *orig_val)
+{
+	return RESPST_ERR_UNSUPPORTED_OPCODE;
+}
 #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 
 #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
index 6cd668a8dfb2..868d2f0b74e9 100644
@@ -469,7 +469,7 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 }
 
 /* Guarantee atomicity of atomic operations at the machine level. */
-static DEFINE_SPINLOCK(atomic_ops_lock);
+DEFINE_SPINLOCK(atomic_ops_lock);
 
 int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
 			u64 compare, u64 swap_add, u64 *orig_val)
drivers/infiniband/sw/rxe/rxe_odp.c +69 -0

72 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_resp.c +4 -1
index e703a3ab82d4..54ba9ee1acc5 100644
@@ -707,7 +707,10 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 		u64 iova = qp->resp.va + qp->resp.offset;
 
 		if (mr->umem->is_odp)
-			err = RESPST_ERR_UNSUPPORTED_OPCODE;
+			err = rxe_odp_atomic_op(mr, iova, pkt->opcode,
+						atmeth_comp(pkt),
+						atmeth_swap_add(pkt),
+						&res->atomic.orig_val);
 		else
 			err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
 						  atmeth_comp(pkt),
Signature · 1 line
2.43.0

A message that is not in the archive (sent off-list or not yet received): CAHjRaAeXCC+AAV+Ne0cJMpZJYxbD8ox28kp966wkdVJLJdSC_g@mail.gmail.com

#7 lore raw
Quoted 1 line · Joe Klein

We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.

Here are the details of the issue:
===== test log =====

ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3
ubuntu@rdma-dev:~$ cd rdma-core
ubuntu@rdma-dev:~/rdma-core$ uname -r
6.13.0-rc1+
ubuntu@rdma-dev:~/rdma-core$ pwd
/home/ubuntu/rdma-core
ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py
..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True>
  def __exit__(self, typ, value, traceback):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
ssssTraceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
s....ssSegmentation fault (core dumped)
===========

=====dmesg=====

[  147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4
[  147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10
[  147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9
[  147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10
[  147.524080] rxe_ens3: rxe_create_cq: returned err = -22
[  147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22
[  147.605719] rxe_ens3: rxe_create_cq: returned err = -95
[  147.606454] rxe_ens3: rxe_create_cq: returned err = -22
[  148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10
[  148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10
[  148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10
[  148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10
[  149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2)
[  149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be 01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89

=====

If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are seeing. I would also appreciate any help you can offer in fixing the upstream issue.

Thanks,
Daisuke

#8 lore raw
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein

We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if

others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Quoted 8 lines

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.

It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.

I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################

ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
* master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
----------------------------------------------------------------------
Ran 321 tests in 3.599s

OK (skipped=211)

ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
sssssssss..ss....s.s
----------------------------------------------------------------------
Ran 20 tests in 0.269s

OK (skipped=13)
####################

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

Thanks,
Daisuke

Quoted 84 lines

Here are the details of the issue:
===== test log =====

ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3
ubuntu@rdma-dev:~$ cd rdma-core
ubuntu@rdma-dev:~/rdma-core$ uname -r
6.13.0-rc1+
ubuntu@rdma-dev:~/rdma-core$ pwd
/home/ubuntu/rdma-core
ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py
..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov'
mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou'
mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO
name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True>
  def __exit__(self, typ, value, traceback):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ssss..............ssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/
usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci'
mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
ssssTraceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
s....ssSegmentation fault (core dumped)
===========

=====dmesg=====

[  147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4
[  147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10
[  147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9
[  147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10
[  147.524080] rxe_ens3: rxe_create_cq: returned err = -22
[  147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22
[  147.605719] rxe_ens3: rxe_create_cq: returned err = -95
[  147.606454] rxe_ens3: rxe_create_cq: returned err = -22
[  148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10
[  148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10
[  148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10
[  148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10
[  149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in
libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2)
[  149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be

01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89
=====

If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are seeing.
I would also appreciate any help you can offer in fixing the upstream issue.

Thanks,
Daisuke

#9 lore raw
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein

We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if

others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Quoted 8 lines

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.

It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.

I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################

ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
* master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
----------------------------------------------------------------------
Ran 321 tests in 3.599s

OK (skipped=211)

ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
sssssssss..ss....s.s
----------------------------------------------------------------------
Ran 20 tests in 0.269s
Quoted 5 lines

OK (skipped=13)
####################

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

Quoted 2 lines

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?

Jason

#10 lore raw
Quoted 48 lines · Jason Gunthorpe
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein

We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if

others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Quoted 8 lines

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.

It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.

I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################

ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
* master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
----------------------------------------------------------------------
Ran 321 tests in 3.599s

OK (skipped=211)

ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
sssssssss..ss....s.s
----------------------------------------------------------------------
Ran 20 tests in 0.269s
Quoted 5 lines

OK (skipped=13)
####################

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

Quoted 7 lines
Quoted 2 lines

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?

Will make tests and let you know the result very soon.

Quoted 2 lines

Jason

#11 lore raw
Quoted 53 lines · Joe Klein
Quoted 51 lines · Jason Gunthorpe
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein

We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if

others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Quoted 8 lines

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.

It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.

I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################

ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
* master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
----------------------------------------------------------------------
Ran 321 tests in 3.599s

OK (skipped=211)

ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
sssssssss..ss....s.s
----------------------------------------------------------------------
Ran 20 tests in 0.269s
Quoted 5 lines

OK (skipped=13)
####################

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

Quoted 2 lines

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

Quoted 6 lines
Quoted 4 lines

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?

Will make tests and let you know the result very soon.

Need time to complete the test scenario configurations.

We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.

Quoted 3 lines
Quoted 3 lines

Jason

#12 lore raw
Quoted 13 lines · Joe Klein
Quoted 13 lines
Quoted 12 lines
Quoted 2 lines

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

Quoted 2 lines

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?

Quoted 7 lines
Quoted 1 line

Will make tests and let you know the result very soon.

Need time to complete the test scenario configurations.

We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.

Can you share your logs so that Daisuke can resolve whatever the problem is?

Jason

#13 lore raw

On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:

Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?

Will make tests and let you know the result very soon.

Quoted 8 lines
Quoted 5 lines

Need time to complete the test scenario configurations.

We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.

Can you share your logs so that Daisuke can resolve whatever the problem is?

Since Joe does not provide any specific information, it would be nice if someone familiar with rxe like Zhu Yanjun, Li Zhijian or Bob Pearson could find the time to test and review the patch.
Cf. https://github.com/ddmatsu/linux/tree/odp_v9

However, to be honest, I do not think Joe's report is valid. It is not normal to complain other's patches without providing any details. It makes difficult to assess the validity of the claim, and blocks any further discussions forever.

The testcases in libibverbs pass as I attached the log. Other than that, I verified my patches work with perftest (with ODP enabled/disabled). I believe I did the test properly and the patches are ready to be merged.

Daisuke

Quoted 2 lines

Jason

#14 lore raw
Quoted 3 lines · Daisuke Matsuda (Fujitsu)

However, to be honest, I do not think Joe's report is valid. It is not normal to complain other's patches without providing any details. It makes difficult to assess the validity of the claim, and blocks any further discussions forever.

Yes, I agree, let's move past anything without concrete actionable details.

Jason

#15 lore raw
In reply to Daisuke Matsuda (Fujitsu)

On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):

Quoted 10 lines

On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:

Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.

On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases, so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.

Could it be possible the update of python version leads to the failure? I saw strange warnings while running build.sh, which did not appear on Ubuntu 22.04.5. I would like to hear from those who have tried building rdma-core on Ubuntu 24.04.1 or with Python 3.12.3.

[422/607] Cythonizing /home/ubuntu/rdma-core/pyverbs/pd.pyx
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:102:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:124:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:142:30: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:158:61: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:180:46: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:272:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:279:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:12:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:21:47: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:16:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:17:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:22:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:32:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/mr.pxd:21:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:28:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:29:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:22:50: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:11:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:13:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:18:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:20:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:35:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:37:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:37:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:38:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:45:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:27:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:38:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword. [607/607] Linking C shared library python/pyverbs/providers/mlx5/mlx5dv.cpython-312-x86_64-linux-gnu.so

Daisuke

#16 lore raw
Quoted 12 lines · Daisuke Matsuda (Fujitsu)

On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):

Quoted 10 lines

On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:

Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.

Quoted 6 lines

On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases, so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.

Could it be possible the update of python version leads to the failure?

Or a cython update? It certainly could..

So maybe there is nothing to worry about kernel side if the test case passes on older OS userspace

Jason

maintainer Applied
#17 lore raw
In reply to Daisuke Matsuda
Quoted 7 lines · Daisuke Matsuda

Daisuke Matsuda (5):
RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP RDMA/rxe: Add page invalidation support
RDMA/rxe: Allow registering MRs for On-Demand Paging RDMA/rxe: Add support for Send/Recv/Write/Read with ODP RDMA/rxe: Add support for the traditional Atomic operations with ODP

Quoted 11 lines
 drivers/infiniband/sw/rxe/Makefile    |   2 +
 drivers/infiniband/sw/rxe/rxe.c       |  18 ++
 drivers/infiniband/sw/rxe/rxe.h       |  37 ---
 drivers/infiniband/sw/rxe/rxe_loc.h   |  34 +++
 drivers/infiniband/sw/rxe/rxe_mr.c    |  13 +-
 drivers/infiniband/sw/rxe/rxe_odp.c   | 324 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c  |  18 +-
 drivers/infiniband/sw/rxe/rxe_verbs.c |   5 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h |  37 +++
 9 files changed, 443 insertions(+), 45 deletions(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c

Applied to for-next, thanks

Jason