On-Demand Paging on SoftRoCE
Subsystems (2)
v9 cover On-Demand Paging on SoftRoCE
This patch series implements the On-Demand Paging feature on SoftRoCE(rxe) driver, which has been available only in mlx5 driver[1] so far.
This series has been blocked because of the hang issue of srp 002 test[2], which was believed to be caused after applying the commit 9b4b7c1f9f54 ("RDMA/rxe: Add workqueue support for rxe tasks"). My patches are dependent on the commit because the ODP feature requires sleeping in kernel space, and it is impossible with the former tasklet implementation.
According to the original reporter[3], the hang issue is already gone in v6.10. Additionally, tasklet is marked deprecated[4]. I think the rxe driver is ready to accept this series since there is no longer any reason to consider reverting back to the old tasklet.
I omitted some contents like the motive behind this series from the cover- letter. Please see the cover letter of v3 for more details[5].
[Overview]
When applications register a memory region(MR), RDMA drivers normally pin pages in the MR so that physical addresses are never changed during RDMA communication. This requires the MR to fit in physical memory and inevitably leads to memory pressure. On the other hand, On-Demand Paging (ODP) allows applications to register MRs without pinning pages. They are paged-in when the driver requires and paged-out when the OS reclaims. As a result, it is possible to register a large MR that does not fit in physical memory without taking up so much physical memory.
[How does ODP work?]
"struct ib_umem_odp" is used to manage pages. It is created for each ODP-enabled MR on its registration. This struct holds a pair of arrays (dma_list/pfn_list) that serve as a driver page table. DMA addresses and PFNs are stored in the driver page table. They are updated on page-in and page-out, both of which use the common interfaces in the ib_uverbs layer.
Page-in can occur when requester, responder or completer access an MR in order to process RDMA operations. If they find that the pages being accessed are not present on physical memory or requisite permissions are not set on the pages, they provoke page fault to make the pages present with proper permissions and at the same time update the driver page table. After confirming the presence of the pages, they execute memory access such as read, write or atomic operations.
Page-out is triggered by page reclaim or filesystem events (e.g. metadata update of a file that is being used as an MR). When creating an ODP-enabled MR, the driver registers an MMU notifier callback. When the kernel issues a page invalidation notification, the callback is provoked to unmap DMA addresses and update the driver page table. After that, the kernel releases the pages.
[Supported operations]
All traditional operations are supported on RC connection. The new Atomic write[6] and RDMA Flush[7] operations are not included in this patchset. I will post them later after this patchset is merged. On UD connection, Send, Recv, and SRQ-Recv are supported.
[How to test ODP?]
There are only a few resources available for testing. pyverbs testcases in rdma-core and perftest[8] are recommendable ones. Other than them, the ibv_rc_pingpong command can also be used for testing. Note that you may have to build perftest from upstream because old versions do not handle ODP capabilities correctly.
The latest ODP tree is available from github:
https://github.com/ddmatsu/linux/tree/odp_v9
[Future work]
My next work is to enable the new Atomic write[6] and RDMA Flush[7] operations with ODP. After that, I am going to implement the prefetch feature. It allows applications to trigger page fault using ibv_advise_mr(3) to optimize performance. Some existing software like librpma[9] use this feature. Additionally, I think we can also add the implicit ODP feature in the future.
[1] Understanding On Demand Paging (ODP)
https://enterprise-support.nvidia.com/s/article/understanding-on-demand-paging--odp-x
[2] [bug report] blktests srp/002 hang
https://lore.kernel.org/linux-rdma/dsg6rd66tyiei32zaxs6ddv5ebefr5vtxjwz6d2ewqrcwisogl@ge7jzan7dg5u/T/
[3] blktests failures with v6.10-rc1 kernel
https://lore.kernel.org/linux-block/wnucs5oboi4flje5yvtea7puvn6zzztcnlrfz3lpzlwgblrxgw@7wvqdzioejgl/
[4] [00/15] ethernet: Convert from tasklet to BH workqueue
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/
[5] [PATCH for-next v3 0/7] On-Demand Paging on SoftRoCE
https://lore.kernel.org/lkml/[email protected]/
[6] [PATCH v7 0/8] RDMA/rxe: Add atomic write operation
https://lore.kernel.org/linux-rdma/[email protected]/
[7] [for-next PATCH 00/10] RDMA/rxe: Add RDMA FLUSH operation
https://lore.kernel.org/lkml/[email protected]/
[8] linux-rdma/perftest: Infiniband Verbs Performance Tests
https://github.com/linux-rdma/perftest
[9] librpma: Remote Persistent Memory Access Library
https://github.com/pmem/rpma
v8->v9:
1) Addressed comments from Jason Gunthorpe
cf. https://lore.kernel.org/lkml/[email protected]/
cf. https://lore.kernel.org/lkml/[email protected]/
2) rebased to 6.13.0-rc1
v7->v8:
1) Dropped the first patch because the same change was made by Bob Pearson. cf. https://github.com/torvalds/linux/commit/23bc06af547f2ca3b7d345e09fd8d04575406274
2) Rebased to 6.12.1-rc2
v6->v7:
1) Rebased to 6.6.0
2) Disabled using hugepages with ODP
3) Addressed comments on v6 from Jason and Zhu
cf. https://lore.kernel.org/lkml/[email protected]/
v5->v6:
Fixed the implementation according to Jason's suggestions
cf. https://lore.kernel.org/all/[email protected]/
cf. https://lore.kernel.org/all/[email protected]/
v4->v5:
1) Rebased to 6.4.0-rc2+
2) Changed to schedule all works on responder and completer to workqueue
v3->v4:
1) Re-designed functions that access MRs to use the MR xarray.
2) Rebased onto the latest jgg-for-next tree.
v2->v3:
1) Removed a patch that changes the common ib_uverbs layer.
2) Re-implemented patches for conversion to workqueue.
3) Fixed compile errors (happened when CONFIG_INFINIBAND_ON_DEMAND_PAGING=n).
4) Fixed some functions that returned incorrect errors.
5) Temporarily disabled ODP for RDMA Flush and Atomic Write.
v1->v2:
1) Fixed a crash issue reported by Haris Iqbal.
2) Tried to make lock patters clearer as pointed out by Romanovsky.
3) Minor clean ups and fixes.
Daisuke Matsuda (5):
RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP RDMA/rxe: Add page invalidation support
RDMA/rxe: Allow registering MRs for On-Demand Paging RDMA/rxe: Add support for Send/Recv/Write/Read with ODP RDMA/rxe: Add support for the traditional Atomic operations with ODP
drivers/infiniband/sw/rxe/Makefile | 2 + drivers/infiniband/sw/rxe/rxe.c | 18 ++ drivers/infiniband/sw/rxe/rxe.h | 37 --- drivers/infiniband/sw/rxe/rxe_loc.h | 34 +++ drivers/infiniband/sw/rxe/rxe_mr.c | 13 +- drivers/infiniband/sw/rxe/rxe_odp.c | 324 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 18 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 5 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 37 +++ 9 files changed, 443 insertions(+), 45 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
Signature · 1 line
2.43.0
PATCH v9 1/5 RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP
rxe_mr_init() and resp_states are going to be used in rxe_odp.c, which is to be created in the subsequent patch.
--- drivers/infiniband/sw/rxe/rxe.h | 37 --------------------------- drivers/infiniband/sw/rxe/rxe_loc.h | 1 + drivers/infiniband/sw/rxe/rxe_mr.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 37 +++++++++++++++++++++++++++ 4 files changed, 39 insertions(+), 38 deletions(-)
drivers/infiniband/sw/rxe/rxe.h +0 -37
@@ -100,43 +100,6 @@#define rxe_info_mw (mw, fmt, ...) ibdev_info_ratelimited((mw)->ibmw.device, \ "mw#%d %s: " fmt, (mw)->elem.index, __func__, ##__VA_ARGS__) -/* responder states */ -enum resp_states { -RESPST_NONE , -RESPST_GET_REQ , -RESPST_CHK_PSN , -RESPST_CHK_OP_SEQ , -RESPST_CHK_OP_VALID , -RESPST_CHK_RESOURCE , -RESPST_CHK_LENGTH , -RESPST_CHK_RKEY , -RESPST_EXECUTE , -RESPST_READ_REPLY , -RESPST_ATOMIC_REPLY , -RESPST_ATOMIC_WRITE_REPLY , -RESPST_PROCESS_FLUSH , -RESPST_COMPLETE , -RESPST_ACKNOWLEDGE , -RESPST_CLEANUP , -RESPST_DUPLICATE_REQUEST , -RESPST_ERR_MALFORMED_WQE , -RESPST_ERR_UNSUPPORTED_OPCODE , -RESPST_ERR_MISALIGNED_ATOMIC , -RESPST_ERR_PSN_OUT_OF_SEQ , -RESPST_ERR_MISSING_OPCODE_FIRST , -RESPST_ERR_MISSING_OPCODE_LAST_C , -RESPST_ERR_MISSING_OPCODE_LAST_D1E , -RESPST_ERR_TOO_MANY_RDMA_ATM_REQ , -RESPST_ERR_RNR , -RESPST_ERR_RKEY_VIOLATION , -RESPST_ERR_INVALIDATE_RKEY , -RESPST_ERR_LENGTH , -RESPST_ERR_CQ_OVERFLOW , -RESPST_ERROR , -RESPST_DONE , -RESPST_EXIT , -}; -void rxe_set_mtu (struct rxe_dev *rxe,unsigned int dev_mtu);int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name);
drivers/infiniband/sw/rxe/rxe_loc.h +1 -0
@@ -58,6 +58,7 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);/* rxe_mr.c */ u8 rxe_get_next_key (u32 last_key); +void rxe_mr_init (int access,struct rxe_mr *mr);void rxe_mr_init_dma (int access,struct rxe_mr *mr);int rxe_mr_init_user (struct rxe_dev *rxe,u64 start,u64 length,int access,struct rxe_mr *mr);
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
@@ -45,7 +45,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) } } -static void rxe_mr_init (int access,struct rxe_mr *mr) +void rxe_mr_init (int access,struct rxe_mr *mr) {u32 key = mr->elem.index <<8 |rxe_get_next_key (-1 );
drivers/infiniband/sw/rxe/rxe_verbs.h +37 -0
@@ -126,6 +126,43 @@ struct rxe_comp_info {u32 rnr_retry; }; +/* responder states */ +enum resp_states { +RESPST_NONE , +RESPST_GET_REQ , +RESPST_CHK_PSN , +RESPST_CHK_OP_SEQ , +RESPST_CHK_OP_VALID , +RESPST_CHK_RESOURCE , +RESPST_CHK_LENGTH , +RESPST_CHK_RKEY , +RESPST_EXECUTE , +RESPST_READ_REPLY , +RESPST_ATOMIC_REPLY , +RESPST_ATOMIC_WRITE_REPLY , +RESPST_PROCESS_FLUSH , +RESPST_COMPLETE , +RESPST_ACKNOWLEDGE , +RESPST_CLEANUP , +RESPST_DUPLICATE_REQUEST , +RESPST_ERR_MALFORMED_WQE , +RESPST_ERR_UNSUPPORTED_OPCODE , +RESPST_ERR_MISALIGNED_ATOMIC , +RESPST_ERR_PSN_OUT_OF_SEQ , +RESPST_ERR_MISSING_OPCODE_FIRST , +RESPST_ERR_MISSING_OPCODE_LAST_C , +RESPST_ERR_MISSING_OPCODE_LAST_D1E , +RESPST_ERR_TOO_MANY_RDMA_ATM_REQ , +RESPST_ERR_RNR , +RESPST_ERR_RKEY_VIOLATION , +RESPST_ERR_INVALIDATE_RKEY , +RESPST_ERR_LENGTH , +RESPST_ERR_CQ_OVERFLOW , +RESPST_ERROR , +RESPST_DONE , +RESPST_EXIT , +}; +enum rdatm_res_state { rdatm_res_state_next, rdatm_res_state_new,
Signature · 1 line
2.43.0
PATCH v9 2/5 RDMA/rxe: Add page invalidation support
In reply to Daisuke MatsudaOn page invalidation, an MMU notifier callback is invoked to unmap DMA addresses and update the driver page table(umem_odp->dma_list). The callback is registered when an ODP-enabled MR is created.
--- drivers/infiniband/sw/rxe/Makefile | 2 ++ drivers/infiniband/sw/rxe/rxe_loc.h | 3 +++ drivers/infiniband/sw/rxe/rxe_odp.c | 38 +++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+) create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
drivers/infiniband/sw/rxe/Makefile +2 -0
@@ -23,3 +23,5 @@ rdma_rxe-y := \ rxe_task.o \ rxe_net.o \ rxe_hw_counters.o + +rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
drivers/infiniband/sw/rxe/rxe_loc.h +3 -0
@@ -182,4 +182,7 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp) return rxe_wr_opcode_info[opcode].mask[qp->ibqp.qp_type]; } +/* rxe_odp.c */ +extern const struct mmu_interval_notifier_ops rxe_mn_ops; +#endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_odp.c new file +38 -0
@@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* +* Copyright (c) 2022-2023 Fujitsu Ltd. All rights reserved. +*/ + +#include <linux/hmm.h> + +#include <rdma/ib_umem_odp.h> + +#include "rxe.h" + +static bool rxe_ib_invalidate_range (struct mmu_interval_notifier *mni, +const struct mmu_notifier_range *range, +unsigned long cur_seq) +{ +struct ib_umem_odp *umem_odp = + container_of(mni, struct ib_umem_odp, notifier); +unsigned long start, end; + +if (!mmu_notifier_range_blockable (range)) +return false; + +mutex_lock (&umem_odp->umem_mutex); +mmu_interval_set_seq (mni, cur_seq); + + start =max_t (u64,ib_umem_start (umem_odp), range->start); + end =min_t (u64,ib_umem_end (umem_odp), range->end); + +/* update umem_odp->dma_list */ +ib_umem_odp_unmap_dma_pages (umem_odp, start, end); + +mutex_unlock (&umem_odp->umem_mutex); +return true; +} + +const struct mmu_interval_notifier_ops rxe_mn_ops = { + .invalidate = rxe_ib_invalidate_range, +};
Signature · 1 line
2.43.0
PATCH v9 3/5 RDMA/rxe: Allow registering MRs for On-Demand Paging
In reply to Daisuke MatsudaAllow userspace to register an ODP-enabled MR, in which case the flag IB_ACCESS_ON_DEMAND is passed to rxe_reg_user_mr(). However, there is no RDMA operation enabled right now. They will be supported later in the subsequent two patches.
rxe_odp_do_pagefault() is called to initialize an ODP-enabled MR. It syncs process address space from the CPU page table to the driver page table (dma_list/pfn_list in umem_odp) when called with RXE_PAGEFAULT_SNAPSHOT flag. Additionally, It can be used to trigger page fault when pages being accessed are not present or do not have proper read/write permissions, and possibly to prefetch pages in the future.
--- drivers/infiniband/sw/rxe/rxe.c | 7 +++ drivers/infiniband/sw/rxe/rxe_loc.h | 12 ++++ drivers/infiniband/sw/rxe/rxe_mr.c | 9 ++- drivers/infiniband/sw/rxe/rxe_odp.c | 86 +++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 15 +++-- drivers/infiniband/sw/rxe/rxe_verbs.c | 5 +- 6 files changed, 128 insertions(+), 6 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +7 -0
@@ -75,6 +75,13 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext =RXE_MAX_UCONTEXT ; + +if (IS_ENABLED (CONFIG_INFINIBAND_ON_DEMAND_PAGING )) { + rxe->attr.kernel_cap_flags |=IBK_ON_DEMAND_PAGING ; + +/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */ + rxe->attr.odp_caps.general_caps |=IB_ODP_SUPPORT ; + } }/* initialize port attributes */
drivers/infiniband/sw/rxe/rxe_loc.h +12 -0
@@ -185,4 +185,16 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp)/* rxe_odp.c */ extern const struct mmu_interval_notifier_ops rxe_mn_ops; +#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING +int rxe_odp_mr_init_user (struct rxe_dev *rxe,u64 start,u64 length, +u64 iova,int access_flags,struct rxe_mr *mr); +#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ +static inline int +rxe_odp_mr_init_user (struct rxe_dev *rxe,u64 start,u64 length,u64 iova, +int access_flags,struct rxe_mr *mr) +{ +return -EOPNOTSUPP ; +} +#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ +#endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +8 -1
@@ -323,7 +323,10 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,return err; } -return rxe_mr_copy_xarray (mr, iova, addr, length, dir); +if (mr->umem->is_odp) +return -EOPNOTSUPP ; +else +return rxe_mr_copy_xarray (mr, iova, addr, length, dir); }/* copy data in or out of a wqe, i.e. sg list
@@ -532,6 +535,10 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value)struct page *page;u64 *va; +/* ODP is not supported right now. WIP. */ +if (mr->umem->is_odp) +return RESPST_ERR_UNSUPPORTED_OPCODE ; +/* See IBA oA19-28 */ if (unlikely (mr->state !=RXE_MR_STATE_VALID )) {rxe_dbg_mr (mr,"mr not in valid state\n" );
drivers/infiniband/sw/rxe/rxe_odp.c +86 -0
89 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_resp.c +11 -4
@@ -649,6 +649,10 @@ static enum resp_states process_flush(struct rxe_qp *qp,struct rxe_mr *mr = qp->resp.mr;struct resp_res *res = qp->resp.res; +/* ODP is not supported right now. WIP. */ +if (mr->umem->is_odp) +return RESPST_ERR_UNSUPPORTED_OPCODE ; +/* oA19-14, oA19-15 */ if (res && res->replay)return RESPST_ACKNOWLEDGE ;
@@ -702,10 +706,13 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,if (!res->replay) {u64 iova = qp->resp.va + qp->resp.offset; - err =rxe_mr_do_atomic_op (mr, iova, pkt->opcode, -atmeth_comp (pkt), -atmeth_swap_add (pkt), - &res->atomic.orig_val); +if (mr->umem->is_odp) + err =RESPST_ERR_UNSUPPORTED_OPCODE ; +else + err =rxe_mr_do_atomic_op (mr, iova, pkt->opcode, +atmeth_comp (pkt), +atmeth_swap_add (pkt), + &res->atomic.orig_val);if (err)return err;
drivers/infiniband/sw/rxe/rxe_verbs.c +4 -1
@@ -1278,7 +1278,10 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, u64 start, mr->ibmr.pd = ibpd; mr->ibmr.device = ibpd->device; - err =rxe_mr_init_user (rxe, start, length, access, mr); +if (access &IB_ACCESS_ON_DEMAND ) + err =rxe_odp_mr_init_user (rxe, start, length, iova, access, mr); +else + err =rxe_mr_init_user (rxe, start, length, access, mr);if (err) {rxe_dbg_mr (mr,"reg_user_mr failed, err = %d\n" , err); goto err_cleanup;
Signature · 1 line
2.43.0
PATCH v9 4/5 RDMA/rxe: Add support for Send/Recv/Write/Read with ODP
In reply to Daisuke Matsudarxe_mr_copy() is used widely to copy data to/from a user MR. requester uses it to load payloads of requesting packets; responder uses it to process Send, Write, and Read operaetions; completer uses it to copy data from response packets of Read and Atomic operations to a user MR.
Allow these operations to be used with ODP by adding a subordinate function rxe_odp_mr_copy(). It is comprised of the following steps:
1. Check the driver page table(umem_odp->dma_list) to see if pages being accessed are present with appropriate permission.
2. If necessary, trigger page fault to map the pages.
3. Convert their user space addresses to kernel logical addresses using
PFNs in the driver page table(umem_odp->pfn_list).
4. Execute data copy to/from the pages.
--- drivers/infiniband/sw/rxe/rxe.c | 10 +++ drivers/infiniband/sw/rxe/rxe_loc.h | 7 ++ drivers/infiniband/sw/rxe/rxe_mr.c | 2 +- drivers/infiniband/sw/rxe/rxe_odp.c | 131 ++++++++++++++++++++++++++++ 4 files changed, 149 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe.c +10 -0
@@ -81,6 +81,16 @@ static void rxe_init_device_param(struct rxe_dev *rxe)/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */ rxe->attr.odp_caps.general_caps |=IB_ODP_SUPPORT ; + + rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |=IB_ODP_SUPPORT_SEND ; + rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |=IB_ODP_SUPPORT_RECV ; + rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; + + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SEND ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_RECV ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_WRITE ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_READ ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; } }
drivers/infiniband/sw/rxe/rxe_loc.h +7 -0
@@ -188,6 +188,8 @@ extern const struct mmu_interval_notifier_ops rxe_mn_ops;#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING int rxe_odp_mr_init_user (struct rxe_dev *rxe,u64 start,u64 length,u64 iova,int access_flags,struct rxe_mr *mr); +int rxe_odp_mr_copy (struct rxe_mr *mr,u64 iova,void *addr,int length, +enum rxe_mr_copy_dir dir);#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ static inline int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -195,6 +197,11 @@ rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, {return -EOPNOTSUPP ; } +static inline int rxe_odp_mr_copy (struct rxe_mr *mr,u64 iova,void *addr, +int length,enum rxe_mr_copy_dir dir) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
@@ -324,7 +324,7 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, }if (mr->umem->is_odp) -return -EOPNOTSUPP ; +return rxe_odp_mr_copy (mr, iova, addr, length, dir);else return rxe_mr_copy_xarray (mr, iova, addr, length, dir); }
drivers/infiniband/sw/rxe/rxe_odp.c +131 -0
134 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v9 5/5 RDMA/rxe: Add support for the traditional Atomic operations with ODP
In reply to Daisuke MatsudaEnable 'fetch and add' and 'compare and swap' operations to be used with ODP. This is comprised of the following steps:
1. Check the driver page table(umem_odp->dma_list) to see if the target page is both readable and writable.
2. If not, then trigger page fault to map the page.
3. Convert its user space address to a kernel logical address using PFNs
in the driver page table(umem_odp->pfn_list).
4. Execute the operation.
--- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 11 +++++ drivers/infiniband/sw/rxe/rxe_mr.c | 2 +- drivers/infiniband/sw/rxe/rxe_odp.c | 69 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 5 +- 5 files changed, 86 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +1 -0
@@ -90,6 +90,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_RECV ; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_WRITE ; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_READ ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_ATOMIC ; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; } }
drivers/infiniband/sw/rxe/rxe_loc.h +11 -0
@@ -81,6 +81,9 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key);int rxe_reg_fast_mr (struct rxe_qp *qp,struct rxe_send_wqe *wqe);void rxe_mr_cleanup (struct rxe_pool_elem *elem); +/* defined in rxe_mr.c; used in rxe_mr.c and rxe_odp.c */ +extern spinlock_t atomic_ops_lock; +/* rxe_mw.c */ int rxe_alloc_mw (struct ib_mw *ibmw,struct ib_udata *udata);int rxe_dealloc_mw (struct ib_mw *ibmw);
@@ -190,6 +193,8 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, int access_flags, struct rxe_mr *mr);int rxe_odp_mr_copy (struct rxe_mr *mr,u64 iova,void *addr,int length,enum rxe_mr_copy_dir dir); +int rxe_odp_atomic_op (struct rxe_mr *mr,u64 iova,int opcode, +u64 compare,u64 swap_add,u64 *orig_val);#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ static inline int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -202,6 +207,12 @@ static inline int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, {return -EOPNOTSUPP ; } +static inline int +rxe_odp_atomic_op (struct rxe_mr *mr,u64 iova,int opcode, +u64 compare,u64 swap_add,u64 *orig_val) +{ +return RESPST_ERR_UNSUPPORTED_OPCODE ; +}#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ #endif /* RXE_LOC_H */
drivers/infiniband/sw/rxe/rxe_mr.c +1 -1
@@ -469,7 +469,7 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) }/* Guarantee atomicity of atomic operations at the machine level. */ -static DEFINE_SPINLOCK (atomic_ops_lock); +DEFINE_SPINLOCK (atomic_ops_lock);int rxe_mr_do_atomic_op (struct rxe_mr *mr,u64 iova,int opcode,u64 compare,u64 swap_add,u64 *orig_val)
drivers/infiniband/sw/rxe/rxe_odp.c +69 -0
72 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_resp.c +4 -1
@@ -707,7 +707,10 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,u64 iova = qp->resp.va + qp->resp.offset;if (mr->umem->is_odp) - err =RESPST_ERR_UNSUPPORTED_OPCODE ; + err =rxe_odp_atomic_op (mr, iova, pkt->opcode, +atmeth_comp (pkt), +atmeth_swap_add (pkt), + &res->atomic.orig_val);else err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode, atmeth_comp(pkt),
Signature · 1 line
2.43.0
A message that is not in the archive (sent off-list or not yet received): CAHjRaAeXCC+AAV+Ne0cJMpZJYxbD8ox28kp966wkdVJLJdSC_g@mail.gmail.com
Quoted 1 line · Joe Klein
We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?
Hi Joe,
Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.
As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.
Here are the details of the issue:
===== test log =====
ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3 ubuntu@rdma-dev:~$ cd rdma-core ubuntu@rdma-dev:~/rdma-core$ uname -r 6.13.0-rc1+ ubuntu@rdma-dev:~/rdma-core$ pwd /home/ubuntu/rdma-core ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py ..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback /usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback .......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True> def __exit__(self, typ, value, traceback): ResourceWarning: Enable tracemalloc to get the object allocation traceback ssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor ssssTraceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor s....ssSegmentation fault (core dumped) ===========
=====dmesg=====
[ 147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4 [ 147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10 [ 147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9 [ 147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10 [ 147.524080] rxe_ens3: rxe_create_cq: returned err = -22 [ 147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22 [ 147.605719] rxe_ens3: rxe_create_cq: returned err = -95 [ 147.606454] rxe_ens3: rxe_create_cq: returned err = -22 [ 148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10 [ 148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10 [ 148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10 [ 148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10 [ 149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2) [ 149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be 01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89
=====
If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are seeing. I would also appreciate any help you can offer in fixing the upstream issue.
Thanks,
Daisuke
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein
We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?
Quoted 8 lines
Hi Joe,
Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.
It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.
I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################
ubuntu@rdma-aarch64:~/rdma-core$ git branch -v * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss ---------------------------------------------------------------------- Ran 321 tests in 3.599s
OK (skipped=211)
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp sssssssss..ss....s.s ---------------------------------------------------------------------- Ran 20 tests in 0.269s
OK (skipped=13)
####################
Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
Thanks,
Daisuke
Quoted 84 lines
Here are the details of the issue:
===== test log =====ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3 ubuntu@rdma-dev:~$ cd rdma-core ubuntu@rdma-dev:~/rdma-core$ uname -r 6.13.0-rc1+ ubuntu@rdma-dev:~/rdma-core$ pwd /home/ubuntu/rdma-core ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py ..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback /usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback .......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True> def __exit__(self, typ, value, traceback): ResourceWarning: Enable tracemalloc to get the object allocation traceback ssss..............ssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/ usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci' mode='rb+' closefd=True> def _remove(item, selfref=ref(self)): ResourceWarning: Enable tracemalloc to get the object allocation traceback Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor ssssTraceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor Exception ignored in: 'pyverbs.pd.PD.__dealloc__' Traceback (most recent call last): File "pd.pyx", line 120, in pyverbs.pd.PD.close pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor s....ssSegmentation fault (core dumped) ================dmesg=====
[ 147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4 [ 147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10 [ 147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9 [ 147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10 [ 147.524080] rxe_ens3: rxe_create_cq: returned err = -22 [ 147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22 [ 147.605719] rxe_ens3: rxe_create_cq: returned err = -95 [ 147.606454] rxe_ens3: rxe_create_cq: returned err = -22 [ 148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10 [ 148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10 [ 148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10 [ 148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10 [ 149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2) [ 149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89
=====If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are seeing.
I would also appreciate any help you can offer in fixing the upstream issue.Thanks,
Daisuke
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein
We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?
Quoted 8 lines
Hi Joe,
Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.
It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (
x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################ubuntu@rdma-aarch64:~/rdma-core$ git branch -v * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss ---------------------------------------------------------------------- Ran 321 tests in 3.599sOK (skipped=211)
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp sssssssss..ss....s.s ---------------------------------------------------------------------- Ran 20 tests in 0.269s
Quoted 5 lines
OK (skipped=13)
####################Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
Quoted 2 lines
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
What am I supposed to do with this though?
Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?
Jason
Quoted 48 lines · Jason Gunthorpe
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein
We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?
Quoted 8 lines
Hi Joe,
Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.
It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (
x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################ubuntu@rdma-aarch64:~/rdma-core$ git branch -v * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss ---------------------------------------------------------------------- Ran 321 tests in 3.599sOK (skipped=211)
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp sssssssss..ss....s.s ---------------------------------------------------------------------- Ran 20 tests in 0.269sQuoted 5 lines
OK (skipped=13)
####################Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
Quoted 7 lines
Quoted 2 lines
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
What am I supposed to do with this though?
Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?
Will make tests and let you know the result very soon.
Quoted 2 lines
Jason
Quoted 53 lines · Joe Klein
Quoted 51 lines · Jason Gunthorpe
Quoted 38 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Daisuke Matsuda (Fujitsu)
Quoted 1 line · Joe Klein
We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?
Quoted 8 lines
Hi Joe,
Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.As far as I know, there is something wrong with the upstream right now. It does not complete the rdma-core testcases, and 'segmentation fault' is observed in the middle of the full test run, which did not happen before October 2024.
It appears that the root cause of this issue lies within the userspace components. My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (
x86_64). It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries. All tests available passed using the rdma for-next tree without any problem. Then, I applied my ODP patches onto it, and everything is still fine.
####################ubuntu@rdma-aarch64:~/rdma-core$ git branch -v * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss ---------------------------------------------------------------------- Ran 321 tests in 3.599sOK (skipped=211)
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp sssssssss..ss....s.s ---------------------------------------------------------------------- Ran 20 tests in 0.269sQuoted 5 lines
OK (skipped=13)
####################Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
Quoted 2 lines
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
Quoted 6 lines
Quoted 4 lines
What am I supposed to do with this though?
Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?
Will make tests and let you know the result very soon.
Need time to complete the test scenario configurations.
We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.
Quoted 3 lines
Quoted 3 lines
Jason
Quoted 13 lines · Joe Klein
Quoted 13 lines
Quoted 12 lines
Quoted 2 lines
Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
Quoted 2 lines
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
What am I supposed to do with this though?
Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?
Quoted 7 lines
Quoted 1 line
Will make tests and let you know the result very soon.
Need time to complete the test scenario configurations.
We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.
Can you share your logs so that Daisuke can resolve whatever the problem is?
Jason
On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
In conclusion, I believe there is nothing in my ODP patches that could cause the rxe driver to fail. I would appreciate any feedback on potential improvements.
What am I supposed to do with this though?
Joe, can you please answer Daisuke's questions on what problems you observed and if you observe them without the ODP patches?
Will make tests and let you know the result very soon.
Quoted 8 lines
Quoted 5 lines
Need time to complete the test scenario configurations.
We made tests again. Some memory errors are from RXE ODP. The whole tests can not be complete with this patch set. Without this patch set, all the tests can pass.
Can you share your logs so that Daisuke can resolve whatever the problem is?
Since Joe does not provide any specific information, it would be nice if someone familiar with rxe like Zhu Yanjun, Li Zhijian or Bob Pearson could find the time to test and review the patch.
Cf. https://github.com/ddmatsu/linux/tree/odp_v9
However, to be honest, I do not think Joe's report is valid. It is not normal to complain other's patches without providing any details. It makes difficult to assess the validity of the claim, and blocks any further discussions forever.
The testcases in libibverbs pass as I attached the log. Other than that, I verified my patches work with perftest (with ODP enabled/disabled). I believe I did the test properly and the patches are ready to be merged.
Daisuke
Quoted 2 lines
Jason
Quoted 3 lines · Daisuke Matsuda (Fujitsu)
However, to be honest, I do not think Joe's report is valid. It is not normal to complain other's patches without providing any details. It makes difficult to assess the validity of the claim, and blocks any further discussions forever.
Yes, I agree, let's move past anything without concrete actionable details.
Jason
On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):
Quoted 10 lines
On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.
On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases, so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.
Could it be possible the update of python version leads to the failure? I saw strange warnings while running build.sh, which did not appear on Ubuntu 22.04.5. I would like to hear from those who have tried building rdma-core on Ubuntu 24.04.1 or with Python 3.12.3.
[422/607] Cythonizing /home/ubuntu/rdma-core/pyverbs/pd.pyx
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:102:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:124:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:142:30: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:158:61: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:180:46: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:272:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:279:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:12:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:21:47: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:16:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:17:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:22:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:32:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/mr.pxd:21:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:28:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:29:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:22:50: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:11:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:13:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:18:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:20:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:35:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:37:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:37:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:38:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:45:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:27:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:38:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword. [607/607] Linking C shared library python/pyverbs/providers/mlx5/mlx5dv.cpython-312-x86_64-linux-gnu.so
Daisuke
Quoted 12 lines · Daisuke Matsuda (Fujitsu)
On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):
Quoted 10 lines
On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
Quoted 6 lines · Joe Klein
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2. We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
That's distressing.
I am going to start bisecting the root cause to fix it. It may take a while, so please stay patient.
Quoted 6 lines
On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases, so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.
Could it be possible the update of python version leads to the failure?
Or a cython update? It certainly could..
So maybe there is nothing to worry about kernel side if the test case passes on older OS userspace
Jason
Quoted 7 lines · Daisuke Matsuda
Daisuke Matsuda (5):
RDMA/rxe: Move some code torxe_loc.hin preparation for ODP RDMA/rxe: Add page invalidation support
RDMA/rxe: Allow registering MRs for On-Demand Paging RDMA/rxe: Add support for Send/Recv/Write/Read with ODP RDMA/rxe: Add support for the traditional Atomic operations with ODP
Quoted 11 lines
drivers/infiniband/sw/rxe/Makefile | 2 + drivers/infiniband/sw/rxe/rxe.c | 18 ++ drivers/infiniband/sw/rxe/rxe.h | 37 --- drivers/infiniband/sw/rxe/rxe_loc.h | 34 +++ drivers/infiniband/sw/rxe/rxe_mr.c | 13 +- drivers/infiniband/sw/rxe/rxe_odp.c | 324 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 18 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 5 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 37 +++ 9 files changed, 443 insertions(+), 45 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
Applied to for-next, thanks
Jason