Skip to content

RFC nfs: Enable PCI Peer-to-Peer DMA (P2PDMA) support

Pranjal Shrivastava linux-nfs netdev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 10 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4sunrpc: add supports_p2pdma to rpc_xprt_opsinclude/linux/sunrpc/xprt.h net/sunrpc/xprtrdma/transport.c+10 -00 replies
2/4nfs: add NFS_CAP_P2PDMA and detect transport supportfs/nfs/client.c fs/nfs/nfs4_fs.h fs/nfs/super.c +1 more+44 -353 replies
3/4nfs: make nfs_page pin-awarefs/nfs/direct.c fs/nfs/pagelist.c include/linux/nfs_page.h+17 -73 replies
4/4nfs: allow P2PDMA in direct I/O pathfs/nfs/direct.c+37 -143 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/4] nfs: Enable PCI Peer-to-Peer DMA (P2PDMA) support

As high-performance storage environments increasingly rely on direct data movement between PCIe endpoints (e.g., moving data directly between an NVMe Controller Memory Buffer and a Network Interface), support for Peer-to-Peer DMA (P2PDMA) in the network filesystem layer becomes essential. This series introduces P2PDMA support for the NFS Direct I/O.

Currently, NFS O_DIRECT operations fail with -EREMOTEIO if the user buffer resides in PCIe BAR memory. This is primarily due to the use of the legacy iov_iter_get_pages_alloc2() API, which cannot pass the required FOLL_PCI_P2PDMA flag, and a request lifecycle that is unaware
of the pinning requirements for P2P memory.

Design
=======
The proposed design centers around making the NFS request lifecycle "pin-aware" and upgrading the infrastructure to support modern memory extraction APIs.

1. 64-bit Capability Infrastructure
The existing nfs_server->caps bitmask is limited to 32 bits and is currently exhausted. This series expands the bitmask to 64 bits to accommodate NFS_CAP_P2PDMA. Crucially, it also refactors the NFS_CAP_* constants to use ULL definitions. This prevents a subtle 32-bit truncation bug where bitwise negations (e.g., caps &= ~NFS_CAP_ACLS) would accidentally clear the high bits of the 64-bit capability field.

2. Transport-Level Detection
P2PDMA support is a property of the local transport hardware. A new supports_p2pdma operation is added to the SunRPC transport ops. For RDMA, this is implemented by querying the underlying device via ib_dma_pci_p2p_dma_supported(). The NFS client queries this during mount and sets the NFS_CAP_P2PDMA bit accordingly.

3. Pin-Aware Request Lifecycle
Standard NFS requests use get_page() and put_page() for memory management. However, memory extracted via iov_iter_extract_pages() requires explicit pinning and unpinning (unpin_user_page()).

This series introduces a PG_PINNED flag in struct nfs_page. When set, the request lifecycle skips standard page referencing and ensures that unpin_user_page() is called only when the I/O is complete. This ensures that physical memory remains pinned for the duration of the DMA transfer

4. API Migration
The Direct I/O path is migrated to the modern iov_iter_extract_pages() API. The ITER_ALLOW_P2PDMA flag is passed to the iterator only when the local mount has signaled P2P support via the capability bit. This ensures that "normal" users on standard TCP/UDP transports see no change in behavior or overhead.

Call for review
===============
Any insights on the proposed changes to the nfs_page lifecycle and the 64-bit capability expansion are appreciated. If this approach is deemed incorrect or if there is a more idiomatic way for this, please direct me in the right direction.

Thanks,
Praan

Pranjal Shrivastava (4):
sunrpc: add supports_p2pdma to rpc_xprt_ops
nfs: add NFS_CAP_P2PDMA and detect transport support
nfs: make nfs_page pin-aware
nfs: allow P2PDMA in direct I/O path

 fs/nfs/client.c                 |  8 ++++
 fs/nfs/direct.c                 | 51 ++++++++++++++++++-------
 fs/nfs/nfs4_fs.h                |  2 +-
 fs/nfs/pagelist.c               | 18 ++++++---
 fs/nfs/super.c                  |  2 +-
 include/linux/nfs_fs_sb.h       | 67 +++++++++++++++++----------------
 include/linux/nfs_page.h        |  2 +
 include/linux/sunrpc/xprt.h     |  1 +
 net/sunrpc/xprtrdma/transport.c |  9 +++++
 9 files changed, 106 insertions(+), 54 deletions(-)
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog