Skip to content

RFC vfio/pci: Support ZONE_DEVICE-backed P2P Registration

Pranjal Shrivastava kvm linux-pci lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest5 patches 8 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5vfio: Add UAPI for ZONE_DEVICE-backed P2P registrationinclude/uapi/linux/vfio.h+15 -00 replies
2/5vfio/pci: Implement ZONE_DEVICE-backed P2P registrationdrivers/vfio/pci/vfio_pci_core.c include/linux/vfio_pci_core.h+40 -00 replies
3/5vfio/pci: Block mmap & dmabuf export for ZONE_DEVICE-registered BARsdrivers/vfio/pci/vfio_pci_core.c drivers/vfio/pci/vfio_pci_dmabuf.c+13 -00 replies
4/5vfio/pci: Block ZONE_DEVICE registration for BARs with active DMABUFsdrivers/vfio/pci/vfio_pci_core.c drivers/vfio/pci/vfio_pci_dmabuf.c drivers/vfio/pci/vfio_pci_priv.h+28 -00 replies
5/5PCI/P2PDMA: Introduce a helper to release P2P resourcesdrivers/pci/p2pdma.c drivers/vfio/pci/vfio_pci_core.c include/linux/pci-p2pdma.h+44 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/5] vfio/pci: Support ZONE_DEVICE-backed P2P Registration

Introduce ZONE_DEVICE backing for VFIO-exposed PCIe BARs. Currently, kernel drivers can register their BARs with the P2PDMA subsystem to enable high-performance, page-backed P2P DMA. However, when a device is bound to vfio-pci, this capability is missing.

As a framework, VFIO should support the same set of capabilities available to kernel drivers, provided appropriate isolation and lifecycles are maintained.

This series adds support for VFIO to register BARs as page-backed P2P providers, enabling userspace drivers to perform zero-copy P2P DMA via standard POSIX APIs (e.g., O_DIRECT) which require struct page metadata.

Design
======
The proposed design enables VFIO to leverage the kernel's pci-p2pdma infrastructure, delegating memory management and slicing to the managed P2P pool.

1. ZONE_DEVICE Registration
A new feature, VFIO_DEVICE_FEATURE_P2P_REGISTER, is introduced to trigger pci_p2pdma_add_resource() for a target BAR which creates the ZONE_DEVICE struct pages and registers the BAR as a standard P2P provider.

2. Mutual Exclusivity Policy
A strict mutual exclusion policy is enforced:
- Registered BARs: Standard mmap() and DMABUF export are blocked.
- Exported BARs: Native P2P registration is blocked.

Users utilize the standard sysfs p2pmem/allocate interface for managing memory slices once a BAR is registered.

3. Lifecycle and Resource Reclamation
A new helper is introduced: pci_p2pdma_remove_resource(), is implemented to support manual teardown of the P2P pool. VFIO invokes this helper during device reset or session termination to ensure the pool is destroyed and the ZONE_DEVICE pages are unmapped.

This is on github: https://github.com/pran005/linux/tree/vfio-p2p

Alternative Approach: Page-backed DMABUF Export
================================================
An alternative implementation has been explored which integrates with the ongoing VFIO DMABUF-mmap refactor [1]. In that approach, rather than registering a BAR as a system-wide P2P provider, VFIO optionally allocates ZONE_DEVICE pages only for specifically exported DMABUFs via a new VFIO_DMA_BUF_FLAG_ALLOC_STRUCT_PAGES flag.

A proof-of-concept of this approach is available at:
https://github.com/pran005/linux/tree/vfio-p2p-alt

[1] https://lore.kernel.org/all/[email protected]/

Pranjal Shrivastava (5):
vfio: Add UAPI for ZONE_DEVICE-backed P2P registration vfio/pci: Implement ZONE_DEVICE-backed P2P registration vfio/pci: Block mmap & dmabuf export for ZONE_DEVICE-registered BARs vfio/pci: Block ZONE_DEVICE registration for BARs with active DMABUFs PCI/P2PDMA: Introduce a helper to release P2P resources

 drivers/pci/p2pdma.c               | 38 ++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c   | 57 ++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_dmabuf.c | 22 ++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h   |  6 ++++
 include/linux/pci-p2pdma.h         |  1 +
 include/linux/vfio_pci_core.h      |  1 +
 include/uapi/linux/vfio.h          | 15 ++++++++
 7 files changed, 140 insertions(+)
Signature · 1 line
2.54.0.1099.g489fc7bff1-goog