RFC vfio/pci: Support ZONE_DEVICE-backed P2P Registration
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest5 patches 8 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/5 | vfio: Add UAPI for ZONE_DEVICE-backed P2P registrationinclude/ | +15 | 0 replies | |
| 2/5 | vfio/pci: Implement ZONE_DEVICE-backed P2P registrationdrivers/ include/ | +40 | 0 replies | |
| 3/5 | vfio/pci: Block mmap & dmabuf export for ZONE_DEVICE-registered BARsdrivers/ drivers/ | +13 | 0 replies | |
| 4/5 | vfio/pci: Block ZONE_DEVICE registration for BARs with active DMABUFsdrivers/ drivers/ drivers/ | +28 | 0 replies | |
| 5/5 | PCI/P2PDMA: Introduce a helper to release P2P resourcesdrivers/ drivers/ include/ | +44 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/5] vfio/pci: Support ZONE_DEVICE-backed P2P Registration
Introduce ZONE_DEVICE backing for VFIO-exposed PCIe BARs. Currently, kernel drivers can register their BARs with the P2PDMA subsystem to enable high-performance, page-backed P2P DMA. However, when a device is bound to vfio-pci, this capability is missing.
As a framework, VFIO should support the same set of capabilities available to kernel drivers, provided appropriate isolation and lifecycles are maintained.
This series adds support for VFIO to register BARs as page-backed P2P providers, enabling userspace drivers to perform zero-copy P2P DMA via standard POSIX APIs (e.g., O_DIRECT) which require struct page metadata.
Design
======
The proposed design enables VFIO to leverage the kernel's pci-p2pdma infrastructure, delegating memory management and slicing to the managed P2P pool.
1. ZONE_DEVICE Registration
A new feature, VFIO_DEVICE_FEATURE_P2P_REGISTER, is introduced to trigger pci_p2pdma_add_resource() for a target BAR which creates the ZONE_DEVICE struct pages and registers the BAR as a standard P2P provider.
2. Mutual Exclusivity Policy
A strict mutual exclusion policy is enforced:
- Registered BARs: Standard mmap() and DMABUF export are blocked.
- Exported BARs: Native P2P registration is blocked.
Users utilize the standard sysfs p2pmem/allocate interface for managing memory slices once a BAR is registered.
3. Lifecycle and Resource Reclamation
A new helper is introduced: pci_p2pdma_remove_resource(), is implemented to support manual teardown of the P2P pool. VFIO invokes this helper during device reset or session termination to ensure the pool is destroyed and the ZONE_DEVICE pages are unmapped.
This is on github: https://github.com/pran005/linux/tree/vfio-p2p
Alternative Approach: Page-backed DMABUF Export
================================================
An alternative implementation has been explored which integrates with the ongoing VFIO DMABUF-mmap refactor [1]. In that approach, rather than registering a BAR as a system-wide P2P provider, VFIO optionally allocates ZONE_DEVICE pages only for specifically exported DMABUFs via a new VFIO_DMA_BUF_FLAG_ALLOC_STRUCT_PAGES flag.
A proof-of-concept of this approach is available at:
https://github.com/pran005/linux/tree/vfio-p2p-alt
[1] https://lore.kernel.org/all/[email protected]/
Pranjal Shrivastava (5):
vfio: Add UAPI for ZONE_DEVICE-backed P2P registration vfio/pci: Implement ZONE_DEVICE-backed P2P registration vfio/pci: Block mmap & dmabuf export for ZONE_DEVICE-registered BARs vfio/pci: Block ZONE_DEVICE registration for BARs with active DMABUFs PCI/P2PDMA: Introduce a helper to release P2P resources
drivers/pci/p2pdma.c | 38 ++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 57 ++++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_dmabuf.c | 22 ++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 6 ++++ include/linux/pci-p2pdma.h | 1 + include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 15 ++++++++ 7 files changed, 140 insertions(+)
Signature · 1 line
2.54.0.1099.g489fc7bff1-goog