Skip to content

RFC vfio: selftests: Add peer-to-peer DMA selftest

David Matlack dri-devel kvm linux-iommu linux-kselftest linux-media linux-pci lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest7 patches 7 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7vfio: selftests: Add vfio_pci_find_mem_bar() helpertools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h tools/testing/selftests/vfio/lib/vfio_pci_device.c+77 -00 replies
2/7vfio: selftests: Verify memory behavior in vfio_pci_find_mem_bar()tools/testing/selftests/vfio/lib/vfio_pci_device.c+33 -00 replies
3/7vfio: selftests: Allow multiple BDFs in VFIO_SELFTESTS_BDFtools/testing/selftests/vfio/lib/libvfio.c+39 -140 replies
4/7vfio: selftests: Add vfio_pci_dmabuf_create() helpertools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h tools/testing/selftests/vfio/lib/vfio_pci_device.c+38 -00 replies
5/7vfio: selftests: Add iommu_map_file() helpertools/testing/selftests/vfio/lib/include/libvfio/iommu.h tools/testing/selftests/vfio/lib/iommu.c+38 -00 replies
6/7vfio: selftests: Add vfio_pci_find_p2p_pair() helpertools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h tools/testing/selftests/vfio/lib/vfio_pci_device.c+56 -00 replies
7/7vfio: selftests: Add P2P DMA selftesttools/testing/selftests/vfio/Makefile tools/testing/selftests/vfio/vfio_pci_p2pdma_test.c+436 -00 replies

Reviews trailers per patch

1 of 7 patches has a Reviewed-by or Acked-by.

PatchDavid Matlack
1/7R-b
2/7
3/7
4/7
5/7
6/7
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 0/7] vfio: selftests: Add peer-to-peer DMA selftest

This series can be found on GitHub:

https://github.com/dmatlack/linux/tree/vfio/selftests/p2pdma/rfc/v1

This series adds a peer-to-peer (P2P) DMA selftest to the VFIO selftests. The test pairs an initiator device supported by the VFIO selftests driver framework (dsa, igb, ioat, mlx5, or nv_falcon) with a target device that exposes a memory BAR, maps the target's BAR into the IOMMU, and uses the initiator to perform DMA reads and writes targeting the BAR.

No changes to struct vfio_pci_driver_ops are needed. Because vfio_pci_driver_memcpy() already takes source and destination IOVAs, any existing selftests driver can drive P2P DMA once a peer's BAR is mapped into its IOMMU address space.

There are two motivations for adding P2P DMA support to VFIO selftests:

1. Exercise the P2P DMA paths in VFIO and iommufd end-to-end, including mapping BAR mmaps via VFIO_IOMMU_MAP_DMA in legacy type1/type1v2 modes and exporting BARs via VFIO_DEVICE_FEATURE_DMA_BUF + IOMMU_IOAS_MAP_FILE (and dma-buf revocation) in native iommufd.

2. Lay the groundwork for testing uninterrupted P2P DMA across a Live Update in the future, building on VFIO PCI Live Update support [1] and the PCI core changes to keep preserved devices accessible across kexec [2].

Patch 1 comes from v3 of Mina's huge_fault series [3] and is included here for convenience. It adds vfio_pci_find_mem_bar() to locate a BAR range that behaves like ordinary memory, starting with QEMU's pci-testdev.

This series applies on top of vfio/next (vfio-v7.4-rc1).

[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://lore.kernel.org/linux-pci/[email protected]/
[3] https://lore.kernel.org/kvm/[email protected]/

Series Overview
---------------

This series implements the following:

1. Add vfio_pci_find_mem_bar() to locate a BAR range that can be safely read and written like memory, starting with QEMU's pci-testdev, and verify that writes stick before returning it so that unbacked pci-testdev BARs are skipped (patches 1-2).

2. Allow VFIO_SELFTESTS_BDF to contain multiple whitespace-separated BDFs so multi-device tests can run via run_kselftest.sh while single-device tests continue to use the first BDF (patch 3).

3. Add vfio_pci_dmabuf_create() and iommu_map_file() helpers to export a VFIO PCI region range via VFIO_DEVICE_FEATURE_DMA_BUF and map it into an iommufd IOAS via IOMMU_IOAS_MAP_FILE (patches 4-5).

4. Add vfio_pci_find_p2p_pair() to find a compatible (initiator, target) pair from an array of VFIO PCI devices, providing a central place to check P2P topology reachability in the future (patch 6).

5. Add vfio_pci_p2pdma_test, covering RAM-to-BAR, BAR-to-RAM, BAR-to-BAR, and RAM-BAR-RAM roundtrip copies across all supported IOMMU modes, plus iommufd dma-buf revocation when PCI_COMMAND_MEMORY is cleared on the target (patch 7).

Open Questions
--------------

- Re-mapping after dma-buf revocation: Clearing PCI_COMMAND_MEMORY on the target device calls vfio_pci_dma_buf_move(vdev, true), which notifies iommufd via iopt_revoke_notify() and unmaps the BAR from the IOMMU domain. Re-enabling PCI_COMMAND_MEMORY calls vfio_pci_dma_buf_move(vdev, false), allowing the dma-buf to be mapped again, but dma-buf has no callback to re-populate iommufd, so the existing IOAS mapping stays unmapped until userspace calls IOMMU_IOAS_UNMAP and IOMMU_IOAS_MAP_FILE again.

Is requiring userspace to re-issue IOMMU_IOAS_MAP_FILE the expected uAPI behavior? VFIO_DEVICE_RESET (and runtime PM transitions) on the target device also bracket the reset with vfio_pci_dma_buf_move(vdev, true) and vfio_pci_dma_buf_move(vdev, false), tearing down any active IOMMU_IOAS_MAP_FILE mapping of that device's BAR even though PCI_COMMAND_MEMORY remains enabled when VFIO_DEVICE_RESET returns.

- Physical target devices and P2P topology checks: mem_bar_descs[] currently only matches QEMU's pci-testdev, and vfio_pci_has_p2p_path() does not yet check PCI topology or ACS to verify that P2P traffic can flow between two physical endpoints.

Suggestions on physical target BARs to add to mem_bar_descs[] and on checking P2P reachability from userspace are welcome.

Testing
-------

All patches were build-tested individually. The series was tested on x86_64 in QEMU 11.1 (membar-backed=on requires QEMU 10.1+) using an emulated Intel 82576 NIC (igb) and a pci-testdev with a 1G RAM-backed BAR2 behind an emulated Intel IOMMU:

  qemu-system-x86_64 \
    -M q35,accel=kvm,kernel-irqchip=split \
    -device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on \
    -netdev user,id=net0 \
    -device pcie-root-port,id=rp0,slot=2 \
    -device igb,bus=rp0,addr=0.0,netdev=net0 \
    -device pci-testdev,membar=1G,membar-backed=on \
    ...

Inside the guest, bind both devices to vfio-pci and run the test (BDF order does not matter):

  # tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0 0000:00:0d.0
  # tools/testing/selftests/vfio/scripts/run.sh \
      tools/testing/selftests/vfio/vfio_pci_p2pdma_test

Or via run_kselftest.sh:

# VFIO_SELFTESTS_BDF="0000:01:00.0 0000:00:0d.0" \
./run_kselftest.sh -t vfio:vfio_pci_p2pdma_test

David Matlack (6):
vfio: selftests: Verify memory behavior in vfio_pci_find_mem_bar()
vfio: selftests: Allow multiple BDFs in VFIO_SELFTESTS_BDF
vfio: selftests: Add vfio_pci_dmabuf_create() helper
vfio: selftests: Add iommu_map_file() helper
vfio: selftests: Add vfio_pci_find_p2p_pair() helper
vfio: selftests: Add P2P DMA selftest

Mina Almasry (1):
vfio: selftests: Add vfio_pci_find_mem_bar() helper

 tools/testing/selftests/vfio/Makefile         |   1 +
 .../vfio/lib/include/libvfio/iommu.h          |   9 +
 .../lib/include/libvfio/vfio_pci_device.h     |  23 +
 tools/testing/selftests/vfio/lib/iommu.c      |  29 ++
 tools/testing/selftests/vfio/lib/libvfio.c    |  53 ++-
 .../selftests/vfio/lib/vfio_pci_device.c      | 181 ++++++++
 .../selftests/vfio/vfio_pci_p2pdma_test.c     | 435 ++++++++++++++++++
 7 files changed, 717 insertions(+), 14 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_p2pdma_test.c

base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b

Signature · 1 line
2.56.0.567.gdb80ae8cd5-goog