Skip to content

PCI passthru on Hyper-V

Mukesh R linux-arch linux-hyperv linux-iommu linux-pci lkml 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest9 patches 17 repliesIn review
  2. v49 patches 8 repliesSuperseded
  3. v311 patches 14 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v211 patches4 repliesSuperseded
v113 patches9 repliesSuperseded

Changes in v5 from the cover letter

o enforce no multiple vms per vmm instance
o make struct mshv_vfio_device_ops const
o add WARN to iommu call. delete mappings in domain free
o mmio_mfn++ in large page check in hv_map_mmio_pages()
o no need to pass gfn to hv_map_mmio_pages()
o use region access to make mmio mappings ro or rw

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9mshv: Provide a way to get partition ID if running in a VMM processdrivers/hv/mshv_root.h drivers/hv/mshv_root_main.c include/asm-generic/mshyperv.h+38 -10 replies
2/9mshv: Add declarations and definitions for VFIO-MSHV bridge devicedrivers/hv/mshv_root.h include/uapi/linux/mshv.h+49 -00 replies
3/9mshv: Introduce basic mshv bridge device for VFIO to build upondrivers/hv/Makefile drivers/hv/mshv_vfio.c include/uapi/linux/mshv.h+214 -17 replies
4/9mshv: Add ioctl support for MSHV-VFIO bridge devicedrivers/hv/mshv_root_main.c+119 -00 replies
5/9mshv: Import data structs around device passthru from hyperv headersinclude/hyperv/hvgdk_mini.h include/hyperv/hvhdk_mini.h+75 -00 replies
6/9PCI: hv: Export hv_build_devid_type_pci() and change return typearch/x86/hyperv/irqdomain.c arch/x86/include/asm/mshyperv.h+11 -41 reply
7/9x86/hyperv: Implement Hyper-V virtual IOMMUarch/x86/kernel/pci-dma.c drivers/iommu/Kconfig drivers/iommu/hyperv/Kconfig +4 more+734 -09 replies
8/9mshv: Populate mmio mappings for PCI passthrudrivers/hv/mshv_root.h drivers/hv/mshv_root_hv_call.c drivers/hv/mshv_root_main.c+185 -420 replies
9/9mshv: Disable movable regions upfront if device passthrudrivers/hv/mshv_root.h drivers/hv/mshv_root_main.c+15 -10 replies

Reviews trailers per patch

3 of 9 patches have a Reviewed-by or Acked-by.

PatchAnirudh Rayabharam (Microsoft)Souradeep Chakrabarti
1/9R-b
2/9R-b
3/9
4/9
5/9
6/9R-b
7/9
8/9
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 0/9] PCI passthru on Hyper-V

Implement passthru of PCI devices to unprivileged virtual machines (VMs) when Linux is running as a privileged VM on Microsoft Hyper-V hypervisor. This support is made to fit within the workings of VFIO framework. This supports both full device passthru and SR-IOV based VFs.

At a high level, the hypervisor supports traditional paging iommu domains that use explicit map and unmap hypercalls for mapping and unmapping guest RAM into the iommu subsystem. Hyper-V also has a concept of direct attach devices (DDA) whereby the iommu subsystem simply uses the guest HW page table (ept/npt/..). This series adds support for the former.

This series rebased to: a4ffc59238be (origin/hyperv-next)

Testing:
o Most testing done on hyperv-next:a4ffc59238be using Cloud Hypervisor (51).
o All tests involved passthru of devices using MSIx.
o Tested on Baremetal dom0/root:
test 1: Mellanox ConnectX-6 Lx passthru
test 2: NVIDIA Tesla Tesla T4 GPU.
test 3: Both of above simultaneous passthru
o VF: Mellanox ConnectX-6 Lx passthru

This series strives to establish a base line. Some pending work items:
o arm64 : some delta to make this work on arm64 (in progress).
o Qemu and OpenVMM support (in progress).
o device sleep/wakeup.
o More stress testing with high end GPUs

Changes in V5:
o enforce no multiple vms per vmm instance
o make struct mshv_vfio_device_ops const
o add WARN to iommu call. delete mappings in domain free
o mmio_mfn++ in large page check in hv_map_mmio_pages()
o no need to pass gfn to hv_map_mmio_pages()
o use region access to make mmio mappings ro or rw

Changes in V4:
o Completely remove direct device attach (DDA) support
o Remove first two patches that were submitted separately and are already merged.
o Change hv_iommu_unmap_pages() to unmap in batches of HV_REP_COUNT_MAX, otherwise if a physical range happens to be larger than that, there would be errors from the hypercall.
o Use large page (2M) flag for mmio mappings as internal testing found that to speed things up greatly.
o rename hv_no_movbl_pgs to hv_memreg in last patch.

Changes in V3:
o patch #8: fix compiler issues incase of !CONFIG_HYPERV. Also, do forward declaration of struct pci_dev instead of including pci.h.
o patch #9: minor changes to comments. Pass hv_domain instead of iommu_domain to hv_iommu_detach_dev() since that's what it needs. Set device private to null if attach fails. Clam down number of PFNs passed to hv_iommu_map_pgs().

Changes in V2:
o rebase to 5170a82e8921
o minor fixes for arm64 build
o drop patch 03: "x86/hyperv: add insufficient memory support in irqdomain.c" as it that path is no longer used
o drop patch 08: "PCI: hv: rename hv_compose_msi_msg .. " and do it separately outside this series.
o minor updates to commit messages

Changes in V1:
o patch 1: Don't tie hyperv-irq.c to CONFIG_HYPERV_IOMMU.
o patch 4: Redesigned to address security vulnerability found by copilot with passing tgid as a parameter. Also, do tgid setting right after setting pt_id.
o patch 5: Remove unused type parameter from mshv_device_ops.device_create
o patch 7: mshv_partition_ioctl_create_device cleanup on copy_to_user.
o patch 10: Add export of hv_build_devid_type_pci here to get rid of

             patch 11.

o patch 12: Move functions to build device ids from patch 11 here for the benefit of arm64. Rename file to: hyperv-iommu-root.c.
o patch 13: removed to be made part of interrupt part II of this support.
o patch 14: get rid of fast path to reduce review noise.
o New (last) patch to pin ram regions if device passthru to a VM.

Thanks,
-Mukesh

Mukesh R (9):
mshv: Provide a way to get partition ID if running in a VMM process
mshv: Add declarations and definitions for VFIO-MSHV bridge device
mshv: Introduce basic mshv bridge device for VFIO to build upon
mshv: Add ioctl support for MSHV-VFIO bridge device
mshv: Import data structs around device passthru from hyperv headers
PCI: hv: Export hv_build_devid_type_pci() and change return type x86/hyperv: Implement Hyper-V virtual IOMMU
mshv: Populate mmio mappings for PCI passthru
mshv: Disable movable regions upfront if device passthru

 arch/x86/hyperv/irqdomain.c          |   9 +-
 arch/x86/include/asm/mshyperv.h      |   6 +
 arch/x86/kernel/pci-dma.c            |   2 +
 drivers/hv/Makefile                  |   3 +-
 drivers/hv/mshv_root.h               |  24 +-
 drivers/hv/mshv_root_hv_call.c       | 104 +++-
 drivers/hv/mshv_root_main.c          | 287 ++++++++++-
 drivers/hv/mshv_vfio.c               | 211 ++++++++
 drivers/iommu/Kconfig                |   1 +
 drivers/iommu/hyperv/Kconfig         |  15 +
 drivers/iommu/hyperv/Makefile        |   1 +
 drivers/iommu/hyperv/hv-iommu-root.c | 706 +++++++++++++++++++++++++++
 include/asm-generic/mshyperv.h       |   8 +
 include/hyperv/hvgdk_mini.h          |   8 +
 include/hyperv/hvhdk_mini.h          |  67 +++
 include/linux/hyperv.h               |   6 +
 include/uapi/linux/mshv.h            |  31 ++
 17 files changed, 1440 insertions(+), 49 deletions(-)
 create mode 100644 drivers/hv/mshv_vfio.c
 create mode 100644 drivers/iommu/hyperv/Kconfig
 create mode 100644 drivers/iommu/hyperv/hv-iommu-root.c
Signature · 1 line
2.51.2.vfs.0.1