Skip to content

RFC intel_iommu: expose Shared Virtual Addressing to VMs

Liu Yi L kvm qemu-devel 9 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v11 RFC Latest25 patches 0 repliesNew
  2. v10 RFC25 patches 1 replySuperseded
  3. v9 RFC25 patches 0 repliesSuperseded
Earlier versions · 6
VersionPostedPatchesRepliesStatus
v8 RFC25 patches0 repliesSuperseded
v7 RFC25 patches0 repliesSuperseded
v6 RFC25 patches0 repliesSuperseded
v3 RFC25 patches40 repliesSuperseded
v2 RFC22 patches54 repliesSuperseded
v1 RFC18 patches39 repliesSuperseded

Changelog from the cover letter · 41 lines

- RFC v10 -> RFC v11:
a) Rebase to latest kernel implementation (5.12-rc1 vsva v8) and adopt to /dev/ioasid for requesting PASID from host.
RFC v10: https://lore.kernel.org/qemu-devel/[email protected]/

- RFC v9 -> RFC v10:
a) Rebase to latest kernel implementation (5.9-rc2 vsva v7)
RFC v9: https://lore.kernel.org/kvm/[email protected]/

- RFC v8 -> RFC v9:
a) Rebase to latest kernel implementation (5.8-rc6 vsva v6)
RFC v8: https://lore.kernel.org/kvm/[email protected]/

- RFC v7 -> RFC v8:
a) Rebase to latest kernel implementation (5.8-rc3 vsva v5)
RFC v7: https://lore.kernel.org/kvm/[email protected]/

- RFC v6 -> RFC v7:
a) Rebase to latest kernel implementation (5.8-rc3 vsva)
RFC v6: https://lore.kernel.org/kvm/[email protected]/

- RFC v5 -> RFC v6:
a) Use RFC instead of formal patch as kernel patch is in progress.
b) Address comments from Peter and Eric.
c) Add get_iommu_attr() to advertise vIOMMU nesting requirement to VFIO.
d) Update per latest kernel UAPI definition.
e) Add patch 0017 to check iommu nesting cap info in set_iommu(). RFC v5: https://www.spinics.net/lists/kvm/msg211475.html

- RFC v4 -> RFC v5:
a) Refactor the vfio HostIOMMUContext init code (patch 0008 - 0009 of v1 series)
b) Refactor the pasid binding handling (patch 0011 - 0016 of v1 series) RFC v4: https://patchwork.ozlabs.org/cover/1259648/

- RFC v3.1 -> RFC v4:
a) Implement HostIOMMUContext in QOM manner.
b) Add pci_set/unset_iommu_context() to register HostIOMMUContext to vIOMMU, thus the lifecircle of HostIOMMUContext is awared in vIOMMU side. In such way, vIOMMU could use the methods provided by the HostIOMMUContext safely.
c) Add back patch "[RFC v3 01/25] hw/pci: modify pci_setup_iommu() to set PCIIOMMUOps" RFCv3.1: https://patchwork.kernel.org/cover/11397879/

- RFC v3 -> v3.1:
a) Drop IOMMUContext, and rename DualStageIOMMUObject to HostIOMMUContext. HostIOMMUContext is per-vfio-container, it is exposed to vIOMMU via PCI layer. VFIO registers a PCIHostIOMMUFunc callback to PCI layer, vIOMMU could get HostIOMMUContext instance via it.
b) Check IOMMU uAPI version by VFIO_CHECK_EXTENSION
c) Add a check on VFIO_PASID_REQ availability via VFIO_GET_IOMMU_IHNFO
d) Reorder the series, put vSVA linux header file update in the beginning put the x-scalable-mode option mofification in the end of the series.
e) Dropped patch "[RFC v3 01/25] hw/pci: modify pci_setup_iommu() to set PCIIOMMUOps"
RFCv3: https://patchwork.kernel.org/cover/11356033/

- RFC v2 -> v3:
a) Introduce DualStageIOMMUObject to abstract the host IOMMU programming capability. e.g. request PASID from host, setup IOMMU nesting translation on host IOMMU. The pasid_alloc/bind_guest_page_table/iommu_cache_flush operations are moved to be DualStageIOMMUOps. Thus, DualStageIOMMUObject is an abstract layer which provides QEMU vIOMMU emulators with an explicit method to program host IOMMU.
b) Compared with RFC v2, the IOMMUContext has also been updated. It is modified to provide an abstract for vIOMMU emulators. It provides the method for pass-through modules (like VFIO) to communicate with host IOMMU. e.g. tell vIOMMU emulators about the IOMMU nesting capability on host side and report the host IOMMU DMA translation faults to vIOMMU emulators. RFC v2: https://www.spinics.net/lists/kvm/msg198556.html

- RFC v1 -> v2:
Introduce IOMMUContext to abstract the connection between VFIO and vIOMMU emulators, which is a replacement of the PCIPASIDOps in RFC v1. Modify x-scalable-mode to be string option instead of adding a new option as RFC v1 did. Refined the pasid cache management and addressed the TODOs mentioned in RFC v1.
RFC v1: https://patchwork.kernel.org/cover/11033657/

Patches in v11 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25scripts/update-linux-headers: Import iommu.hscripts/update-linux-headers.sh+1 -10 replies
2/25scripts/update-linux-headers: Import ioasid.hscripts/update-linux-headers.sh+1 -10 replies
3/25header file update VFIO/IOMMU vSVA APIs kernel 5.12-rc1linux-headers/linux/iommu.h linux-headers/linux/vfio.h+497 -00 replies
4/25hw/pci: modify pci_setup_iommu() to set PCIIOMMUOpshw/alpha/typhoon.c hw/arm/smmu-common.c hw/hppa/dino.c +15 more+96 -240 replies
5/25hw/pci: introduce pci_device_get_iommu_attr()hw/pci/pci.c include/hw/pci/pci.h+37 -50 replies
6/25intel_iommu: add get_iommu_attr() callbackhw/i386/intel_iommu.c+23 -00 replies
7/25vfio: pass nesting requirement into vfio_get_group()hw/vfio/ap.c hw/vfio/ccw.c hw/vfio/common.c +3 more+15 -60 replies
8/25vfio: check VFIO_TYPE1_NESTING_IOMMU supporthw/vfio/common.c+26 -110 replies
9/25hw/iommu: introduce HostIOMMUContexthw/Kconfig hw/iommu/Kconfig hw/iommu/host_iommu_context.c +3 more+195 -00 replies
10/25hw/pci: introduce pci_device_set/unset_iommu_context()hw/pci/pci.c include/hw/pci/pci.h+38 -00 replies
11/25intel_iommu: add set/unset_iommu_context callbackhw/i386/intel_iommu.c include/hw/i386/intel_iommu.h+83 -90 replies
12/25vfio: add HostIOMMUContext supporthw/vfio/common.c include/hw/iommu/host_iommu_context.h include/hw/vfio/vfio-common.h+76 -00 replies
13/25vfio: init HostIOMMUContext per-containerhw/vfio/common.c hw/vfio/pci.c include/hw/vfio/vfio-common.h+118 -350 replies
14/25intel_iommu: sync IOMMU nesting cap info for assigned deviceshw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h include/hw/i386/intel_iommu.h+127 -00 replies
15/25intel_iommu: add virtual command capability supporthw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h hw/i386/trace-events +1 more+188 -10 replies
16/25intel_iommu: process PASID cache invalidationhw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h hw/i386/trace-events+50 -50 replies
17/25intel_iommu: add PASID cache management infrastructurehw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h hw/i386/trace-events +1 more+520 -10 replies
18/25intel_iommu: bind/unbind guest page table to hosthw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h+143 -140 replies
19/25intel_iommu: replay pasid binds after context cache invalidationhw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h hw/i386/trace-events+52 -00 replies
20/25intel_iommu: do not pass down pasid bind for PASID #0hw/i386/intel_iommu.c+10 -00 replies
21/25vfio: add support for flush iommu stage-1 cachehw/iommu/host_iommu_context.c hw/vfio/common.c include/hw/iommu/host_iommu_context.h+51 -00 replies
22/25intel_iommu: process PASID-based iotlb invalidationhw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h+66 -00 replies
23/25intel_iommu: propagate PASID-based iotlb invalidation to hosthw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h+120 -00 replies
24/25intel_iommu: process PASID-based Device-TLB invalidationhw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h+19 -00 replies
25/25intel_iommu: modify x-scalable-mode to be string optionhw/i386/intel_iommu.c hw/i386/intel_iommu_internal.h include/hw/i386/intel_iommu.h+75 -40 replies

Reviews trailers per patch

10 of 25 patches have a Reviewed-by or Acked-by.

PatchCornelia HuckDavid GibsonPeter Xu
1/25A-b
2/25
3/25
4/25R-b R-b
5/25
6/25
7/25
8/25
9/25
10/25R-b
11/25
12/25
13/25
14/25
15/25R-b
16/25R-b
17/25
18/25
19/25
20/25R-b
21/25A-b
22/25R-b
23/25
24/25R-b
25/25R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v11 thread

Cover letter [RFC v11 00/25] intel_iommu: expose Shared Virtual Addressing to VMs

Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.

This QEMU series is intended to expose SVA usage to VMs. i.e. Sharing guest application address space with passthru devices. This is called vSVA in this series. The whole vSVA enabling requires QEMU/VFIO/IOMMU changes.

The high-level architecture for SVA virtualization is as below, the key design of vSVA support is to utilize the dual-stage IOMMU translation ( also known as IOMMU nesting translation) capability in host IOMMU.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

The complete vSVA kernel upstream patches are divided into three phases:
1. Common APIs and PCI device direct assignment
2. IOMMU-backed Mediated Device assignment
3. Page Request Services (PRS) support

This QEMU patchset is aiming for the phase 1 and phase 2. Compared with last version, the major change in this series is the PASID allocation. Instead of utilizing VFIO's uAPI to request PASID from kernel, this series uses /dev/ioasid uAPI to do it. For the reason of the change, please refer to [1]. This series is based on two kernel series [2] and [3] posted earlier.

Patch Overview:
1. patch 0001 - 0003: update kernel header files
2. patch 0004 - 0008: select VFIO_TYPE1_NESTING_IOMMU for vIOMMU built on IOMMU nesting translation.
3. patch 0009 - 0013: introduce HostIOMMUContext, and implements in VFIO.
4. patch 0014 : IOMMU nesting capability sync between vIOMMU and host.
5. patch 0015 : PASID allocation/free support in Intel vIOMMU.
6. patch 0016 - 0017: PASID cache management for Intel vIOMMU.
7. patch 0018 - 0020: bind guest page table to host.
8. patch 0021 - 0024: flush first level/stage cache for vIOMMU.
9. patch 0025 : expose SVA to VM by x-scalable-mode="modern"

The complete QEMU set can be found in [4], and complete kernel can be found in [5].

Tests: basci vSVA functionality test, VM reboot/shutdown/crash, kernel build in guest, boot VM with vSVA disabled, full comapilation with all archs, passthru entire PCI device, passthru Scalable IOV ADI.

[1] https://lore.kernel.org/kvm/DM5PR11MB14351121729909028D6EB365C31D0@DM5PR11MB1435.namprd11.prod.outlook.com/
[2] https://lore.kernel.org/linux-iommu/[email protected]/
[3] https://lore.kernel.org/linux-iommu/[email protected]/
[4] https://github.com/jacobpan/linux/tree/vsva-linux-5.12-rc1-v8
[5] https://github.com/luxis1999/qemu/tree/vsva_5.12_rc1_qemu_rfcv11

Regards,
Yi Liu

Changelog:
- RFC v10 -> RFC v11:
a) Rebase to latest kernel implementation (5.12-rc1 vsva v8) and adopt to /dev/ioasid for requesting PASID from host.
RFC v10: https://lore.kernel.org/qemu-devel/[email protected]/

- RFC v9 -> RFC v10:
a) Rebase to latest kernel implementation (5.9-rc2 vsva v7)
RFC v9: https://lore.kernel.org/kvm/[email protected]/

- RFC v8 -> RFC v9:
a) Rebase to latest kernel implementation (5.8-rc6 vsva v6)
RFC v8: https://lore.kernel.org/kvm/[email protected]/

- RFC v7 -> RFC v8:
a) Rebase to latest kernel implementation (5.8-rc3 vsva v5)
RFC v7: https://lore.kernel.org/kvm/[email protected]/

- RFC v6 -> RFC v7:
a) Rebase to latest kernel implementation (5.8-rc3 vsva)
RFC v6: https://lore.kernel.org/kvm/[email protected]/

- RFC v5 -> RFC v6:
a) Use RFC instead of formal patch as kernel patch is in progress.
b) Address comments from Peter and Eric.
c) Add get_iommu_attr() to advertise vIOMMU nesting requirement to VFIO.
d) Update per latest kernel UAPI definition.
e) Add patch 0017 to check iommu nesting cap info in set_iommu(). RFC v5: https://www.spinics.net/lists/kvm/msg211475.html

- RFC v4 -> RFC v5:
a) Refactor the vfio HostIOMMUContext init code (patch 0008 - 0009 of v1 series)
b) Refactor the pasid binding handling (patch 0011 - 0016 of v1 series) RFC v4: https://patchwork.ozlabs.org/cover/1259648/

- RFC v3.1 -> RFC v4:
a) Implement HostIOMMUContext in QOM manner.
b) Add pci_set/unset_iommu_context() to register HostIOMMUContext to vIOMMU, thus the lifecircle of HostIOMMUContext is awared in vIOMMU side. In such way, vIOMMU could use the methods provided by the HostIOMMUContext safely.
c) Add back patch "[RFC v3 01/25] hw/pci: modify pci_setup_iommu() to set PCIIOMMUOps" RFCv3.1: https://patchwork.kernel.org/cover/11397879/

- RFC v3 -> v3.1:
a) Drop IOMMUContext, and rename DualStageIOMMUObject to HostIOMMUContext. HostIOMMUContext is per-vfio-container, it is exposed to vIOMMU via PCI layer. VFIO registers a PCIHostIOMMUFunc callback to PCI layer, vIOMMU could get HostIOMMUContext instance via it.
b) Check IOMMU uAPI version by VFIO_CHECK_EXTENSION
c) Add a check on VFIO_PASID_REQ availability via VFIO_GET_IOMMU_IHNFO
d) Reorder the series, put vSVA linux header file update in the beginning put the x-scalable-mode option mofification in the end of the series.
e) Dropped patch "[RFC v3 01/25] hw/pci: modify pci_setup_iommu() to set PCIIOMMUOps"
RFCv3: https://patchwork.kernel.org/cover/11356033/

- RFC v2 -> v3:
a) Introduce DualStageIOMMUObject to abstract the host IOMMU programming capability. e.g. request PASID from host, setup IOMMU nesting translation on host IOMMU. The pasid_alloc/bind_guest_page_table/iommu_cache_flush operations are moved to be DualStageIOMMUOps. Thus, DualStageIOMMUObject is an abstract layer which provides QEMU vIOMMU emulators with an explicit method to program host IOMMU.
b) Compared with RFC v2, the IOMMUContext has also been updated. It is modified to provide an abstract for vIOMMU emulators. It provides the method for pass-through modules (like VFIO) to communicate with host IOMMU. e.g. tell vIOMMU emulators about the IOMMU nesting capability on host side and report the host IOMMU DMA translation faults to vIOMMU emulators. RFC v2: https://www.spinics.net/lists/kvm/msg198556.html

- RFC v1 -> v2:
Introduce IOMMUContext to abstract the connection between VFIO and vIOMMU emulators, which is a replacement of the PCIPASIDOps in RFC v1. Modify x-scalable-mode to be string option instead of adding a new option as RFC v1 did. Refined the pasid cache management and addressed the TODOs mentioned in RFC v1.
RFC v1: https://patchwork.kernel.org/cover/11033657/

---
Eric Auger (1):
scripts/update-linux-headers: Import iommu.h

Liu Yi L (24):
scripts/update-linux-headers: Import ioasid.h
header file update VFIO/IOMMU vSVA APIs kernel 5.12-rc1 hw/pci: modify pci_setup_iommu() to set PCIIOMMUOps hw/pci: introduce pci_device_get_iommu_attr()
intel_iommu: add get_iommu_attr() callback
vfio: pass nesting requirement into vfio_get_group()
vfio: check VFIO_TYPE1_NESTING_IOMMU support hw/iommu: introduce HostIOMMUContext
hw/pci: introduce pci_device_set/unset_iommu_context()
intel_iommu: add set/unset_iommu_context callback
vfio: add HostIOMMUContext support
vfio: init HostIOMMUContext per-container
intel_iommu: sync IOMMU nesting cap info for assigned devices
intel_iommu: add virtual command capability support
intel_iommu: process PASID cache invalidation
intel_iommu: add PASID cache management infrastructure
intel_iommu: bind/unbind guest page table to host
intel_iommu: replay pasid binds after context cache invalidation
intel_iommu: do not pass down pasid bind for PASID #0
vfio: add support for flush iommu stage-1 cache
intel_iommu: process PASID-based iotlb invalidation
intel_iommu: propagate PASID-based iotlb invalidation to host
intel_iommu: process PASID-based Device-TLB invalidation
intel_iommu: modify x-scalable-mode to be string option

 hw/Kconfig                            |    3 +
 hw/alpha/typhoon.c                    |    6 +-
 hw/arm/smmu-common.c                  |    6 +-
 hw/hppa/dino.c                        |    6 +-
 hw/i386/amd_iommu.c                   |    6 +-
 hw/i386/intel_iommu.c                 | 1279 ++++++++++++++++++++++++-
 hw/i386/intel_iommu_internal.h        |  132 +++
 hw/i386/trace-events                  |    6 +
 hw/iommu/Kconfig                      |    4 +
 hw/iommu/host_iommu_context.c         |  125 +++
 hw/iommu/meson.build                  |    6 +
 hw/meson.build                        |    1 +
 hw/pci-host/designware.c              |    6 +-
 hw/pci-host/pnv_phb3.c                |    6 +-
 hw/pci-host/pnv_phb4.c                |    6 +-
 hw/pci-host/ppce500.c                 |    6 +-
 hw/pci-host/prep.c                    |    6 +-
 hw/pci-host/sabre.c                   |    6 +-
 hw/pci/pci.c                          |   73 +-
 hw/ppc/ppc440_pcix.c                  |    6 +-
 hw/ppc/spapr_pci.c                    |    6 +-
 hw/s390x/s390-pci-bus.c               |    8 +-
 hw/vfio/ap.c                          |    2 +-
 hw/vfio/ccw.c                         |    2 +-
 hw/vfio/common.c                      |  249 ++++-
 hw/vfio/pci.c                         |   26 +-
 hw/vfio/platform.c                    |    2 +-
 hw/virtio/virtio-iommu.c              |    6 +-
 include/hw/i386/intel_iommu.h         |   65 +-
 include/hw/iommu/host_iommu_context.h |   86 ++
 include/hw/pci/pci.h                  |   25 +-
 include/hw/pci/pci_bus.h              |    2 +-
 include/hw/vfio/vfio-common.h         |    7 +-
 linux-headers/linux/iommu.h           |  413 ++++++++
 linux-headers/linux/vfio.h            |   84 ++
 scripts/update-linux-headers.sh       |    2 +-
 36 files changed, 2595 insertions(+), 85 deletions(-)
 create mode 100644 hw/iommu/Kconfig
 create mode 100644 hw/iommu/host_iommu_context.c
 create mode 100644 hw/iommu/meson.build
 create mode 100644 include/hw/iommu/host_iommu_context.h
 create mode 100644 linux-headers/linux/iommu.h
Signature · 1 line
2.25.1