amd_iommu: Allow IOMMU to attach on non system bus
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest8 patches 3 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/8 | amd_iommu: Assign ioapic_as only when interrupt remapping is supported.hw/ | +10 | 0 replies | |
| 2/8 | x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrievalhw/ hw/ include/ +1 more | +39 | 2 replies | |
| 3/8 | amd_iommu: Add support for get_x86_iommu callbackhw/ | +8 | 0 replies | |
| 4/8 | intel_iommu: Add support for get_x86_iommu callbackhw/ | +8 | 1 reply | |
| 5/8 | i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookuphw/ target/ | +14 | 0 replies | |
| 6/8 | amd_iommu: Introduce root bushw/ hw/ | +5 | 0 replies | |
| 7/8 | acpi-build: Create PCI topology using amd iommu root bushw/ | +74 | 0 replies | |
| 8/8 | amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bushw/ | +51 | 0 replies |
Reviews trailers per patch
1 of 8 patches has a Reviewed-by or Acked-by.
| Patch | Clement Mathieu--Drif |
|---|---|
| 1/8 | |
| 2/8 | |
| 3/8 | |
| 4/8 | R-b |
| 5/8 | |
| 6/8 | |
| 7/8 | |
| 8/8 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 0/8] amd_iommu: Allow IOMMU to attach on non system bus
Overview
--------
This patch series enables AMD IOMMU to be attached to non-system root buses (e. g. , pcie-pxb, pci-root-bus), allowing device isolation between VFIO passthrough devices and emulated devices. This is a prerequisite for hardware-accelerated virtual IOMMU (vIOMMU) support.
Problem Statement
-----------------
The current AMD IOMMU implementation has the following limitations:
1. Only supports attachment to the default pcie.0 system bus
2. Builds IVRS (I/O Virtualization Reporting Structure) by iterating over all system buses, preventing device isolation
3. Cannot separate VFIO passthrough devices from emulated devices on different buses
This creates a fundamental issue for hardware-accelerated vIOMMU, which requires VFIO devices to be isolated from emulated devices, as the vIOMMU can only operate on passthrough devices.
Solution Approach
-----------------
This series introduces the following changes:
1. Allow AMD IOMMU to be attached to arbitrary root buses using the "bus=" parameter, while Maintaining the backward compatibility by selecting the pcie.0 by default if this parameter is missing.
2. Implement root bus discovery by traversing the parent hierarchy to find the first root bus
3. Register IOMMU operations on the discovered root bus
4. Modify IVRS generation to build IVHD (I/O Virtualization Hardware Definition) entries based on the specified root bus, traversing child buses and devices in a top-down manner
5. Introduce x86_iommu_device_get() and the get_x86_iommu() callback so the KVM layer can determine which IOMMU serves a given device.
kvm_arch_fixup_msi_route() and kvm_arch_add_msi_route_post() call x86_iommu_get_default() for every MSI route. That returns the single machine IOMMU and is correct only when that IOMMU is attached to pcie.0 and serves every device.
Once the IOMMU is attached below another root bus, it serves only the devices under that bus. The KVM layer must program interrupt-remapping routes through that IOMMU and leave devices outside its scope untranslated. The PCI core can already find the bus that owns a device, but PCIIOMMUOps cannot return the corresponding X86IOMMUState. get_x86_iommu() provides that link. x86_iommu_device_get() uses it, and both AMD and Intel IOMMU implement the callback because this KVM path is shared.
Command line
------------
qemu-system-x86_64 \
-enable-kvm -m 10G -smp 8 \
-netdev user,id=USER0,hostfwd=tcp::3333-:22 \
-cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield \
-machine q35,kernel_irqchip=split \
-global kvm-pit.lost_tick_policy=discard \
-nographic \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMG> \
-device virtio-blk-pci,num-queues=8,drive=drive0 \
-device pxb-pcie,id=pcie.1,bus=pcie.0,bus_nr=0x50 \
-device pcie-root-port,id=rp1,bus=pcie.1,chassis=3,slot=1 \
-device AMDVI-PCI,id=iommupci1,bus=rp1 \
-device amd-iommu,intremap=on,dma-remap=on,xtsup=on,pci-id=iommupci1 \
-device pcie-root-port,id=rp2,bus=pcie.2,chassis=4,slot=1 \
-device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=rp2 \
Example
-------
Example1, for below PCI configuration, IOMMU will serve all the devices under pxb-pci (i.e. Device1 and Device2) since its a first root bus, but will not serve Device3 and Device4, effectively isolating the devices under pxb-pci bus.
pcie.0 -+- pxb-pci -+- pcie-root-port1 --- AMD IOMMU
| +- pcie-root-port2 --- Device1
| \- pcie-root-port3 --- Device2
+- Device3
\- Device4Example2, for below PCI configuration, IOMMU will serve all the devices Device[1-4], since first root bus in the parent hierarchy is pcie.0
pcie.0 -+- pcie-root-port1 --- AMD IOMMU
+- pcie-root-port2 --- Device1
+- pcie-root-port3 --- Device2
+- Device3
\- Device4Example3, Below three PCI topologies will cause failure as AMD IOMMU can only attach to **PCIE** hierarchy.
[A]
pcie.0 -+- pcie-pci-bridge --- AMD IOMMU
\- Device1
[B]
pcie.0 -+- pci-bridge --- AMD IOMMU
\- Device1
[C]
pcie.0 -+- pci-bridge --- pcie-root-port --- AMD IOMMU
\- Device1
Sairaj Kodilkar (8):
amd_iommu: Assign ioapic_as only when interrupt remapping is
supported.
x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrieval
amd_iommu: Add support for get_x86_iommu callback
intel_iommu: Add support for get_x86_iommu callback i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookup
amd_iommu: Introduce root bus
acpi-build: Create PCI topology using amd iommu root bus
amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bus
hw/i386/acpi-build.c | 142 +++++++++++++++++++----------------- hw/i386/amd_iommu.c | 75 +++++++++++++++++-- hw/i386/amd_iommu.h | 3 + hw/i386/intel_iommu.c | 8 ++ hw/i386/x86-iommu-stub.c | 5 ++ hw/i386/x86-iommu.c | 16 ++++ hw/intc/ioapic.c | 3 +- include/hw/i386/x86-iommu.h | 6 ++ include/hw/pci/pci.h | 12 +++ target/i386/kvm/kvm.c | 15 +++- 10 files changed, 208 insertions(+), 77 deletions(-)
base-commit: 562bae590f194fb590beb5c65da44fc35ab9f64a
Signature · 1 line
2.34.1