Skip to contentSkip to conversation

SMMU handling for PCIe Passthrough on ARM

PATCH In review Mykyta Poturai 2 participants 5 messages Jul 24, 2025

lore
Share & tools
lore
#0 lore raw

v14 cover SMMU handling for PCIe Passthrough on ARM

This series introduces SMMU handling for PCIe passthrough on ARM. These patches should be able to be upstreamed independently from the vPCI series [1]. See [2] for notes about test cases.

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html

v13->v14:
* see individual patches

v12->v13:
* removed merged patches
* move addition of pci-passthrough enabled check in reset physdev op to a separate patch

v11->v12:
* see individual patches

v10->v11:
* see individual patches

v9->v10:
* drop iommu/arm: Add iommu_dt_xlate()
* see individual patches

v8->v9:
* see individual patches

v7->v8:
* no changes

v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")

v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")

v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into a new separate patch
* see individual patches for further details

v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device() platform_ops/iommu_ops hook

Mykyta Poturai (1):
arm/pci: add pci_passhtrough_enabled check to pci_device_reset

Stewart Hildebrand (1):
xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h | 10 ++++++++++
 xen/drivers/pci/physdev.c      |  7 +++++--
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 35 insertions(+), 3 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v14 1/2 arm/pci: add pci_passhtrough_enabled check to pci_device_reset

On Arm without pci-passthrough=on Xen does not know anything about present PCI devices due to PHYSDEVOP_pci_device_add not executing. This causes PHYSDEVOP_pci_device_reset to fail when trying to find the device that is being reset and return ENODEV.

Add explicit is_pci_passthrough_enabled check to return a more meaningful EOPNOTSUPP code.
This change should not affect x86 behavior as pci-passthrough is always enabled there.

---
v13->v14:
* clarify the commit message

v13:
* new patch

---
 xen/drivers/pci/physdev.c | 3 +++
 1 file changed, 3 insertions(+)
xen/drivers/pci/physdev.c +3 -0
index 0161a85e1e..78de67ec64 100644
@@ -74,6 +74,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
+        if ( !is_pci_passthrough_enabled() )
+            return -EOPNOTSUPP;
+
         ret = -EFAULT;
         if ( copy_from_guest(&dev_reset, arg, 1) != 0 )
             break;
Signature · 1 line
2.34.1
#3 lore raw

PATCH v14 2/2 xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

In reply to Mykyta Poturai

From: Stewart Hildebrand <[email protected]>

Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes". Due to possible platform specific dependencies of the PCI host, we rely on dom0 to initialize it and perform a PCI PHYSDEVOP calls to add each device to SMMU.

Because pci_passthrough is not always enabled on all architectures, add a new function arch_pci_device_physdevop that checks if we need to enable a subset of the PCI subsystem related to managing IOMMU configuration for PCI devices.

Enable pci_init() for initializing Xen's internal PCI subsystem, and allow PCI PHYSDEV ops when pci-passthrough is disabled.

---
This is the last patch from SMMU-PCIe on ARM series
https://patchew.org/Xen/[email protected]/
hmm. Since
dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ? Not sure if PPC/RISC-V will plan on using this check.

v13->v14:
* make x86 arch_pci_device_physdevop more clear

v12->v13:
* move introduction of reset op check to a separate patch
* update the commit message

v11->v12:
* add enabled checks to pci_device_reset
* fix style issues

v10->v11:
* always_inline -> inline
* add comments
* clarify reset sub-op handling in the commit message

v9->v10:
* move iommu_enabled check in a separate arch function
* add Stefano's RB

v8->v9:
* move iommu_enabled check inside is_pci_passthrough_enabled()

v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()

v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter

v5->v6:
* new patch - this effectively replaces
("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")

---
 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h | 10 ++++++++++
 xen/drivers/pci/physdev.c      |  6 +++---
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 33 insertions(+), 4 deletions(-)
xen/arch/arm/include/asm/pci.h +2 -0
index 37a6f14dd4..08ffcd4438 100644
@@ -151,6 +151,8 @@ void pci_generic_init_bus_range_child(struct dt_device_node *dev,
                                       struct pci_host_bridge *bridge,
                                       struct pci_config_window *cfg);
 
+bool arch_pci_device_physdevop(void);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct pci_dev;
xen/arch/arm/pci/pci.c +13 -1
index 8d9692c92e..beb1f971fa 100644
@@ -16,6 +16,7 @@
 #include <xen/device_tree.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iommu.h>
 #include <xen/param.h>
 #include <xen/pci.h>
 
@@ -75,6 +76,17 @@ static int __init acpi_pci_init(void)
 }
 #endif
 
+/*
+ * Platform-specific PCI host dependencies require dom0 to handle
+ * initialization and issue PHYSDEVOP_pci_device_add/remove calls for SMMU
+ * device registration. This check is used to enable the minimal PCI
+ * subsystem required for dom0 operation when PCI passthrough is disabled.
+ */
+bool arch_pci_device_physdevop(void)
+{
+    return iommu_enabled;
+}
+
 /* By default pci passthrough is disabled. */
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
@@ -85,7 +97,7 @@ static int __init pci_init(void)
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
      */
-    if ( !pci_passthrough_enabled )
+    if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
         return 0;
 
     if ( pci_add_segment(0) )
xen/arch/x86/include/asm/pci.h +10 -0
index 2e67cba8b9..665b321165 100644
@@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)
     return true;
 }
 
+/*
+ * Since PCI passthrough is always enabled on x86, physdevop handling doesn't
+ * need special arch-specific behavior. Current call sites work with either
+ * return value, but true is more consistent with passthrough being enabled.
+ */
+static inline bool arch_pci_device_physdevop(void)
+{
+    return true;
+}
+
 void arch_pci_init_pdev(struct pci_dev *pdev);
 
 bool pci_check_bar(const struct pci_dev *pdev, mfn_t start, mfn_t end);
xen/drivers/pci/physdev.c +3 -3
index 78de67ec64..d46501b884 100644
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev_info pdev_info;
         nodeid_t node = NUMA_NO_NODE;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_device_remove: {
         struct physdev_pci_device dev;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -74,7 +74,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
xen/include/xen/pci.h +5 -0
index ef60196653..130c2a8c1a 100644
@@ -79,6 +79,11 @@ static inline bool is_pci_passthrough_enabled(void)
     return false;
 }
 
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif
 
 struct pci_dev_info {
Signature · 1 line
2.34.1
Reviewed-by on v14 1/2
#2 lore raw

Re: [PATCH v14 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

In reply to Mykyta Poturai
Quoted 7 lines · Mykyta Poturai

On Arm without pci-passthrough=on Xen does not know anything about present PCI devices due to PHYSDEVOP_pci_device_add not executing. This causes PHYSDEVOP_pci_device_reset to fail when trying to find the device that is being reset and return ENODEV.

Add explicit is_pci_passthrough_enabled check to return a more meaningful EOPNOTSUPP code.

I.e. just like PHYSDEVOP_pci_device_{add,remove} already have it. Which also helps ...

Quoted 2 lines

This change should not affect x86 behavior as pci-passthrough is always enabled there.

... proving the no-effect aspect for x86 here.

Quoted 1 line

However, the subject prefix is entirely misleading. The change helps Arm, but is in arch-neutral code. Without "physdev" or "physdevop" anywhere in the subject, whether "pci_device_reset" refers to a function (and you merely omitted the parentheses) or something else doesn't become clear.

Also, nit: "is_pci_passthrough_enabled()" or "pci-passthrough-enabled" in the subject, both to correct spelling and to make clear that either you explicitly mean an existing predicate, or you merely mean to describe that verbally.

Jan

Acked-by on v14 2/2
#4 lore raw

Re: [PATCH v14 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

In reply to Mykyta Poturai
Quoted 12 lines · Mykyta Poturai

From: Stewart Hildebrand <[email protected]>

Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes". Due to possible platform specific dependencies of the PCI host, we rely on dom0 to initialize it and perform a PCI PHYSDEVOP calls to add each device to SMMU.

Because pci_passthrough is not always enabled on all architectures, add a new function arch_pci_device_physdevop that checks if we need to enable a subset of the PCI subsystem related to managing IOMMU configuration for PCI devices.

Quoted 6 lines

Enable pci_init() for initializing Xen's internal PCI subsystem, and allow PCI PHYSDEV ops when pci-passthrough is disabled.

Nevertheless I'd like to point out that ...

Quoted 5 lines
xen/arch/x86/include/asm/pci.h +0 -0
@@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)
     return true;
 }

xen/arch/x86/include/asm/pci.h static always_inline bool is_pci_passthrough_enabled(void) · line 68

Quoted 10 lines
xen/arch/x86/include/asm/pci.h +9 -0
 
+/*
+ * Since PCI passthrough is always enabled on x86, physdevop handling doesn't
+ * need special arch-specific behavior. Current call sites work with either
+ * return value, but true is more consistent with passthrough being enabled.
+ */
+static inline bool arch_pci_device_physdevop(void)
+{
+    return true;
+}

... the comment still isn't quite right. PCI-passthrough isn't always enabled on x86. Both AMD_IOMMU and INTEL_IOMMU Kconfig options can be used to turn respective support off. Things then work as if no IOMMU was found, which means largely "no pass-through".

Jan