SMMU handling for PCIe Passthrough on ARM
v14 cover SMMU handling for PCIe Passthrough on ARM
This series introduces SMMU handling for PCIe passthrough on ARM. These patches should be able to be upstreamed independently from the vPCI series [1]. See [2] for notes about test cases.
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html
v13->v14:
* see individual patches
v12->v13:
* removed merged patches
* move addition of pci-passthrough enabled check in reset physdev op to a separate patch
v11->v12:
* see individual patches
v10->v11:
* see individual patches
v9->v10:
* drop iommu/arm: Add iommu_dt_xlate()
* see individual patches
v8->v9:
* see individual patches
v7->v8:
* no changes
v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")
v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0
v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into a new separate patch
* see individual patches for further details
v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device() platform_ops/iommu_ops hook
Mykyta Poturai (1):
arm/pci: add pci_passhtrough_enabled check to pci_device_reset
Stewart Hildebrand (1):
xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
xen/arch/arm/include/asm/pci.h | 2 ++ xen/arch/arm/pci/pci.c | 14 +++++++++++++- xen/arch/x86/include/asm/pci.h | 10 ++++++++++ xen/drivers/pci/physdev.c | 7 +++++-- xen/include/xen/pci.h | 5 +++++ 5 files changed, 35 insertions(+), 3 deletions(-)
Signature · 1 line
2.34.1
PATCH v14 1/2 arm/pci: add pci_passhtrough_enabled check to pci_device_reset
On Arm without pci-passthrough=on Xen does not know anything about present PCI devices due to PHYSDEVOP_pci_device_add not executing. This causes PHYSDEVOP_pci_device_reset to fail when trying to find the device that is being reset and return ENODEV.
Add explicit is_pci_passthrough_enabled check to return a more meaningful EOPNOTSUPP code.
This change should not affect x86 behavior as pci-passthrough is always enabled there.
---v13->v14:
* clarify the commit message
v13:
* new patch
--- xen/drivers/pci/physdev.c | 3 +++ 1 file changed, 3 insertions(+)
xen/drivers/pci/physdev.c +3 -0
@@ -74,6 +74,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev *pdev;pci_sbdf_t sbdf; +if ( !is_pci_passthrough_enabled () ) +return -EOPNOTSUPP ; + ret = -EFAULT ;if (copy_from_guest (&dev_reset, arg,1 ) !=0 )break ;
Signature · 1 line
2.34.1
Re: [PATCH v14 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset
Quoted 7 lines · Mykyta Poturai
On Arm without pci-passthrough=on Xen does not know anything about present PCI devices due to
PHYSDEVOP_pci_device_addnot executing. This causesPHYSDEVOP_pci_device_resetto fail when trying to find the device that is being reset and return ENODEV.Add explicit
is_pci_passthrough_enabledcheck to return a more meaningful EOPNOTSUPP code.
I.e. just like PHYSDEVOP_pci_device_{add,remove} already have it. Which also helps ...
Quoted 2 lines
This change should not affect x86 behavior as pci-passthrough is always enabled there.
... proving the no-effect aspect for x86 here.
Quoted 1 line
Signed-off-by Mykyta Poturai <[email protected]>
However, the subject prefix is entirely misleading. The change helps Arm, but is in arch-neutral code. Without "physdev" or "physdevop" anywhere in the subject, whether "pci_device_reset" refers to a function (and you merely omitted the parentheses) or something else doesn't become clear.
Also, nit: "is_pci_passthrough_enabled()" or "pci-passthrough-enabled" in the subject, both to correct spelling and to make clear that either you explicitly mean an existing predicate, or you merely mean to describe that verbally.
Jan
PATCH v14 2/2 xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
In reply to Mykyta PoturaiFrom: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes". Due to possible platform specific dependencies of the PCI host, we rely on dom0 to initialize it and perform a PCI PHYSDEVOP calls to add each device to SMMU.
Because pci_passthrough is not always enabled on all architectures, add a new function arch_pci_device_physdevop that checks if we need to enable a subset of the PCI subsystem related to managing IOMMU configuration for PCI devices.
Enable pci_init() for initializing Xen's internal PCI subsystem, and allow PCI PHYSDEV ops when pci-passthrough is disabled.
---
This is the last patch from SMMU-PCIe on ARM series
https://patchew.org/Xen/[email protected]/
hmm. Since
dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ? Not sure if PPC/RISC-V will plan on using this check.
v13->v14:
* make x86 arch_pci_device_physdevop more clear
v12->v13:
* move introduction of reset op check to a separate patch
* update the commit message
v11->v12:
* add enabled checks to pci_device_reset
* fix style issues
v10->v11:
* always_inline -> inline
* add comments
* clarify reset sub-op handling in the commit message
v9->v10:
* move iommu_enabled check in a separate arch function
* add Stefano's RB
v8->v9:
* move iommu_enabled check inside is_pci_passthrough_enabled()
v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()
v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter
v5->v6:
* new patch - this effectively replaces
("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
--- xen/arch/arm/include/asm/pci.h | 2 ++ xen/arch/arm/pci/pci.c | 14 +++++++++++++- xen/arch/x86/include/asm/pci.h | 10 ++++++++++ xen/drivers/pci/physdev.c | 6 +++--- xen/include/xen/pci.h | 5 +++++ 5 files changed, 33 insertions(+), 4 deletions(-)
xen/arch/arm/include/asm/pci.h +2 -0
@@ -151,6 +151,8 @@ void pci_generic_init_bus_range_child(struct dt_device_node *dev,struct pci_host_bridge *bridge, struct pci_config_window *cfg); +bool arch_pci_device_physdevop (void ); +#else /*!CONFIG_HAS_PCI*/ struct pci_dev ;
xen/arch/arm/pci/pci.c +13 -1
@@ -16,6 +16,7 @@#include <xen/device_tree.h> #include <xen/errno.h> #include <xen/init.h> +#include <xen/iommu.h> #include <xen/param.h> #include <xen/pci.h>
@@ -75,6 +76,17 @@ static int __init acpi_pci_init(void) }#endif +/* +* Platform-specific PCI host dependencies require dom0 to handle +* initialization and issue PHYSDEVOP_pci_device_add/remove calls for SMMU +* device registration. This check is used to enable the minimal PCI +* subsystem required for dom0 operation when PCI passthrough is disabled. +*/ +bool arch_pci_device_physdevop (void ) +{ +return iommu_enabled; +} +/* By default pci passthrough is disabled. */ bool __read_mostly pci_passthrough_enabled;boolean_param ("pci-passthrough" , pci_passthrough_enabled);
@@ -85,7 +97,7 @@ static int __init pci_init(void)* Enable PCI passthrough when has been enabled explicitly * (pci-passthrough=on). */ -if ( !pci_passthrough_enabled ) +if ( !is_pci_passthrough_enabled () && !arch_pci_device_physdevop () )return 0 ;if (pci_add_segment (0 ) )
xen/arch/x86/include/asm/pci.h +10 -0
@@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)return true; } +/* +* Since PCI passthrough is always enabled on x86, physdevop handling doesn't +* need special arch-specific behavior. Current call sites work with either +* return value, but true is more consistent with passthrough being enabled. +*/ +static inline bool arch_pci_device_physdevop (void ) +{ +return true; +} +void arch_pci_init_pdev (struct pci_dev *pdev);bool pci_check_bar (const struct pci_dev *pdev,mfn_t start,mfn_t end);
xen/drivers/pci/physdev.c +3 -3
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev_info pdev_info;nodeid_t node =NUMA_NO_NODE ; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !arch_pci_device_physdevop () )return -EOPNOTSUPP ; ret = -EFAULT ;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)case PHYSDEVOP_pci_device_remove: {struct physdev_pci_device dev; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !arch_pci_device_physdevop () )return -EOPNOTSUPP ; ret = -EFAULT ;
@@ -74,7 +74,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev *pdev;pci_sbdf_t sbdf; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !arch_pci_device_physdevop () )return -EOPNOTSUPP ; ret = -EFAULT ;
xen/include/xen/pci.h +5 -0
@@ -79,6 +79,11 @@ static inline bool is_pci_passthrough_enabled(void)return false; } +static inline bool arch_pci_device_physdevop (void ) +{ +return false; +} +#endif struct pci_dev_info {
Signature · 1 line
2.34.1
Re: [PATCH v14 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
Quoted 12 lines · Mykyta Poturai
From: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes". Due to possible platform specific dependencies of the PCI host, we rely on dom0 to initialize it and perform a PCI PHYSDEVOP calls to add each device to SMMU.
Because
pci_passthroughis not always enabled on all architectures, add a new functionarch_pci_device_physdevopthat checks if we need to enable a subset of the PCI subsystem related to managing IOMMU configuration for PCI devices.
Quoted 6 lines
Enable
pci_init()for initializing Xen's internal PCI subsystem, and allow PCI PHYSDEV ops when pci-passthrough is disabled.Signed-off-by Stewart Hildebrand <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>Reviewed-by Stefano Stabellini <[email protected]>
Nevertheless I'd like to point out that ...
Quoted 5 lines
xen/arch/x86/include/asm/pci.h +0
-0@@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)return true; }
xen/arch/x86/include/asm/pci.h static always_inline bool is_pci_passthrough_enabled(void) · line 68
Quoted 10 lines
xen/arch/x86/include/asm/pci.h +9
-0+/* +* Since PCI passthrough is always enabled on x86, physdevop handling doesn't +* need special arch-specific behavior. Current call sites work with either +* return value, but true is more consistent with passthrough being enabled. +*/ +static inline bool arch_pci_device_physdevop (void ) +{ +return true; +}
... the comment still isn't quite right. PCI-passthrough isn't always enabled on x86. Both AMD_IOMMU and INTEL_IOMMU Kconfig options can be used to turn respective support off. Things then work as if no IOMMU was found, which means largely "no pass-through".
Jan