SMMU handling for PCIe Passthrough on ARM
PATCH v8 1/8 iommu/arm: Add iommu_dt_xlate()
In reply to Mykyta PoturaiFrom: Oleksandr Tyshchenko <[email protected]>
Move code for processing DT IOMMU specifier to a separate helper. This helper will be re-used for adding PCI devices by the subsequent patches as we will need exact the same actions for processing DT PCI-IOMMU specifier.
While at it introduce DT_NO_IOMMU to avoid magic "1".
---v7->v8:
* explain NO_IOMMU better and rename to DT_NO_IOMMU
v6->v7:
* explained NO_IOMMU in comments
v5->v6:
* pass ops parameter to iommu_dt_xlate()
* add Julien's R-b
v4->v5:
* rebase on top of "dynamic node programming using overlay dtbo" series
* move #define NO_IOMMU 1 to header
* s/these/this/ inside comment
v3->v4:
* make dt_phandle_args *iommu_spec const
* move !ops->add_device check to helper
v2->v3:
* no change
v1->v2:
* no change
downstream->v1:
* trivial rebase
* s/dt_iommu_xlate/iommu_dt_xlate/
(cherry picked from commit c26bab0415ca303df86aba1d06ef8edc713734d3 from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
--- xen/drivers/passthrough/device_tree.c | 48 +++++++++++++++++---------- xen/include/xen/iommu.h | 8 +++++ 2 files changed, 38 insertions(+), 18 deletions(-)
xen/drivers/passthrough/device_tree.c +30 -18
@@ -137,6 +137,30 @@ int iommu_release_dt_devices(struct domain *d)return 0 ; } +static int iommu_dt_xlate (struct device *dev, +const struct dt_phandle_args *iommu_spec, +const struct iommu_ops *ops) +{ +int rc; + +if ( !ops->dt_xlate ) +return -EINVAL ; + +if ( !dt_device_is_available (iommu_spec->np) ) +return DT_NO_IOMMU ; + + rc =iommu_fwspec_init (dev, &iommu_spec->np->dev); +if ( rc ) +return rc; + +/* +* Provide DT IOMMU specifier which describes the IOMMU master +* interfaces of that device (device IDs, etc) to the driver. +* The driver is responsible to decide how to interpret them. +*/ +return ops->dt_xlate (dev, iommu_spec); +} +int iommu_remove_dt_device (struct dt_device_node *np) {const struct iommu_ops *ops =iommu_get_ops ();
@@ -146,7 +170,7 @@ int iommu_remove_dt_device(struct dt_device_node *np)ASSERT (rw_is_locked (&dt_host_lock));if ( !iommu_enabled ) -return 1 ; +return DT_NO_IOMMU ;if ( !ops )return -EOPNOTSUPP ;
@@ -187,12 +211,12 @@ int iommu_add_dt_device(struct dt_device_node *np)const struct iommu_ops *ops =iommu_get_ops ();struct dt_phandle_args iommu_spec;struct device *dev =dt_to_dev (np); -int rc =1 , index =0 ; +int rc =DT_NO_IOMMU , index =0 ;ASSERT (system_state < SYS_STATE_active ||rw_is_locked (&dt_host_lock));if ( !iommu_enabled ) -return 1 ; +return DT_NO_IOMMU ;if ( !ops )return -EINVAL ;
@@ -215,27 +239,15 @@ int iommu_add_dt_device(struct dt_device_node *np) {/* * The driver which supports generic IOMMU DT bindings must have -* these callback implemented. +* this callback implemented. */ -if ( !ops->add_device || !ops->dt_xlate ) +if ( !ops->add_device ) { rc = -EINVAL ; goto fail; } -if ( !dt_device_is_available (iommu_spec.np) ) -break ; - - rc =iommu_fwspec_init (dev, &iommu_spec.np->dev); -if ( rc ) -break ; - -/* -* Provide DT IOMMU specifier which describes the IOMMU master -* interfaces of that device (device IDs, etc) to the driver. -* The driver is responsible to decide how to interpret them. -*/ - rc = ops->dt_xlate (dev, &iommu_spec); + rc =iommu_dt_xlate (dev, &iommu_spec, ops);if ( rc )break ;
xen/include/xen/iommu.h +8 -0
@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1 +#endif /* HAS_DEVICE_TREE */ struct page_info ;
Signature · 1 line
2.34.1
v8 cover SMMU handling for PCIe Passthrough on ARM
This series introduces SMMU handling for PCIe passthrough on ARM. These patches should be able to be upstreamed independently from the vPCI series [1]. See [2] for notes about test cases.
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html
v7-v8:
* no changes
v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")
v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0
v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into a new separate patch
* see individual patches for further details
v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device() platform_ops/iommu_ops hook
Oleksandr Andrushchenko (1):
xen/arm: smmuv2: Add PCI devices support for SMMUv2
Oleksandr Tyshchenko (2):
iommu/arm: Add iommu_dt_xlate()
iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
Rahul Singh (3):
xen/arm: smmuv3: Add PCI devices support for SMMUv3 xen/arm: Fix mapping for PCI bridge mmio region xen/arm: Map ITS doorbell register to IOMMU page tables
Stewart Hildebrand (2):
iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
xen/arch/arm/device.c | 2 +- xen/arch/arm/pci/pci.c | 5 +- xen/arch/arm/vgic-v3-its.c | 20 +++ xen/common/device-tree/device-tree.c | 91 ++++++++++++ xen/drivers/passthrough/arm/smmu-v3.c | 117 ++++++++++++++-- xen/drivers/passthrough/arm/smmu.c | 190 ++++++++++++++++++++------ xen/drivers/passthrough/device_tree.c | 97 ++++++++++--- xen/drivers/passthrough/iommu.c | 13 ++ xen/drivers/pci/physdev.c | 4 +- xen/include/xen/device_tree.h | 23 ++++ xen/include/xen/iommu.h | 40 +++++- 11 files changed, 524 insertions(+), 78 deletions(-)
Signature · 1 line
2.34.1
PATCH v8 2/8 iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
From: Oleksandr Tyshchenko <[email protected]>
The main purpose of this patch is to add a way to register PCI device (which is behind the IOMMU) using the generic PCI-IOMMU DT bindings [1] before assigning that device to a domain.
This behaves similarly to the existing iommu_add_dt_device API, except it handles PCI devices, and it is to be invoked from the add_device hook in the SMMU driver.
The function dt_map_id to translate an ID through a downstream mapping (which is also suitable for mapping Requester ID) was borrowed from Linux (v5.10-rc6) and updated according to the Xen code base.
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
---
Regarding pci_for_each_dma_alias question: getting host bridge node directly seems like a simpler solution with the same result. AFAIU with pci_for_each_dma_alias in linux we would arrive to the host brige node anyway, but also try to call dt_map_id for each device along the way. I am not sure why exactly it is done this way in linux, as according to the pci-iommu.txt, iommu-map node can only be present in the PCI root.
v7->v8:
* ENOSYS->EOPNOTSUPP
* move iommu_add_pci_sideband_ids to iommu.c to fix x86 build
* simplify iommu_add_pci_sideband_ids
* add docstrings to iommu_add_{dt_}pci_sideband_ids
v6->v7:
* put iommu_add_pci_sideband_ids under ifdef
* remove ifdef CONFIG_APCI
* style: add newline for symmetry
v5->v6:
* pass ops to iommu_dt_xlate()
v4->v5:
* style: add newlines after variable declarations and before return in iommu.h
* drop device_is_protected() check in iommu_add_dt_pci_sideband_ids()
* rebase on top of ("dynamic node programming using overlay dtbo") series
* fix typo in commit message
* remove #ifdef around dt_map_id() prototype
* move dt_map_id() to xen/common/device_tree.c
* add function name in error prints
* use dprintk for debug prints
* use GENMASK and #include <xen/bitops.h>
* fix typo in comment
* remove unnecessary (int) cast in loop condition
* assign *id_out and return success in case of no translation in dt_map_id()
* don't initialize local variable unnecessarily
* return error in case of ACPI/no DT in iommu_add_{dt_}pci_sideband_ids()
v3->v4:
* wrap #include <asm/acpi.h> and if ( acpi_disabled ) in #ifdef CONFIG_ACPI
* fix Michal's remarks about style, parenthesis, and print formats
* remove !ops->dt_xlate check since it is already in iommu_dt_xlate helper
* rename s/iommu_dt_pci_map_id/dt_map_id/ because it is generic, not specific to iommu
* update commit description
v2->v3:
* new patch title (was: iommu/arm: Introduce iommu_add_dt_pci_device API)
* renamed function
from: iommu_add_dt_pci_device
to: iommu_add_dt_pci_sideband_ids
* removed stale ops->add_device check
* iommu.h: add empty stub iommu_add_dt_pci_sideband_ids for !HAS_DEVICE_TREE
* iommu.h: add iommu_add_pci_sideband_ids helper
* iommu.h: don't wrap prototype in #ifdef CONFIG_HAS_PCI
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/
v1->v2:
* remove extra devfn parameter since pdev fully describes the device
* remove ops->add_device() call from iommu_add_dt_pci_device(). Instead, rely on the existing iommu call in iommu_add_device().
* move the ops->add_device and ops->dt_xlate checks earlier
downstream->v1:
* rebase
* add const qualifier to struct dt_device_node *np arg in dt_map_id()
* add const qualifier to struct dt_device_node *np declaration in iommu_add_pci_device()
* use stdint.h types instead of u8/u32/etc...
* rename functions:
s/dt_iommu_xlate/iommu_dt_xlate/
s/dt_map_id/iommu_dt_pci_map_id/ s/iommu_add_pci_device/iommu_add_dt_pci_device/
* add device_is_protected check in iommu_add_dt_pci_device
* wrap prototypes in CONFIG_HAS_PCI
(cherry picked from commit 734e3bf6ee77e7947667ab8fa96c25b349c2e1da from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
--- xen/common/device-tree/device-tree.c | 91 +++++++++++++++++++++++++++ xen/drivers/passthrough/device_tree.c | 42 +++++++++++++ xen/drivers/passthrough/iommu.c | 13 ++++ xen/include/xen/device_tree.h | 23 +++++++ xen/include/xen/iommu.h | 32 +++++++++- 5 files changed, 200 insertions(+), 1 deletion(-)
xen/common/device-tree/device-tree.c +91 -0
@@ -10,6 +10,7 @@* published by the Free Software Foundation. */ +#include <xen/bitops.h> #include <xen/types.h> #include <xen/init.h> #include <xen/guest_access.h>
@@ -2243,6 +2244,96 @@ int dt_get_pci_domain_nr(struct dt_device_node *node)return (u16 )domain; } +int dt_map_id (const struct dt_device_node *np,uint32_t id, +const char *map_name,const char *map_mask_name, +struct dt_device_node **target,uint32_t *id_out) +{ +uint32_t map_mask, masked_id, map_len; +const __be32 *map =NULL ; + +if ( !np || !map_name || (!target && !id_out) ) +return -EINVAL ; + + map =dt_get_property (np, map_name, &map_len); +if ( !map ) + { +if ( target ) +return -ENODEV ; + +/* Otherwise, no map implies no translation */ + *id_out = id; +return 0 ; + } + +if ( !map_len || (map_len % (4 *sizeof (*map))) ) + { +printk (XENLOG_ERR "%s(): %s: Error: Bad %s length: %u\n" , __func__, + np->full_name, map_name, map_len); +return -EINVAL ; + } + +/* The default is to select all bits. */ + map_mask =GENMASK (31 ,0 ); + +/* +* Can be overridden by "{iommu,msi}-map-mask" property. +* If dt_property_read_u32() fails, the default is used. +*/ +if ( map_mask_name ) +dt_property_read_u32 (np, map_mask_name, &map_mask); + + masked_id = map_mask & id; +for ( ; map_len >0 ; map_len -=4 *sizeof (*map), map +=4 ) + { +struct dt_device_node *phandle_node; +uint32_t id_base =be32_to_cpup (map +0 ); +uint32_t phandle =be32_to_cpup (map +1 ); +uint32_t out_base =be32_to_cpup (map +2 ); +uint32_t id_len =be32_to_cpup (map +3 ); + +if ( id_base & ~map_mask ) + { +printk (XENLOG_ERR "%s(): %s: Invalid %s translation - %s-mask (0x%" PRIx32") ignores id-base (0x%" PRIx32")\n" , + __func__, np->full_name, map_name, map_name, map_mask, + id_base); +return -EFAULT ; + } + +if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) ) +continue ; + + phandle_node =dt_find_node_by_phandle (phandle); +if ( !phandle_node ) +return -ENODEV ; + +if ( target ) + { +if ( !*target ) + *target = phandle_node; + +if ( *target != phandle_node ) +continue ; + } + +if ( id_out ) + *id_out = masked_id - id_base + out_base; + +dprintk (XENLOG_DEBUG ,"%s: %s, using mask %08" PRIx32", id-base: %08" PRIx32", out-base: %08" PRIx32", length: %08" PRIx32", id: %08" PRIx32" -> %08" PRIx32"\n" , + np->full_name, map_name, map_mask, id_base, out_base, id_len, id, + masked_id - id_base + out_base); +return 0 ; + } + +dprintk (XENLOG_DEBUG ,"%s: no %s translation for id 0x%" PRIx32" on %s\n" , + np->full_name, map_name, id, + (target && *target) ? (*target)->full_name :NULL ); + +if ( id_out ) + *id_out = id; + +return 0 ; +} +/* * Local variables: * mode: C
xen/drivers/passthrough/device_tree.c +42 -0
48 lines not on this page · raw message
xen/drivers/passthrough/iommu.c +13 -0
@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86
@@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; } +int iommu_add_pci_sideband_ids (struct pci_dev *pdev) +{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +} +/* * Local variables: * mode: C
xen/include/xen/device_tree.h +23 -0
@@ -946,6 +946,29 @@ int dt_count_phandle_with_args(const struct dt_device_node *np,*/ int dt_get_pci_domain_nr (struct dt_device_node *node); +/** +* dt_map_id - Translate an ID through a downstream mapping. +* @np: root complex device node. +* @id: device ID to map. +* @map_name: property name of the map to use. +* @map_mask_name: optional property name of the mask to use. +* @target: optional pointer to a target device node. +* @id_out: optional pointer to receive the translated ID. +* +* Given a device ID, look up the appropriate implementation-defined +* platform ID and/or the target device which receives transactions on that +* ID, as per the "iommu-map" and "msi-map" bindings. Either of @target or +* @id_out may be NULL if only the other is required. If @target points to +* a non-NULL device node pointer, only entries targeting that node will be +* matched; if it points to a NULL value, it will receive the device node of +* the first matching target phandle, with a reference held. +* +* Return: 0 on success or a standard error code on failure. +*/ +int dt_map_id (const struct dt_device_node *np,uint32_t id, +const char *map_name,const char *map_mask_name, +struct dt_device_node **target,uint32_t *id_out); +struct dt_device_node *dt_find_node_by_phandle (dt_phandle handle);#ifdef CONFIG_DEVICE_TREE_DEBUG
xen/include/xen/iommu.h +31 -1
@@ -215,7 +215,8 @@ int iommu_dt_domain_init(struct domain *d);int iommu_release_dt_devices (struct domain *d);/* -* Helper to add master device to the IOMMU using generic IOMMU DT bindings. +* Helpers to add master device to the IOMMU using generic (PCI-)IOMMU +* DT bindings. * * Return values: * 0 : device is protected by an IOMMU
@@ -225,6 +226,19 @@ int iommu_release_dt_devices(struct domain *d);*/ int iommu_add_dt_device (struct dt_device_node *np); +/* +* Fills out the device's IOMMU fwspec with IOMMU ids from the DT. +* Ids are specified in the iommu-map property in the host bridge node. +* More information on the iommu-map property format can be found in +* Documentation/devicetree/bindings/pci/pci-iommu.txt from Linux. +* +* Return values: +* 0 : iommu_fwspec is filled out successfully. +* <0 : error while filling out the iommu_fwspec. +* >0 : IOMMU is not enabled/present or device is not connected to it. +*/ +int iommu_add_dt_pci_sideband_ids (struct pci_dev *pdev); +int iommu_do_dt_domctl (struct xen_domctl *domctl,struct domain *d,XEN_GUEST_HANDLE_PARAM (xen_domctl_t) u_domctl);
@@ -246,8 +260,24 @@ int iommu_remove_dt_device(struct dt_device_node *np);#define DT_NO_IOMMU 1 +#else /* !HAS_DEVICE_TREE */ +static inline int iommu_add_dt_pci_sideband_ids (struct pci_dev *pdev) +{ +return -EOPNOTSUPP ; +} +#endif /* HAS_DEVICE_TREE */ +/* +* Fills out the device's IOMMU fwspec with IOMMU ids. +* +* Return values: +* 0 : iommu_fwspec is filled out successfully. +* <0 : error while filling out the iommu_fwspec. +* >0 : IOMMU is not enabled/present or device is not connected to it. +*/ +int iommu_add_pci_sideband_ids (struct pci_dev *pdev); +struct page_info ;/*
Signature · 1 line
2.34.1
PATCH v8 4/8 xen/arm: smmuv2: Add PCI devices support for SMMUv2
In reply to Mykyta PoturaiFrom: Oleksandr Andrushchenko <[email protected]>
Implement support for PCI devices in the SMMU driver. Make arm_smmu_master structure to hold a pointer to the device to allow it to hold PCI devices. Trigger iommu-map parsing when new PCI device is added. Add checks to assign/deassign functions to ensure PCI devices are handled correctly. Implement basic quarantining.
All pci devices are automatically assigned to hardware domain if it exists to ensure it can probe them.
TODO:
Implement scratch page quarantining support.
---v7->v8:
* no change
v6->v7:
* use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to support future dom0less use case when dom0 is using vPCI
* remove stale todo in dev_get_dev_node
* don't print ""
* remove redundant dt_device_is_protected check
* remove assign/deassing prints
* change assign logic to remove reassign reimplementation
* check if pdev->domain exists before assigning to it
* explain pdev->devfn check
* make reassign check stricter and update comment
v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()
v4->v5:
* assign device to pdev->domain (usually dom0) by default in add_device() hook
* deassign from hwdom
* rebase on top of ("dynamic node programming using overlay dtbo") series
* remove TODO in comment about device prints
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")
v3->v4:
* add new device_is_protected check in add_device hook to match SMMUv3 and IPMMU-VMSA drivers
v2->v3:
* invoke iommu_add_pci_sideband_ids() from add_device hook
v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions (i.e. devfn != pdev->devfn)
downstream->v1:
* wrap unused function in #ifdef 0
* remove the remove_device() stub since it was submitted separately to the list [XEN][PATCH v6 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops
https://lists.xenproject.org/archives/html/xen-devel/2023-05/msg00204.html
* arm_smmu_(de)assign_dev: return error instead of crashing system
* update condition in arm_smmu_reassign_dev
* style fixup
* add && !is_hardware_domain(d) into condition in arm_smmu_assign_dev()
(cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
the downstream branch spider-master from
https://github.com/xen-troops/xen.git)
--- xen/drivers/passthrough/arm/smmu.c | 190 ++++++++++++++++++++++------- 1 file changed, 147 insertions(+), 43 deletions(-)
xen/drivers/passthrough/arm/smmu.c +147 -43
@@ -132,11 +132,21 @@ enum irqreturn {typedef enum irqreturn irqreturn_t ; -/* Device logger functions -* TODO: Handle PCI -*/ -#define dev_print (dev, lvl, fmt, ...) \ - printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__) +/* Device logger functions */ +#ifndef CONFIG_HAS_PCI +#define dev_print (dev, lvl, fmt, ...) \ + printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__) +#else +#define dev_print (dev, lvl, fmt, ...) ({ \ + if ( !dev_is_pci((dev)) ) \ + printk(lvl "smmu: %s: " fmt, dev_name((dev)), ## __VA_ARGS__); \ + else \ + { \ + struct pci_dev *pdev = dev_to_pci((dev)); \ + printk(lvl "smmu: %pp: " fmt, &pdev->sbdf, ## __VA_ARGS__); \ + } \ +}) +#endif #define dev_dbg (dev, fmt, ...) dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__)#define dev_notice (dev, fmt, ...) dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__)
@@ -188,6 +198,7 @@ static void __iomem *devm_ioremap_resource(struct device *dev,* Xen: PCI functions * TODO: It should be implemented when PCI will be supported */ +#if 0 /* unused */ #define to_pci_dev (dev) (NULL) static inline int pci_for_each_dma_alias(struct pci_dev *pdev, int (*fn) (struct pci_dev *pdev,
@@ -197,6 +208,7 @@ static inline int pci_for_each_dma_alias(struct pci_dev *pdev,BUG (); return 0; } +#endif /* Xen: misc */ #define PHYS_MASK_SHIFT PADDR_BITS
@@ -631,7 +643,7 @@ struct arm_smmu_master_cfg {for (i =0 ; idx = (cfg)->smendx[i], (i) < (num); ++(i)) struct arm_smmu_master { -struct device_node *of_node; +struct device *dev;struct rb_node node;struct arm_smmu_master_cfg cfg; };
@@ -723,7 +735,7 @@ arm_smmu_get_fwspec(struct arm_smmu_master_cfg *cfg) {struct arm_smmu_master *master =container_of (cfg, struct arm_smmu_master, cfg); -return dev_iommu_fwspec_get (&master->of_node->dev); +return dev_iommu_fwspec_get (master->dev); }static void parse_driver_options (struct arm_smmu_device *smmu)
@@ -742,21 +754,11 @@ static void parse_driver_options(struct arm_smmu_device *smmu)static struct device_node *dev_get_dev_node (struct device *dev) { -#if 0 /* Xen: TODO: Add support for PCI */ -if (dev_is_pci (dev)) { -struct pci_bus *bus =to_pci_dev (dev)->bus; - -while (!pci_is_root_bus (bus)) - bus = bus->parent; -return bus->bridge->parent->of_node; - } -#endif -return dev->of_node; }static struct arm_smmu_master *find_smmu_master (struct arm_smmu_device *smmu, -struct device_node *dev_node) +struct device *dev) {struct rb_node *node = smmu->masters.rb_node;
@@ -765,9 +767,9 @@ static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu, master = container_of(node, struct arm_smmu_master, node); -if (dev_node < master->of_node) +if (dev < master->dev) node = node->rb_left; -else if (dev_node > master->of_node) +else if (dev > master->dev) node = node->rb_right;else return master;
@@ -802,9 +804,9 @@ static int insert_smmu_master(struct arm_smmu_device *smmu, = container_of(*new, struct arm_smmu_master, node); parent = *new; -if (master->of_node < this->of_node) +if (master->dev < this->dev) new = &((*new)->rb_left); -else if (master->of_node > this->of_node) +else if (master->dev > this->dev) new = &((*new)->rb_right);else return -EEXIST ;
@@ -836,28 +838,30 @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,struct arm_smmu_master *master;struct device_node *dev_node =dev_get_dev_node (dev); - master =find_smmu_master (smmu, dev_node); + master =find_smmu_master (smmu, dev);if (master) {dev_err (dev, -"rejecting multiple registrations for master device %s\n" , - dev_node->name); +"rejecting multiple registrations for master device\n" );return -EBUSY ; } master =devm_kzalloc (dev,sizeof (*master),GFP_KERNEL );if (!master)return -ENOMEM ; - master->of_node = dev_node; + master->dev = dev; -/* Xen: Let Xen know that the device is protected by an SMMU */ -dt_device_set_protected (dev_node); +if ( !dev_is_pci (dev) ) + { +/* Xen: Let Xen know that the device is protected by an SMMU */ +dt_device_set_protected (dev_node); + }for (i =0 ; i < fwspec->num_ids; ++i) {if (!(smmu->features &ARM_SMMU_FEAT_STREAM_MATCH ) && (fwspec->ids[i] >= smmu->num_mapping_groups)) {dev_err (dev, -"stream ID for master device %s greater than maximum allowed (%d)\n" , - dev_node->name, smmu->num_mapping_groups); +"SMMU stream ID %d is greater than maximum allowed (%d)\n" , + fwspec->ids[i], smmu->num_mapping_groups);return -ERANGE ; } master->cfg.smendx[i] =INVALID_SMENDX ;
@@ -872,7 +876,7 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,struct device_node *dev_node =dev_get_dev_node (dev);int ret; - master =find_smmu_master (smmu, dev_node); + master =find_smmu_master (smmu, dev);if (master ==NULL ) { dev_err(dev, "No registrations found for master device %s\n",
@@ -884,8 +888,9 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,if (ret)return ret; -/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */ - dev_node->is_protected = false; +if ( !dev_is_pci (dev) ) +/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */ + dev_node->is_protected = false;kfree (master);return 0 ;
@@ -914,6 +919,12 @@ static int register_smmu_master(struct arm_smmu_device *smmu, fwspec); } +/* Forward declaration */ +static int arm_smmu_assign_dev (struct domain *d,u8 devfn, +struct device *dev,u32 flag); +static int arm_smmu_deassign_dev (struct domain *d,uint8_t devfn, +struct device *dev); +/* * The driver which supports generic IOMMU DT bindings must have this * callback implemented.
@@ -938,6 +949,23 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev) {struct arm_smmu_device *smmu;struct iommu_fwspec *fwspec; +int ret; + +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); +int ret; + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + + ret =iommu_add_pci_sideband_ids (pdev); +if ( ret <0 ) +iommu_fwspec_free (dev); + } +#endif fwspec =dev_iommu_fwspec_get (dev);if (fwspec ==NULL )
@@ -947,7 +975,25 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)if (smmu ==NULL )return -ENXIO ; -return arm_smmu_dt_add_device_legacy (smmu, dev, fwspec); + ret =arm_smmu_dt_add_device_legacy (smmu, dev, fwspec); +if ( ret ) +return ret; + +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* +* During PHYSDEVOP_pci_device_add, Xen does not assign the +* device, so we must do it here. +*/ +if ( pdev->domain ) + ret =arm_smmu_assign_dev (pdev->domain, devfn, dev,0 ); + } +#endif + +return ret; } static int arm_smmu_dt_xlate_generic(struct device *dev,
@@ -970,11 +1016,10 @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev) {struct arm_smmu_device *smmu;struct arm_smmu_master *master =NULL ; -struct device_node *dev_node =dev_get_dev_node (dev);spin_lock (&arm_smmu_devices_lock);list_for_each_entry (smmu, &arm_smmu_devices, list) { - master =find_smmu_master (smmu, dev_node); + master =find_smmu_master (smmu, dev);if (master)break ; }
@@ -2066,6 +2111,7 @@ static bool arm_smmu_capable(enum iommu_cap cap) }#endif +#if 0 /* Not used */ static int __arm_smmu_get_pci_sid (struct pci_dev *pdev,u16 alias,void *data) { *((u16 *)data) = alias;
@@ -2076,6 +2122,7 @@ static void __arm_smmu_release_pci_iommudata(void *data) {kfree (data); } +#endif static int arm_smmu_add_device (struct device *dev) {
@@ -2083,12 +2130,13 @@ static int arm_smmu_add_device(struct device *dev)struct arm_smmu_master_cfg *cfg;struct iommu_group *group;void (*releasefn)(void *data) =NULL ; -int ret; smmu =find_smmu_for_device (dev);if (!smmu)return -ENODEV ; +/* There is no need to distinguish here, thanks to PCI-IOMMU DT bindings */ +#if 0 if (dev_is_pci (dev)) {struct pci_dev *pdev =to_pci_dev (dev);struct iommu_fwspec *fwspec;
@@ -2113,10 +2161,12 @@ static int arm_smmu_add_device(struct device *dev) &fwspec->ids[0]); releasefn = __arm_smmu_release_pci_iommudata; cfg->smmu = smmu; - }else { + } else +#endif + {struct arm_smmu_master *master; - master =find_smmu_master (smmu, dev->of_node); + master =find_smmu_master (smmu, dev);if (!master) {return -ENODEV ; }
@@ -2784,6 +2834,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,return -ENOMEM ; } +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + +ASSERT (pcidevs_locked ()); + +write_lock (&pdev->domain->pci_lock); +list_del (&pdev->domain_list); +write_unlock (&pdev->domain->pci_lock); + + pdev->domain = d; + +write_lock (&d->pci_lock); +list_add (&pdev->domain_list, &d->pdev_list); +write_unlock (&d->pci_lock); + +/* dom_io is used as a sentinel for quarantined devices */ +if ( d == dom_io ) + { +struct iommu_domain *domain =dev_iommu_domain (dev); +if ( !iommu_quarantine ) +return 0 ; + +if ( domain && domain->priv ) +arm_smmu_deassign_dev (domain->priv->cfg.domain, devfn, dev); + +return 0 ; + } + } +#endif +if (!dev_iommu_group (dev)) { ret =arm_smmu_add_device (dev);if (ret)
@@ -2833,11 +2919,27 @@ out:return ret; } -static int arm_smmu_deassign_dev (struct domain *d,struct device *dev) +static int arm_smmu_deassign_dev (struct domain *d,uint8_t devfn, +struct device *dev) {struct iommu_domain *domain =dev_iommu_domain (dev);struct arm_smmu_xen_domain *xen_domain; +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + +/* dom_io is used as a sentinel for quarantined devices */ +if ( d == dom_io ) +return 0 ; + } +#endif + xen_domain =dom_iommu (d)->arch.priv;if (!domain || domain->priv->cfg.domain != d) {
@@ -2864,14 +2966,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t, {int ret =0 ; -/* Don't allow remapping on other domain than hwdom */ -if ( t && !is_hardware_domain (t) ) +/* Don't allow remapping on other domain than hwdom +* or dom_io for PCI devices +*/ +if ( t && !is_hardware_domain (t) && (t != dom_io || !dev_is_pci (dev)) )return -EPERM ;if (t == s)return 0 ; - ret =arm_smmu_deassign_dev (s, dev); + ret =arm_smmu_deassign_dev (s, devfn, dev);if (ret)return ret;
Signature · 1 line
2.34.1
PATCH v8 6/8 xen/arm: Fix mapping for PCI bridge mmio region
In reply to Mykyta PoturaiFrom: Rahul Singh <[email protected]>
Current code skip the mapping for PCI bridge MMIO region to dom0 when pci_passthrough_enabled flag is set. Mapping should be skip when has_vpci(d) is enabled for the domain, as we need to skip the mapping only when VPCI handler are registered for ECAM.
---
This patch was originally picked up from [1]
v7->v8:
* no change
v6->v7:
* add Julien's A-b
v5->v6:
* drop unrelated change in xen/arch/arm/domain_build.c:handle_linux_pci_domain()
v4->v5:
* new patch
changes since picking up from [1]:
* rebase on top of "dynamic node programming using overlay dtbo" series
* replace !is_pci_passthrough_enabled() check with !IS_ENABLED(CONFIG_HAS_PCI) instead of removing
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
--- xen/arch/arm/device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/arm/device.c +1 -1
@@ -268,7 +268,7 @@ int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt, .d = d, .p2mt = p2mt, .skip_mapping = !own_device || - (is_pci_passthrough_enabled () && + (has_vpci (d) && (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)), .iomem_ranges = iomem_ranges, .irq_ranges = irq_ranges
Signature · 1 line
2.34.1
PATCH v8 3/8 iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
In reply to Mykyta PoturaiFrom: Stewart Hildebrand <[email protected]>
Handle phantom functions in iommu_add_dt_pci_sideband_ids(). Each phantom function will have a unique requestor ID (RID)/BDF. On ARM, we need to map/translate the RID/BDF to an AXI stream ID for each phantom function according to the pci-iommu device tree mapping [1]. The RID/BDF -> AXI stream ID mapping in DT could allow phantom devices (i.e. devices with phantom functions) to use different AXI stream IDs based on the (phantom) function.
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
---v7->v8:
* no change
v6->v7:
* no change
v5->v6:
* no change
v4->v5:
* no change
v3->v4:
* s/iommu_dt_pci_map_id/dt_map_id/
v2->v3:
* new patch title (was: iommu/arm: iommu_add_dt_pci_device phantom handling)
* rework loop to reduce duplication
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/
v1->v2:
* new patch
---
TODO: investigate Jan's comment [2]
[2] https://lore.kernel.org/xen-devel/[email protected]/
--- xen/drivers/passthrough/device_tree.c | 33 ++++++++++++++++----------- 1 file changed, 20 insertions(+), 13 deletions(-)
xen/drivers/passthrough/device_tree.c +20 -13
@@ -169,6 +169,7 @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)struct device *dev =pci_to_dev (pdev);const struct dt_device_node *np;int rc; +unsigned int devfn = pdev->devfn;if ( !iommu_enabled )return DT_NO_IOMMU ;
@@ -183,21 +184,27 @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)if ( !np )return -ENODEV ; -/* -* According to the Documentation/devicetree/bindings/pci/pci-iommu.txt -* from Linux. -*/ - rc =dt_map_id (np,PCI_BDF (pdev->bus, pdev->devfn),"iommu-map" , -"iommu-map-mask" , &iommu_spec.np, iommu_spec.args); -if ( rc ) -return (rc == -ENODEV ) ?DT_NO_IOMMU : rc; +do { +/* +* According to the Documentation/devicetree/bindings/pci/pci-iommu.txt +* from Linux. +*/ + rc =dt_map_id (np,PCI_BDF (pdev->bus, devfn),"iommu-map" , +"iommu-map-mask" , &iommu_spec.np, iommu_spec.args); +if ( rc ) +return (rc == -ENODEV ) ?DT_NO_IOMMU : rc; - rc =iommu_dt_xlate (dev, &iommu_spec, ops); -if ( rc <0 ) - { -iommu_fwspec_free (dev); -return -EINVAL ; + rc =iommu_dt_xlate (dev, &iommu_spec, ops); +if ( rc <0 ) + { +iommu_fwspec_free (dev); +return -EINVAL ; + } + + devfn += pdev->phantom_stride; } +while ( (devfn != pdev->devfn) && + (PCI_SLOT (devfn) ==PCI_SLOT (pdev->devfn)) );return rc; }
Signature · 1 line
2.34.1
PATCH v8 5/8 xen/arm: smmuv3: Add PCI devices support for SMMUv3
In reply to Mykyta PoturaiFrom: Rahul Singh <[email protected]>
Implement support for PCI devices in the SMMU driver. Trigger iommu-map parsing when new PCI device is added. Add checks to assign/deassign functions to ensure PCI devices are handled correctly. Implement basic quarantining.
All pci devices are automatically assigned to hardware domain if it exists to ensure it can probe them.
TODO:
Implement scratch page quarantining support.
---v7->v8:
* no change
v6->v7:
* address TODO: use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to support future dom0less use case when dom0 is using vPCI
* check if pdev->domain exists before assigning to it
* don't print ""
* change assign logic to remove reassign reimplementation
* explain pdev->devfn check
* make reassign check stricter and update comment
v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()
v4->v5:
* deassign from hwdom
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")
v3->v4:
* no change
v2->v3:
* rebase
* invoke iommu_add_pci_sideband_ids() from add_device hook
v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions (i.e. devfn != pdev->devfn)
downstream->v1:
* rebase
* move 2 replacements of s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/ from this commit to ("xen/arm: Move is_protected flag to struct device")
so as to not break ability to bisect
* adjust patch title (remove stray space)
* arm_smmu_(de)assign_dev: return error instead of crashing system
* remove arm_smmu_remove_device() stub
* update condition in arm_smmu_reassign_dev
* style fixup
(cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
--- xen/drivers/passthrough/arm/smmu-v3.c | 117 +++++++++++++++++++++++--- 1 file changed, 106 insertions(+), 11 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.c +106 -11
@@ -1467,14 +1467,35 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid) }/* Forward declaration */ static struct arm_smmu_device *arm_smmu_get_by_dev (const struct device *dev); +static int arm_smmu_assign_dev (struct domain *d,u8 devfn,struct device *dev, +u32 flag); +static int arm_smmu_deassign_dev (struct domain *d,uint8_t devfn, +struct device *dev);static int arm_smmu_add_device (u8 devfn,struct device *dev) {int i, ret;struct arm_smmu_device *smmu;struct arm_smmu_master *master; -struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); +struct iommu_fwspec *fwspec; + +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); +int ret; + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + + ret =iommu_add_pci_sideband_ids (pdev); +if ( ret <0 ) +iommu_fwspec_free (dev); + } +#endif + fwspec =dev_iommu_fwspec_get (dev);if (!fwspec)return -ENODEV ;
@@ -1519,17 +1540,38 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)*/ arm_smmu_enable_pasid (master); -if (dt_device_is_protected (dev_to_dt (dev))) { -dev_err (dev,"Already added to SMMUv3\n" ); -return -EEXIST ; - } +if ( !dev_is_pci (dev) ) + { +if (dt_device_is_protected (dev_to_dt (dev))) { +dev_err (dev,"Already added to SMMUv3\n" ); +return -EEXIST ; + } -/* Let Xen know that the master device is protected by an IOMMU. */ -dt_device_set_protected (dev_to_dt (dev)); +/* Let Xen know that the master device is protected by an IOMMU. */ +dt_device_set_protected (dev_to_dt (dev)); + }dev_info (dev,"Added master device (SMMUv3 %s StreamIds %u)\n" ,dev_name (fwspec->iommu_dev), fwspec->num_ids); +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* +* During PHYSDEVOP_pci_device_add, Xen does not assign the +* device, so we must do it here. +*/ +if ( pdev->domain ) + { + ret =arm_smmu_assign_dev (pdev->domain, devfn, dev,0 ); +if (ret) + goto err_free_master; + } + } +#endif +return 0 ; err_free_master:
@@ -2622,6 +2664,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,struct arm_smmu_domain *smmu_domain;struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv; +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + +ASSERT (pcidevs_locked ()); + +write_lock (&pdev->domain->pci_lock); +list_del (&pdev->domain_list); +write_unlock (&pdev->domain->pci_lock); + + pdev->domain = d; + +write_lock (&d->pci_lock); +list_add (&pdev->domain_list, &d->pdev_list); +write_unlock (&d->pci_lock); + +/* dom_io is used as a sentinel for quarantined devices */ +if ( d == dom_io ) + { +struct arm_smmu_master *master =dev_iommu_priv_get (dev); +if ( !iommu_quarantine ) +return 0 ; + +if ( master && master->domain ) +arm_smmu_deassign_dev (master->domain->d, devfn, dev); + +return 0 ; + } + } +#endif +spin_lock (&xen_domain->lock);/*
@@ -2655,7 +2733,7 @@ out:return ret; } -static int arm_smmu_deassign_dev (struct domain *d,struct device *dev) +static int arm_smmu_deassign_dev (struct domain *d,uint8_t devfn,struct device *dev) {struct iommu_domain *io_domain =arm_smmu_get_domain (d, dev);struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv;
@@ -2667,6 +2745,21 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)return -ESRCH ; } +#ifdef CONFIG_HAS_PCI +if (dev_is_pci (dev) ) + { +struct pci_dev *pdev =dev_to_pci (dev); + +/* Ignore calls for phantom functions */ +if ( devfn != pdev->devfn ) +return 0 ; + +/* dom_io is used as a sentinel for quarantined devices */ +if ( d == dom_io ) +return 0 ; + } +#endif +spin_lock (&xen_domain->lock);arm_smmu_detach_dev (master);
@@ -2685,14 +2778,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t, {int ret =0 ; -/* Don't allow remapping on other domain than hwdom */ -if ( t && !is_hardware_domain (t) ) +/* Don't allow remapping on other domain than hwdom +* or dom_io for PCI devices +*/ +if ( t && !is_hardware_domain (t) && (t != dom_io || !dev_is_pci (dev)) )return -EPERM ;if (t == s)return 0 ; - ret =arm_smmu_deassign_dev (s, dev); + ret =arm_smmu_deassign_dev (s, devfn, dev);if (ret)return ret;
Signature · 1 line
2.34.1
PATCH v8 7/8 xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
In reply to Mykyta PoturaiFrom: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes". We rely on dom0 to initialize the PCI controller and perform a PHYSDEVOP_pci_device_add call to add each device to SMMU.
Enable pci_init() for initializing Xen's internal PCI subsystem, and allow PHYSDEVOP_pci_device_add when pci-passthrough is disabled.
---
hmm. Since
dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ? Not sure if PPC/RISC-V will plan on using this check.
v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()
v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter
v5->v6:
* new patch - this effectively replaces
("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
--- xen/arch/arm/pci/pci.c | 5 +++-- xen/drivers/pci/physdev.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/pci/pci.c +3 -2
@@ -16,6 +16,7 @@#include <xen/device_tree.h> #include <xen/errno.h> #include <xen/init.h> +#include <xen/iommu.h> #include <xen/param.h> #include <xen/pci.h>
@@ -83,9 +84,9 @@ static int __init pci_init(void) {/* * Enable PCI passthrough when has been enabled explicitly -* (pci-passthrough=on). +* (pci-passthrough=on) or IOMMU is present and enabled. */ -if ( !pci_passthrough_enabled ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return 0 ;pci_segments_init ();
xen/drivers/pci/physdev.c +2 -2
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev_info pdev_info;nodeid_t node =NUMA_NO_NODE ; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)case PHYSDEVOP_pci_device_remove: {struct physdev_pci_device dev; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;
Signature · 1 line
2.34.1
PATCH v8 8/8 xen/arm: Map ITS doorbell register to IOMMU page tables
In reply to Mykyta PoturaiFrom: Rahul Singh <[email protected]>
When ITS is enabled and PCI devices that are behind an SMMU generate an MSI interrupt, SMMU fault will be observed as there is currently no mapping in p2m table for the ITS translation register (GITS_TRANSLATER).
A mapping is required in the iommu page tables so that the device can generate the MSI interrupt writing to the GITS_TRANSLATER register.
The GITS_TRANSLATER register is a 32-bit register, and there is nothing else in a page containing it, so map that page.
---
This patch was originally picked up from [1], and commit description loosely borrowed from [2].
Example SMMUv3 fault (qemu-system-aarch64 virt model), ITS base 0x8080000:
(XEN) SMMUv3: /smmuv3@9050000: event 0x10 received:
(XEN) SMMUv3: /smmuv3@9050000: 0x0000000800000010 (XEN) SMMUv3: /smmuv3@9050000: 0x0000008000000000 (XEN) SMMUv3: /smmuv3@9050000: 0x0000000008090040 (XEN) SMMUv3: /smmuv3@9050000: 0x0000000000000000
Example SMMUv2 fault (AMD/Xilinx Versal), ITS base 0xf9020000:
(XEN) smmu: /axi/smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0xf9030040, fsynr=0x12, cb=0
v7->v8:
* no changes
v6->v7:
* add tlb flush after mapping
* style: update formatting
* revert back to printk with XENLOG_G_ERR
v5->v6:
* switch to iommu_map() interface
* fix page_count argument
* style fixup
* use gprintk instead of printk
* add my Signed-off-by
* move to vgic_v3_its_init_virtual()
v4->v5:
* new patch
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
[2] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/6232a0d53377009bb7fbc3c3ab81d0153734be6b
--- xen/arch/arm/vgic-v3-its.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
xen/arch/arm/vgic-v3-its.c +20 -0
@@ -1478,6 +1478,26 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,register_mmio_handler (d, &vgic_its_mmio_handler, guest_addr,SZ_64K , its); +if (is_iommu_enabled (its->d) ) + { +mfn_t mfn =maddr_to_mfn (its->doorbell_address); +unsigned int flush_flags =0 ; +int ret =iommu_map (its->d,_dfn (mfn_x (mfn)), mfn,1 , IOMMUF_writable, + &flush_flags); + +if ( ret <0 ) + { +printk (XENLOG_G_ERR +"GICv3: Map ITS translation register for %pd failed.\n" , + its->d); +return ret; + } + + ret =iommu_iotlb_flush (its->d,_dfn (mfn_x (mfn)),1 , flush_flags); +if ( ret <0 ) +return ret; + } +/* Register the virtual ITS to be able to clean it up later. */ list_add_tail (&its->vits_list, &d->arch.vgic.vits_list);
Signature · 1 line
2.34.1
Quoted 9 lines · Mykyta Poturai
This series introduces SMMU handling for PCIe passthrough on ARM. These patches should be able to be upstreamed independently from the vPCI series [1]. See [2] for notes about test cases.
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.htmlv7-v8:
* no changes
And why exactly was this posted then as a new version?
Jan
Re: [PATCH v8 1/8] iommu/arm: Add iommu_dt_xlate()
In reply to Mykyta Poturaixen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d, · line 247
Quoted 13 lines · Mykyta Poturai
xen/include/xen/iommu.h +7
-0@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1
While an improvement, it still isn't clear whose "status code" this is. The #define is effectively hanging in the air, from all I can tell. And from it not being a normal error code it is pretty clear that it better would have only very narrow use.
Also can you please omit an interspersing blank line when the comment is specifically tied to a definition or declaration?
Jan
Quoted 11 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
This series introduces SMMU handling for PCIe passthrough on ARM. These patches should be able to be upstreamed independently from the vPCI series [1]. See [2] for notes about test cases.
v7-v8:
* no changesAnd why exactly was this posted then as a new version?
Jan
That is supposed to mean there are no changes in the series's structure, only in the individual patches. Sorry for possible confusion, I will try to write something clearer next time.
Signature · 1 line
Mykyta
Re: [PATCH v8 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
In reply to Mykyta PoturaiQuoted 15 lines · Mykyta Poturai
xen/drivers/passthrough/iommu.c +2
-0@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86 @@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; } +int iommu_add_pci_sideband_ids (struct pci_dev *pdev)
xen/drivers/passthrough/iommu.c int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, · line 758
Quoted 10 lines
xen/drivers/passthrough/iommu.c +10
-0+{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +}
This function has no caller, which violates a Misra rule iirc. Considering all information given here it's also unclear why it would gain a caller on x86 (at least as long as DT isn't used there).
Jan
Re: [PATCH v8 7/8] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
In reply to Mykyta PoturaiQuoted 4 lines · Mykyta Poturai
From: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes".
Why? It _is_ passing through, even if Dom0 is special.
xen/arch/arm/pci/pci.c static int __init pci_init(void) · line 90
Quoted 10 lines
xen/arch/arm/pci/pci.c +2
-2@@ -83,9 +84,9 @@ static int __init pci_init(void) {/* * Enable PCI passthrough when has been enabled explicitly -* (pci-passthrough=on). +* (pci-passthrough=on) or IOMMU is present and enabled. */ -if ( !pci_passthrough_enabled ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return 0 ;
I can't reasonably judge on this adjustment, but ...
Quoted 12 lines
xen/drivers/pci/physdev.c +1
-1@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev_info pdev_info;nodeid_t node =NUMA_NO_NODE ; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;
xen/drivers/pci/physdev.c ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 63
Quoted 8 lines
xen/drivers/pci/physdev.c +1
-1@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)case PHYSDEVOP_pci_device_remove: {struct physdev_pci_device dev; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;
... these two certainly look wrong to be made. If an Arm-specific adjustment is needed (and can be justified), a per-arch hook may need introducing.
Jan
Re: [PATCH v8 7/8] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
Quoted 7 lines · Jan Beulich
Quoted 4 lines · Mykyta Poturai
From: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes".
Why? It _is_ passing through, even if Dom0 is special.
Do you think it would be better to drop this completely and require pci-passthrough=yes for PCI to work in Dom0?
Quoted 29 lines
xen/arch/arm/pci/pci.c static int __init pci_init(void) · line 90
Quoted 11 lines
xen/arch/arm/pci/pci.c +2
-2@@ -83,9 +84,9 @@ static int __init pci_init(void) {/* * Enable PCI passthrough when has been enabled explicitly -* (pci-passthrough=on). +* (pci-passthrough=on) or IOMMU is present and enabled. */ -if ( !pci_passthrough_enabled ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return 0 ;I can't reasonably judge on this adjustment, but ...
Quoted 15 lines
xen/drivers/pci/physdev.c +1
-1@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev_info pdev_info;nodeid_t node =NUMA_NO_NODE ; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)case PHYSDEVOP_pci_device_remove: {struct physdev_pci_device dev;
Quoted 8 lines
xen/drivers/pci/physdev.c ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 25
Quoted 5 lines
xen/drivers/pci/physdev.c +1
-1-if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;... these two certainly look wrong to be made. If an Arm-specific adjustment is needed (and can be justified), a per-arch hook may need introducing.
This should not affect x86 in any way if I'm not missing something, as !is_pci_passthrough_enabled() will always be false. Or are you concerned about something else?
Quoted 2 lines
Jan
Signature · 1 line
Mykyta
Re: [PATCH v8 7/8] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
Quoted 11 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 4 lines · Mykyta Poturai
From: Stewart Hildebrand <[email protected]>
Enable the use of IOMMU + PCI in dom0 without having to specify "pci-passthrough=yes".
Why? It _is_ passing through, even if Dom0 is special.
Do you think it would be better to drop this completely and require pci-passthrough=yes for PCI to work in Dom0?
From an abstract perspective: Yes. I don't know any of the Arm background, though.
Quoted 32 lines
Quoted 32 lines
xen/arch/arm/pci/pci.c static int __init pci_init(void) · line 90
Quoted 10 lines
xen/arch/arm/pci/pci.c +2
-2@@ -83,9 +84,9 @@ static int __init pci_init(void) {/* * Enable PCI passthrough when has been enabled explicitly -* (pci-passthrough=on). +* (pci-passthrough=on) or IOMMU is present and enabled. */ -if ( !pci_passthrough_enabled ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return 0 ;I can't reasonably judge on this adjustment, but ...
Quoted 19 lines
xen/drivers/pci/physdev.c +2
-2@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)struct pci_dev_info pdev_info;nodeid_t node =NUMA_NO_NODE ; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ; ret = -EFAULT ;@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)case PHYSDEVOP_pci_device_remove: {struct physdev_pci_device dev; -if ( !is_pci_passthrough_enabled () ) +if ( !is_pci_passthrough_enabled () && !iommu_enabled )return -EOPNOTSUPP ;
Quoted 8 lines
Quoted 4 lines
xen/drivers/pci/physdev.c ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 63
Quoted 1 line
xen/drivers/pci/physdev.c +0
-0ret = -EFAULT ;... these two certainly look wrong to be made. If an Arm-specific adjustment is needed (and can be justified), a per-arch hook may need introducing.
This should not affect x86 in any way if I'm not missing something, as !is_pci_passthrough_enabled() will always be false. Or are you concerned about something else?
Indeed I am - the wrong look of it. Readers like me will have the immediate impression of there being something fishy here.
Jan
Re: [PATCH v8 1/8] iommu/arm: Add iommu_dt_xlate()
In reply to Jan BeulichQuoted 20 lines · Jan Beulich
xen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d, · line 247
Quoted 13 lines · Mykyta Poturai
xen/include/xen/iommu.h +7
-0@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1While an improvement, it still isn't clear whose "status code" this is. The #define is effectively hanging in the air, from all I can tell. And from it not being a normal error code it is pretty clear that it better would have only very narrow use.
Quoted 4 lines
Also can you please omit an interspersing blank line when the comment is specifically tied to a definition or declaration?
Jan
Hi Jan
What would you say about removing this status code entirely and returning something like -ENODEV instead, with adding special handling for this return to the callers where needed?
Signature · 1 line
Mykyta
Re: [PATCH v8 1/8] iommu/arm: Add iommu_dt_xlate()
Quoted 21 lines · Mykyta Poturai
Quoted 20 lines · Jan Beulich
xen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d, · line 247
Quoted 13 lines · Mykyta Poturai
xen/include/xen/iommu.h +7
-0@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1While an improvement, it still isn't clear whose "status code" this is. The #define is effectively hanging in the air, from all I can tell. And from it not being a normal error code it is pretty clear that it better would have only very narrow use.
Quoted 6 lines
Quoted 2 lines
Also can you please omit an interspersing blank line when the comment is specifically tied to a definition or declaration?
What would you say about removing this status code entirely and returning something like
-ENODEVinstead, with adding special handling for this return to the callers where needed?
I'd be okay with that; Arm folks also need to be, though.
Jan
Re: [PATCH v8 1/8] iommu/arm: Add iommu_dt_xlate()
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Mykyta Poturai
Quoted 20 lines · Jan Beulich
xen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d, · line 247
Quoted 13 lines · Mykyta Poturai
xen/include/xen/iommu.h +7
-0@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1While an improvement, it still isn't clear whose "status code" this is. The #define is effectively hanging in the air, from all I can tell. And from it not being a normal error code it is pretty clear that it better would have only very narrow use.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Also can you please omit an interspersing blank line when the comment is specifically tied to a definition or declaration?
What would you say about removing this status code entirely and returning something like
-ENODEVinstead, with adding special handling for this return to the callers where needed?I'd be okay with that; Arm folks also need to be, though.
Oh, and: Of course it then needs to be pretty clear / obvious that -ENODEV cannot come into play for other reasons / from other origins.
Jan
Re: [PATCH v8 1/8] iommu/arm: Add iommu_dt_xlate()
Hi,
Quoted 26 lines · Jan Beulich
Quoted 25 lines · Jan Beulich
Quoted 24 lines · Mykyta Poturai
Quoted 22 lines · Jan Beulich
xen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d, · line 247
Quoted 13 lines · Mykyta Poturai
xen/include/xen/iommu.h +7
-0@@ -238,6 +238,14 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,*/ int iommu_remove_dt_device (struct dt_device_node *np); +/* +* Status code indicating that DT device cannot be added to the IOMMU +* or removed from it because the IOMMU is disabled or the device is not +* connected to it. +*/ + +#define DT_NO_IOMMU 1While an improvement, it still isn't clear whose "status code" this is. The #define is effectively hanging in the air, from all I can tell. And from it not being a normal error code it is pretty clear that it better would have only very narrow use.
Also can you please omit an interspersing blank line when the comment is specifically tied to a definition or declaration?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What would you say about removing this status code entirely and returning something like
-ENODEVinstead, with adding special handling for this return to the callers where needed?I'd be okay with that; Arm folks also need to be, though.
Oh, and: Of course it then needs to be pretty clear / obvious that
-ENODEVcannot come into play for other reasons / from other origins.
It would be difficult to guarantee that all the callbacks will never return -ENODEV. So I am quite reluctant to use -ENODEV to indicate the IOMMU is not available or the device is not behind an IOMMU.
Anyway, I can't fully remember the previous discussion. Can someone remind me what we are trying to solve with introducing DT_NO_IOMMU? The meaning of the value is already properly documented on each function that can return the value:
* >0 : device doesn't need to be protected by an IOMMU
* (IOMMU is not enabled/present or device is not connected to it).
It seems to me it would be easier to open-code the value because there is no question of how the define is going to be used.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v8 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
In reply to Jan BeulichQuoted 31 lines · Jan Beulich
Quoted 15 lines · Mykyta Poturai
xen/drivers/passthrough/iommu.c +2
-0@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86 @@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; } +int iommu_add_pci_sideband_ids (struct pci_dev *pdev)xen/drivers/passthrough/iommu.c int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, · line 758
Quoted 10 lines
xen/drivers/passthrough/iommu.c +10
-0+{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +}This function has no caller, which violates a Misra rule iirc. Considering all information given here it's also unclear why it would gain a caller on x86 (at least as long as DT isn't used there).
Quoted 1 line
Jan
Would it be ok to wrap it with CONFIG_ARM? I am not quite sure how relevant this mapping functionality is to X86 iommus, although Linux has similar implementations for ACPI.
Alternatively, we can remove this abstraction for now, to call iommu_add_dt_pci_sideband_ids from Arm directly and only introduce it back when at least some ACPI implementation is done.
Also, just want to mention the issue that forced me to move this from the header in the first place in case it is not known. There is a conflict in fixed width integers definitions between actypes.h and efibind.h and it was revealed when including acpi.h into iommu.h. I initially tried to fix the source of this conflict, but I don't know enough about ACPI and EFI quirks to confidently do it.
In file included from ./include/acpi/acpi.h:57,
from ./include/xen/acpi.h:57,
from ./include/xen/iommu.h:28,
from ./include/xen/sched.h:12,
from ./arch/x86/include/asm/paging.h:17,
from ./arch/x86/include/asm/guest_access.h:11,
from ./include/xen/guest_access.h:10,
from arch/x86/efi/runtime.c:5:./include/acpi/actypes.h:130:35: error: conflicting types for ‘UINT64’; have ‘long long unsigned int’
130 | typedef COMPILER_DEPENDENT_UINT64 UINT64;
| ^~~~~~
In file included from ./arch/x86/include/asm/efibind.h:2,
from ./common/efi/efi.h:1,
from arch/x86/efi/runtime.c:1:
./arch/x86/include/asm/x86_64/efibind.h:89:20: note: previous
declaration of ‘UINT64’ with type ‘UINT64’ {aka ‘long unsigned int’}
89 | typedef uint64_t UINT64;Signature · 1 line
Mykyta
Re: [PATCH v8 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
Quoted 32 lines · Mykyta Poturai
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Mykyta Poturai
xen/drivers/passthrough/iommu.c +1
-0@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86 @@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; }Quoted 15 lines
xen/drivers/passthrough/iommu.c int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, · line 758
Quoted 11 lines
xen/drivers/passthrough/iommu.c +11
-0+int iommu_add_pci_sideband_ids (struct pci_dev *pdev) +{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +}This function has no caller, which violates a Misra rule iirc. Considering all information given here it's also unclear why it would gain a caller on x86 (at least as long as DT isn't used there).
Quoted 3 lines
Would it be ok to wrap it with
CONFIG_ARM? I am not quite sure how relevant this mapping functionality is to X86 iommus, although Linux has similar implementations for ACPI.
Besides it being unclear to me whether the function is really Arm-specific (what about RISC-V or PPC), I also don't see how that would address the Misra concern. (If the function was truly Arm-specific, it would better move into an Arm-specific source file.)
Quoted 3 lines
Alternatively, we can remove this abstraction for now, to call
iommu_add_dt_pci_sideband_idsfrom Arm directly and only introduce it back when at least some ACPI implementation is done.
I'd leave that to Arm folks to judge.
Quoted 20 lines
Also, just want to mention the issue that forced me to move this from the header in the first place in case it is not known. There is a conflict in fixed width integers definitions between
actypes.handefibind.hand it was revealed when includingacpi.hintoiommu.h. I initially tried to fix the source of this conflict, but I don't know enough about ACPI and EFI quirks to confidently do it.In file included from
./include/acpi/acpi.h:57,from ./include/xen/acpi.h:57, from ./include/xen/iommu.h:28, from ./include/xen/sched.h:12, from ./arch/x86/include/asm/paging.h:17, from ./arch/x86/include/asm/guest_access.h:11, from ./include/xen/guest_access.h:10, from arch/x86/efi/runtime.c:5:
./include/acpi/actypes.h:130:35: error: conflicting types for ‘UINT64’; have ‘long long unsigned int’
130 | typedefCOMPILER_DEPENDENT_UINT64UINT64;
| ^~~~~~
In file included from./arch/x86/include/asm/efibind.h:2,
Quoted 5 lines
from ./common/efi/efi.h:1,
from arch/x86/efi/runtime.c:1:
./arch/x86/include/asm/x86_64/efibind.h:89:20: note: previous
declaration of ‘UINT64’ with type ‘UINT64’ {aka ‘long unsigned int’}
89 | typedef uint64_t UINT64;Yeah, sadly ACPI and EFI headers (both imported from different origins) aren't overly compatible with one another.
Jan
Re: [PATCH v8 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
Quoted 74 lines · Jan Beulich
Quoted 32 lines · Mykyta Poturai
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Mykyta Poturai
xen/drivers/passthrough/iommu.c +1
-0@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86 @@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; }Quoted 15 lines
xen/drivers/passthrough/iommu.c int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, · line 758
Quoted 11 lines
xen/drivers/passthrough/iommu.c +11
-0+int iommu_add_pci_sideband_ids (struct pci_dev *pdev) +{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +}This function has no caller, which violates a Misra rule iirc. Considering all information given here it's also unclear why it would gain a caller on x86 (at least as long as DT isn't used there).
Quoted 3 lines
Would it be ok to wrap it with
CONFIG_ARM? I am not quite sure how relevant this mapping functionality is to X86 iommus, although Linux has similar implementations for ACPI.Besides it being unclear to me whether the function is really Arm-specific (what about RISC-V or PPC), I also don't see how that would address the Misra concern. (If the function was truly Arm-specific, it would better move into an Arm-specific source file.)
Quoted 3 lines
Alternatively, we can remove this abstraction for now, to call
iommu_add_dt_pci_sideband_idsfrom Arm directly and only introduce it back when at least some ACPI implementation is done.I'd leave that to Arm folks to judge.
Quoted 17 lines
Also, just want to mention the issue that forced me to move this from the header in the first place in case it is not known. There is a conflict in fixed width integers definitions between
actypes.handefibind.hand it was revealed when includingacpi.hintoiommu.h. I initially tried to fix the source of this conflict, but I don't know enough about ACPI and EFI quirks to confidently do it.In file included from
./include/acpi/acpi.h:57,from ./include/xen/acpi.h:57, from ./include/xen/iommu.h:28, from ./include/xen/sched.h:12, from ./arch/x86/include/asm/paging.h:17, from ./arch/x86/include/asm/guest_access.h:11, from ./include/xen/guest_access.h:10, from arch/x86/efi/runtime.c:5:
./include/acpi/actypes.h:130:35: error: conflicting types for ‘UINT64’; have ‘long long unsigned int’Quoted 8 lines
130 | typedef COMPILER_DEPENDENT_UINT64 UINT64; | ^~~~~~ In file included from ./arch/x86/include/asm/efibind.h:2, from ./common/efi/efi.h:1, from arch/x86/efi/runtime.c:1: ./arch/x86/include/asm/x86_64/efibind.h:89:20: note: previous declaration of ‘UINT64’ with type ‘UINT64’ {aka ‘long unsigned int’} 89 | typedef uint64_t UINT64;
Quoted 4 lines
Yeah, sadly ACPI and EFI headers (both imported from different origins) aren't overly compatible with one another.
Jan
Hi everyone
I searched for an appropriate place to put this function but I am a little stuck here:
- Can't be put in the header as static inline because of header conflict.
- Putting it in Arm only files or defines feels wrong because it is not in fact Arm-specific.
- In iommu.c it can become dead code on some architectures.
- Removing it and calling iommu_add_dt_pci_sideband_ids goes against the effort to make DT and ACPI able to co-exist.
Could you please suggest a good way forward from here? I see two possible ones:
1. Fix the header conflict and return it to the header as static inline
- best solution in my opinion
2. Move to arm files it gains callers on other architectures.
If we are going for the first approach maybe you can provide some pointers on how to better deal with this header conflict? Adding ifdef guards to the definitions? Renaming the types in one of them to something like EFI_UINT64 or ACPI_UINT64? Would a successful boot on the ACPI/EFI system be enough to confirm that I didn't break anything or will it require some more specific tests?
Signature · 1 line
Mykyta
Re: [PATCH v8 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
Quoted 92 lines · Mykyta Poturai
Quoted 74 lines · Jan Beulich
Quoted 32 lines · Mykyta Poturai
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Mykyta Poturai
xen/drivers/passthrough/iommu.c +1
-0@@ -20,6 +20,7 @@#include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> +#include <xen/acpi.h> #include <xsm/xsm.h> #ifdef CONFIG_X86 @@ -744,6 +745,18 @@ int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func,return 0 ; }Quoted 15 lines
xen/drivers/passthrough/iommu.c int __init iommu_get_extra_reserved_device_memory(iommu_grdm_t *func, · line 758
Quoted 11 lines
xen/drivers/passthrough/iommu.c +11
-0+int iommu_add_pci_sideband_ids (struct pci_dev *pdev) +{ +int ret = -EOPNOTSUPP ; + +#ifdef CONFIG_HAS_PCI +if ( acpi_disabled ) + ret =iommu_add_dt_pci_sideband_ids (pdev); +#endif + +return ret; +}This function has no caller, which violates a Misra rule iirc. Considering all information given here it's also unclear why it would gain a caller on x86 (at least as long as DT isn't used there).
Quoted 3 lines
Would it be ok to wrap it with
CONFIG_ARM? I am not quite sure how relevant this mapping functionality is to X86 iommus, although Linux has similar implementations for ACPI.Besides it being unclear to me whether the function is really Arm-specific (what about RISC-V or PPC), I also don't see how that would address the Misra concern. (If the function was truly Arm-specific, it would better move into an Arm-specific source file.)
Quoted 3 lines
Alternatively, we can remove this abstraction for now, to call
iommu_add_dt_pci_sideband_idsfrom Arm directly and only introduce it back when at least some ACPI implementation is done.I'd leave that to Arm folks to judge.
Quoted 17 lines
Also, just want to mention the issue that forced me to move this from the header in the first place in case it is not known. There is a conflict in fixed width integers definitions between
actypes.handefibind.hand it was revealed when includingacpi.hintoiommu.h. I initially tried to fix the source of this conflict, but I don't know enough about ACPI and EFI quirks to confidently do it.In file included from
./include/acpi/acpi.h:57,from ./include/xen/acpi.h:57, from ./include/xen/iommu.h:28, from ./include/xen/sched.h:12, from ./arch/x86/include/asm/paging.h:17, from ./arch/x86/include/asm/guest_access.h:11, from ./include/xen/guest_access.h:10, from arch/x86/efi/runtime.c:5:
./include/acpi/actypes.h:130:35: error: conflicting types for ‘UINT64’; have ‘long long unsigned int’Quoted 8 lines
130 | typedef COMPILER_DEPENDENT_UINT64 UINT64; | ^~~~~~ In file included from ./arch/x86/include/asm/efibind.h:2, from ./common/efi/efi.h:1, from arch/x86/efi/runtime.c:1: ./arch/x86/include/asm/x86_64/efibind.h:89:20: note: previous declaration of ‘UINT64’ with type ‘UINT64’ {aka ‘long unsigned int’} 89 | typedef uint64_t UINT64;Quoted 4 lines
Yeah, sadly ACPI and EFI headers (both imported from different origins) aren't overly compatible with one another.
Jan
Hi everyone
I searched for an appropriate place to put this function but I am a little stuck here:
- Can't be put in the header as static inline because of header conflict.
- Putting it in Arm only files or defines feels wrong because it is not in fact Arm-specific.
- Iniommu.cit can become dead code on some architectures.
- Removing it and callingiommu_add_dt_pci_sideband_idsgoes against the effort to make DT and ACPI able to co-exist.
Quoted 5 lines
Could you please suggest a good way forward from here? I see two possible ones:
1. Fix the header conflict and return it to the header as static inline
- best solution in my opinion
I'm very likely to say "no" to anything trying to go this route. The headers in question want leaving alone as much as possible, to stay close to their originals. The only acceptable approach there would be to propose adjusting said originals.
Quoted 1 line
2. Move to arm files it gains callers on other architectures.
I fear I don't understand this one.
Jan
Quoted 7 lines
If we are going for the first approach maybe you can provide some pointers on how to better deal with this header conflict? Adding ifdef guards to the definitions? Renaming the types in one of them to something like
EFI_UINT64orACPI_UINT64? Would a successful boot on the ACPI/EFI system be enough to confirm that I didn't break anything or will it require some more specific tests?