Skip to contentSkip to conversation

dom0less pci passthrough support on Arm part 1

PATCH In review Mykyta Poturai 3 participants 11 messages Aug 20, 2025 – Aug 22, 2025

lore
Share & tools
lore
#1 lore raw

PATCH v2 1/3 arm/pci: Add pci-scan boot argument

In reply to Mykyta Poturai

From: Edward Pickup <[email protected]>

This patch adds a Xen boot arguments that, if enabled, causes a call to existing code to scan pci devices enumerated by the firmware.

This will be needed ahead of dom0less support for pci passthrough on arm.

---
(cherry picked from commit bce463e1588a45e1bfdf59fc0d5f88b16604e439 from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* remove is_pci_scan_enabled wrapper
* make pci_scan_enabled ro_after_init
* drop debug prints
* drop Edward's SOB

changes since cherry-pick:
* s/always_inline/inline/
* replace additional kconfig option with config DEBUG

---
 docs/misc/xen-command-line.pandoc  |  7 +++++++
 xen/arch/arm/include/asm/pci.h     |  3 +++
 xen/arch/arm/pci/pci-host-common.c |  1 +
 xen/arch/arm/pci/pci.c             | 24 ++++++++++++++++++++++--
 4 files changed, 33 insertions(+), 2 deletions(-)
docs/misc/xen-command-line.pandoc +7 -0
index a75b6c9301..762a1a4f5f 100644
@@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present).
 
 Flag to enable or disable support for PCI passthrough
 
+### pci-scan (arm)
+> `= <boolean>`
+
+> Default: `false`
+
+Flag to enable or disable Xen PCI scan at boot.
+
 ### pcid (x86)
 > `= <boolean> | xpti=<bool>`
 
xen/arch/arm/include/asm/pci.h +3 -0
index 08ffcd4438..7289f7688b 100644
@@ -23,6 +23,7 @@
 #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
 
 extern bool pci_passthrough_enabled;
+extern bool pci_scan_enabled;
 
 struct rangeset;
 
@@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);
 
 #else   /*!CONFIG_HAS_PCI*/
 
+#define pci_scan_enabled false
+
 struct pci_dev;
 
 static inline void arch_pci_init_pdev(struct pci_dev *pdev) {}
xen/arch/arm/pci/pci-host-common.c +1 -0
index 487c545f3a..d3481b05eb 100644
@@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev,
     }
 
     pci_add_host_bridge(bridge);
+    pci_add_segment(bridge->segment);
 
     return bridge;
 
xen/arch/arm/pci/pci.c +22 -2
index beb1f971fa..1b34e17517 100644
@@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
 
+/* By default pci scan is disabled. */
+bool __ro_after_init pci_scan_enabled;
+boolean_param("pci-scan", pci_scan_enabled);
+
 static int __init pci_init(void)
 {
+    int ret;
+
     /*
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
@@ -104,9 +110,23 @@ static int __init pci_init(void)
         panic("Could not initialize PCI segment 0\n");
 
     if ( acpi_disabled )
-        return dt_pci_init();
+        ret = dt_pci_init();
     else
-        return acpi_pci_init();
+        ret = acpi_pci_init();
+
+    if ( ret < 0 )
+        return ret;
+
+    if ( pci_scan_enabled )
+    {
+        ret = scan_pci_devices();
+
+        if ( ret < 0 )
+            return ret;
+
+    }
+
+    return 0;
 }
 __initcall(pci_init);
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH v2 2/3 xen/pci: modify pci_add_device to handle device add by Xen

In reply to Mykyta Poturai

From: Luca Fancellu <[email protected]>

Currently pci_add_device is called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed that hardware_domain is the owner.

On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.

Modify pci_add_device to have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).

---
(cherry picked from commit f0c85d9043f7c9402e85b73361c8a13c683428ca from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* remove dom_io check
* fixup pci_add_device parameters
* use current->domain instead of hardware_domain

changes since cherry-pick:
* s/hardware_domain/d/ in some write_unlock calls in xen/drivers/passthrough/pci.c

---
 xen/arch/x86/physdev.c        |  9 ++++-----
 xen/drivers/passthrough/pci.c | 29 +++++++++++++++++++----------
 xen/drivers/pci/physdev.c     |  3 ++-
 xen/include/xen/pci.h         |  5 +++--
 4 files changed, 28 insertions(+), 18 deletions(-)
xen/arch/x86/physdev.c +4 -5
index 4dfa1c0191..04d179e81b 100644
@@ -446,8 +446,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( copy_from_guest(&manage_pci, arg, 1) != 0 )
             break;
 
-        ret = pci_add_device(0, manage_pci.bus, manage_pci.devfn,
-                             NULL, NUMA_NO_NODE);
+        ret = pci_add_device(0, manage_pci.bus, manage_pci.devfn, NULL,
+                             NUMA_NO_NODE, current->domain);
         break;
     }
 
@@ -477,9 +477,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         pdev_info.is_virtfn = manage_pci_ext.is_virtfn;
         pdev_info.physfn.bus = manage_pci_ext.physfn.bus;
         pdev_info.physfn.devfn = manage_pci_ext.physfn.devfn;
-        ret = pci_add_device(0, manage_pci_ext.bus,
-                             manage_pci_ext.devfn,
-                             &pdev_info, NUMA_NO_NODE);
+        ret = pci_add_device(0, manage_pci_ext.bus, manage_pci_ext.devfn,
+                             &pdev_info, NUMA_NO_NODE, current->domain);
         break;
     }
 
xen/drivers/passthrough/pci.c +19 -10
index 3edcfa8a04..09b424d1b3 100644
@@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,
     return is64bits ? 2 : 1;
 }
 
-int pci_add_device(u16 seg, u8 bus, u8 devfn,
-                   const struct pci_dev_info *info, nodeid_t node)
+int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
+                   const struct pci_dev_info *info, nodeid_t node,
+                   struct domain *d)
 {
     struct pci_seg *pseg;
     struct pci_dev *pdev;
@@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     const char *type;
     int ret;
 
+    if ( !d )
+        return -EINVAL;
+
     if ( !info )
         type = "device";
     else if ( info->is_virtfn )
@@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     ret = 0;
     if ( !pdev->domain )
     {
-        pdev->domain = hardware_domain;
-        write_lock(&hardware_domain->pci_lock);
-        list_add(&pdev->domain_list, &hardware_domain->pdev_list);
+        pdev->domain = d;
+        write_lock(&d->pci_lock);
+        list_add(&pdev->domain_list, &pdev->domain->pdev_list);
 
         /*
          * For devices not discovered by Xen during boot, add vPCI handlers
@@ -779,25 +783,30 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
         if ( ret )
         {
             list_del(&pdev->domain_list);
-            write_unlock(&hardware_domain->pci_lock);
+            write_unlock(&d->pci_lock);
             pdev->domain = NULL;
             printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
             goto out;
         }
-        write_unlock(&hardware_domain->pci_lock);
+        write_unlock(&d->pci_lock);
         ret = iommu_add_device(pdev);
         if ( ret )
         {
-            write_lock(&hardware_domain->pci_lock);
+            write_lock(&d->pci_lock);
             vpci_deassign_device(pdev);
             list_del(&pdev->domain_list);
-            write_unlock(&hardware_domain->pci_lock);
+            write_unlock(&d->pci_lock);
             pdev->domain = NULL;
             goto out;
         }
     }
-    else
+    else if ( pdev->domain == d )
         iommu_enable_device(pdev);
+    else
+    {
+        ret = -EINVAL;
+        goto out;
+    }
 
     pci_enable_acs(pdev);
 
xen/drivers/pci/physdev.c +2 -1
index d46501b884..cd3a36903b 100644
@@ -50,7 +50,8 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         }
 #endif
 
-        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
+        ret = pci_add_device(add.seg, add.bus, add.devfn,
+                             &pdev_info, node, current->domain);
         break;
     }
 
xen/include/xen/pci.h +3 -2
index 130c2a8c1a..89f3281b7c 100644
@@ -226,8 +226,9 @@ void setup_hwdom_pci_devices(struct domain *d,
 int pci_release_devices(struct domain *d);
 int pci_add_segment(u16 seg);
 const unsigned long *pci_get_ro_map(u16 seg);
-int pci_add_device(u16 seg, u8 bus, u8 devfn,
-                   const struct pci_dev_info *info, nodeid_t node);
+int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
+                   const struct pci_dev_info *info, nodeid_t node,
+                   struct domain *d);
 int pci_remove_device(u16 seg, u8 bus, u8 devfn);
 int pci_ro_device(int seg, int bus, int devfn);
 int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn);
Signature · 1 line
2.34.1
#0 lore raw

v2 cover dom0less pci passthrough support on Arm part 1

This series adds basic PCI device enumeration in Xen. This will allow us to not rely on Dom0 enumeration for supported controllers, which will enable PCI passthrough for dom0less setups.

Enumeration is disabled by default and can be enabled with "pci-scan" cmdline option. Discovered devices are added to Xen and assigned to dom_io by default.

Edward Pickup (1):
arm/pci: Add pci-scan boot argument

Luca Fancellu (2):
xen/pci: modify pci_add_device to handle device add by Xen xen/pci: add discovered PCI device at boot

 docs/misc/xen-command-line.pandoc           |  7 ++
 xen/arch/arm/include/asm/pci.h              |  3 +
 xen/arch/arm/pci/pci-host-common.c          |  1 +
 xen/arch/arm/pci/pci.c                      | 25 +++++++-
 xen/arch/x86/physdev.c                      |  9 ++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  2 +-
 xen/drivers/passthrough/pci.c               | 71 ++++++++++++++-------
 xen/drivers/passthrough/vtd/iommu.c         |  2 +-
 xen/drivers/pci/physdev.c                   |  3 +-
 xen/include/xen/pci.h                       | 10 +--
 10 files changed, 94 insertions(+), 39 deletions(-)
Signature · 1 line
2.34.1
#6 lore raw

PATCH v2 3/3 xen/pci: add discovered PCI device at boot

From: Luca Fancellu <[email protected]>

In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.

In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.

---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io

---
 xen/arch/arm/pci/pci.c                      |  1 +
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  2 +-
 xen/drivers/passthrough/pci.c               | 42 ++++++++++++++-------
 xen/drivers/passthrough/vtd/iommu.c         |  2 +-
 xen/include/xen/pci.h                       |  5 +--
 5 files changed, 33 insertions(+), 19 deletions(-)
xen/arch/arm/pci/pci.c +1 -0
index 1b34e17517..909fbdd694 100644
@@ -124,6 +124,7 @@ static int __init pci_init(void)
         if ( ret < 0 )
             return ret;
 
+        setup_pci_devices(dom_io, NULL);
     }
 
     return 0;
xen/drivers/passthrough/amd/pci_amd_iommu.c +1 -1
index 3a14770855..f3a83a0ab7 100644
@@ -401,7 +401,7 @@ static void __hwdom_init cf_check amd_iommu_hwdom_init(struct domain *d)
 
     /* Make sure workarounds are applied (if needed) before adding devices. */
     arch_iommu_hwdom_init(d);
-    setup_hwdom_pci_devices(d, amd_iommu_add_device);
+    setup_pci_devices(d, amd_iommu_add_device);
 }
 
 static void amd_iommu_disable_domain_device(const struct domain *domain,
xen/drivers/passthrough/pci.c +28 -14

103 lines not on this page · raw message

xen/drivers/passthrough/vtd/iommu.c +1 -1
index c55f02c97e..1096c16327 100644
@@ -1451,7 +1451,7 @@ static void __hwdom_init cf_check intel_iommu_hwdom_init(struct domain *d)
 {
     struct acpi_drhd_unit *drhd;
 
-    setup_hwdom_pci_devices(d, setup_hwdom_device);
+    setup_pci_devices(d, setup_hwdom_device);
     setup_hwdom_rmrr(d);
     /* Make sure workarounds are applied before enabling the IOMMU(s). */
     arch_iommu_hwdom_init(d);
xen/include/xen/pci.h +2 -3
index 89f3281b7c..61f69a8a1b 100644
@@ -220,9 +220,8 @@ int scan_pci_devices(void);
 enum pdev_type pdev_type(u16 seg, u8 bus, u8 devfn);
 int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);
 
-void setup_hwdom_pci_devices(struct domain *d,
-                             int (*handler)(uint8_t devfn,
-                                            struct pci_dev *pdev));
+void setup_pci_devices(struct domain *d, int (*handler)(uint8_t devfn,
+                                                        struct pci_dev *pdev));
 int pci_release_devices(struct domain *d);
 int pci_add_segment(u16 seg);
 const unsigned long *pci_get_ro_map(u16 seg);
Signature · 1 line
2.34.1
Acked-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] arm/pci: Add pci-scan boot argument

In reply to Mykyta Poturai
Quoted 10 lines · Mykyta Poturai

From: Edward Pickup <[email protected]>

This patch adds a Xen boot arguments that, if enabled, causes a call to existing code to scan pci devices enumerated by the firmware.

This will be needed ahead of dom0less support for pci passthrough on arm.

Quoted 122 lines

---
(cherry picked from commit bce463e1588a45e1bfdf59fc0d5f88b16604e439 from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* remove is_pci_scan_enabled wrapper
* make pci_scan_enabled ro_after_init
* drop debug prints
* drop Edward's SOB

changes since cherry-pick:
* s/always_inline/inline/
* replace additional kconfig option with config DEBUG

---
 docs/misc/xen-command-line.pandoc  |  7 +++++++
 xen/arch/arm/include/asm/pci.h     |  3 +++
 xen/arch/arm/pci/pci-host-common.c |  1 +
 xen/arch/arm/pci/pci.c             | 24 ++++++++++++++++++++++--
 4 files changed, 33 insertions(+), 2 deletions(-)
docs/misc/xen-command-line.pandoc +7 -0
index a75b6c9301..762a1a4f5f 100644
@@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present).
 
 Flag to enable or disable support for PCI passthrough
 
+### pci-scan (arm)
+> `= <boolean>`
+
+> Default: `false`
+
+Flag to enable or disable Xen PCI scan at boot.
+
 ### pcid (x86)
 > `= <boolean> | xpti=<bool>`
 
xen/arch/arm/include/asm/pci.h +3 -0
index 08ffcd4438..7289f7688b 100644
@@ -23,6 +23,7 @@
 #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
 
 extern bool pci_passthrough_enabled;
+extern bool pci_scan_enabled;
 
 struct rangeset;
 
@@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);
 
 #else   /*!CONFIG_HAS_PCI*/
 
+#define pci_scan_enabled false
+
 struct pci_dev;
 
 static inline void arch_pci_init_pdev(struct pci_dev *pdev) {}
xen/arch/arm/pci/pci-host-common.c +1 -0
index 487c545f3a..d3481b05eb 100644
@@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev,
     }
 
     pci_add_host_bridge(bridge);
+    pci_add_segment(bridge->segment);
 
     return bridge;
 
xen/arch/arm/pci/pci.c +22 -2
index beb1f971fa..1b34e17517 100644
@@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
 
+/* By default pci scan is disabled. */
+bool __ro_after_init pci_scan_enabled;
+boolean_param("pci-scan", pci_scan_enabled);
+
 static int __init pci_init(void)
 {
+    int ret;
+
     /*
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
@@ -104,9 +110,23 @@ static int __init pci_init(void)
         panic("Could not initialize PCI segment 0\n");
 
     if ( acpi_disabled )
-        return dt_pci_init();
+        ret = dt_pci_init();
     else
-        return acpi_pci_init();
+        ret = acpi_pci_init();
+
+    if ( ret < 0 )
+        return ret;
+
+    if ( pci_scan_enabled )
+    {
+        ret = scan_pci_devices();
+
+        if ( ret < 0 )
+            return ret;
+
+    }
+
+    return 0;
 }
 __initcall(pci_init);
 
Signature · 1 line
2.34.1
Reviewed-by on v2 2/3
#4 lore raw

Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen

In reply to Mykyta Poturai
Quoted 11 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

Currently pci_add_device is called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed that hardware_domain is the owner.

On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.

Quoted 8 lines

Modify pci_add_device to have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).

#5 lore raw

Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen

In reply to Mykyta Poturai
Quoted 24 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

Currently pci_add_device is called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed that hardware_domain is the owner.

On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.

Modify pci_add_device to have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).

---
(cherry picked from commit f0c85d9043f7c9402e85b73361c8a13c683428ca from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

Quoted 4 lines

v1->v2:
* remove dom_io check
* fixup pci_add_device parameters
* use current->domain instead of hardware_domain

What I'm missing (as per my v1 comment) is discussion of the hardware_domain -> current->domain change, including the XSM aspect. Because of the XSM aspect, please also Cc the XSM maintainer going forward (I'm adding him here as well).

Quoted 15 lines
xen/drivers/passthrough/pci.c +3 -2
@@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,
     return is64bits ? 2 : 1;
 }
 
-int pci_add_device(u16 seg, u8 bus, u8 devfn,
-                   const struct pci_dev_info *info, nodeid_t node)
+int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
+                   const struct pci_dev_info *info, nodeid_t node,
+                   struct domain *d)
 {
     struct pci_seg *pseg;
     struct pci_dev *pdev;

xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 668

Quoted 5 lines
xen/drivers/passthrough/pci.c +2 -0
@@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     const char *type;
     int ret;
 
+    if ( !d )
+        return -EINVAL;

This should't be needed. Very remotely ASSERT(d) could be added here, but we don't normally do so elsewhere.

xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 776

Quoted 10 lines
xen/drivers/passthrough/pci.c +3 -3
@@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     ret = 0;
     if ( !pdev->domain )
     {
-        pdev->domain = hardware_domain;
-        write_lock(&hardware_domain->pci_lock);
-        list_add(&pdev->domain_list, &hardware_domain->pdev_list);
+        pdev->domain = d;
+        write_lock(&d->pci_lock);
+        list_add(&pdev->domain_list, &pdev->domain->pdev_list);

Why pdev->domain instead of the shorter and more efficient d?

Jan

#7 lore raw

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

In reply to Mykyta Poturai
Quoted 19 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.

In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.

Quoted 8 lines

---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io

What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the dom_io intermediary is necessary in the first place.

xen/drivers/passthrough/pci.c line 38

Quoted 7 lines
xen/drivers/passthrough/pci.c +1 -0
@@ -35,6 +35,7 @@
 #include <xen/msi.h>
 #include <xsm/xsm.h>
 #include "ats.h"
+#include "xen/dom0less-build.h"

No, please don't, at the very least not this way (using quotes rather than angle brackets). I may guess that it's for is_dom0less_mode(), but even then I wonder whether that declaration wouldn't better move elsewhere. It simply feels somewhat wrong to include this header here.

xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1194

Quoted 13 lines
xen/drivers/passthrough/pci.c +2 -2
@@ -1181,19 +1185,21 @@ int __init scan_pci_devices(void)
     return ret;
 }
 
-struct setup_hwdom {
+struct setup_ctxt {
     struct domain *d;
     int (*handler)(uint8_t devfn, struct pci_dev *pdev);
 };
 
-static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
+static void __hwdom_init setup_one_pci_device(const struct setup_ctxt *ctxt,
                                                 struct pci_dev *pdev)

Nit: Indentation then also needds to change on this following line.

xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1197

Quoted 4 lines
xen/drivers/passthrough/pci.c +1 -1
 {
     u8 devfn = pdev->devfn;
-    int err;
+    int err = 0;

This doesn't suffice, as ...

xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1205

Quoted 8 lines
xen/drivers/passthrough/pci.c +3 -1
     do {
-        err = ctxt->handler(devfn, pdev);
+        if ( ctxt->handler )
+            err = ctxt->handler(devfn, pdev);
+
         if ( err )
         {
             printk(XENLOG_ERR "setup %pp for d%d failed (%d)\n",

... below here we may continue the loop even if we got an error. "err" needs setting unconditionally in the loop body, and hence maybe better with a conditional expression.

xen/drivers/passthrough/pci.c static int __hwdom_init cf_check _setup_hwdom_pci_devices( · line 1238

Quoted 5 lines
xen/drivers/passthrough/pci.c +1 -0
@@ -1229,18 +1235,26 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices(
             if ( !pdev )
                 continue;
 
+            if ( is_dom0less_mode() ) {

We're in a __hwdom_init function. You can't call an __init one from here.

Also nit (style): Brace placement.

xen/drivers/passthrough/pci.c static int __hwdom_init cf_check _setup_hwdom_pci_devices( · line 1241

Quoted 3 lines
xen/drivers/passthrough/pci.c +3 -0
+                int ret = pci_add_device(pdev->seg, pdev->bus, pdev->devfn, NULL,
+                                         NUMA_NO_NODE, ctxt->d);
+                if (ret)

Nit (style): Missing blanks.

xen/include/xen/pci.h int scan_pci_devices(void); · line 224

Quoted 11 lines
xen/include/xen/pci.h +2 -3
@@ -220,9 +220,8 @@ int scan_pci_devices(void);
 enum pdev_type pdev_type(u16 seg, u8 bus, u8 devfn);
 int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);
 
-void setup_hwdom_pci_devices(struct domain *d,
-                             int (*handler)(uint8_t devfn,
-                                            struct pci_dev *pdev));
+void setup_pci_devices(struct domain *d, int (*handler)(uint8_t devfn,
+                                                        struct pci_dev *pdev));

I think in this case the 2nd parameter would better remain on the following line, to limit overall indentation.

Jan

#8 lore raw

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

Quoted 29 lines · Jan Beulich
Quoted 19 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.

In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.

Quoted 8 lines

---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io

Quoted 5 lines

What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the dom_io intermediary is necessary in the first place.

Jan

I've done some testing and indeed everything seems to work if we call pci_add_device directly during domain construction instead of reassigning them. Do you think this would be a better approach? If so then I guess this series needs to be dropped, and I will prepare a new one with direct device assignment to DomUs.

Signature · 1 line
Mykyta
#9 lore raw

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

Quoted 30 lines · Mykyta Poturai
Quoted 29 lines · Jan Beulich
Quoted 27 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.

In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.

---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io

Quoted 7 lines
Quoted 3 lines

What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the dom_io intermediary is necessary in the first place.

I've done some testing and indeed everything seems to work if we call pci_add_device directly during domain construction instead of reassigning them. Do you think this would be a better approach?

I think so, yes, but first and foremost you'll need Arm maintainer buyoff on either approach (or yet another one).

Jan

Quoted 3 lines

If so
then I guess this series needs to be dropped, and I will prepare a new one with direct device assignment to DomUs.

#10 lore raw

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Mykyta Poturai
Quoted 32 lines · Jan Beulich
Quoted 19 lines · Mykyta Poturai

From: Luca Fancellu <[email protected]>

In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.

In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.

Quoted 8 lines

---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io

What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the dom_io intermediary is necessary in the first place.

Quoted 6 lines
Quoted 3 lines

I've done some testing and indeed everything seems to work if we call pci_add_device directly during domain construction instead of reassigning them. Do you think this would be a better approach?

I think so, yes, but first and foremost you'll need Arm maintainer buyoff on either approach (or yet another one).

I am OK with that in principle