dom0less pci passthrough support on Arm part 1
PATCH v2 1/3 arm/pci: Add pci-scan boot argument
In reply to Mykyta PoturaiFrom: Edward Pickup <[email protected]>
This patch adds a Xen boot arguments that, if enabled, causes a call to existing code to scan pci devices enumerated by the firmware.
This will be needed ahead of dom0less support for pci passthrough on arm.
---
(cherry picked from commit bce463e1588a45e1bfdf59fc0d5f88b16604e439 from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
v1->v2:
* remove is_pci_scan_enabled wrapper
* make pci_scan_enabled ro_after_init
* drop debug prints
* drop Edward's SOB
changes since cherry-pick:
* s/always_inline/inline/
* replace additional kconfig option with config DEBUG
--- docs/misc/xen-command-line.pandoc | 7 +++++++ xen/arch/arm/include/asm/pci.h | 3 +++ xen/arch/arm/pci/pci-host-common.c | 1 + xen/arch/arm/pci/pci.c | 24 ++++++++++++++++++++++-- 4 files changed, 33 insertions(+), 2 deletions(-)
docs/misc/xen-command-line.pandoc +7 -0
@@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present). Flag to enable or disable support for PCI passthrough +### pci-scan (arm) +> `= <boolean>` + +> Default: `false` + +Flag to enable or disable Xen PCI scan at boot. + ### pcid (x86) > `= <boolean> | xpti=<bool>`
xen/arch/arm/include/asm/pci.h +3 -0
@@ -23,6 +23,7 @@#define pci_to_dev (pcidev) (&(pcidev)->arch.dev)extern bool pci_passthrough_enabled; +extern bool pci_scan_enabled;struct rangeset ;
@@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);#else /*!CONFIG_HAS_PCI*/ +#define pci_scan_enabled false +struct pci_dev ;static inline void arch_pci_init_pdev (struct pci_dev *pdev) {}
xen/arch/arm/pci/pci-host-common.c +1 -0
@@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev, }pci_add_host_bridge (bridge); +pci_add_segment (bridge->segment);return bridge;
xen/arch/arm/pci/pci.c +22 -2
@@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)bool __read_mostly pci_passthrough_enabled;boolean_param ("pci-passthrough" , pci_passthrough_enabled); +/* By default pci scan is disabled. */ +bool __ro_after_init pci_scan_enabled; +boolean_param ("pci-scan" , pci_scan_enabled); +static int __init pci_init (void ) { +int ret; +/* * Enable PCI passthrough when has been enabled explicitly * (pci-passthrough=on).
@@ -104,9 +110,23 @@ static int __init pci_init(void)panic ("Could not initialize PCI segment 0\n" );if ( acpi_disabled ) -return dt_pci_init (); + ret =dt_pci_init ();else -return acpi_pci_init (); + ret =acpi_pci_init (); + +if ( ret <0 ) +return ret; + +if ( pci_scan_enabled ) + { + ret =scan_pci_devices (); + +if ( ret <0 ) +return ret; + + } + +return 0 ; }__initcall (pci_init);
Signature · 1 line
2.34.1
PATCH v2 2/3 xen/pci: modify pci_add_device to handle device add by Xen
In reply to Mykyta PoturaiFrom: Luca Fancellu <[email protected]>
Currently pci_add_device is called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed that hardware_domain is the owner.
On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.
Modify pci_add_device to have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).
---
(cherry picked from commit f0c85d9043f7c9402e85b73361c8a13c683428ca from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
v1->v2:
* remove dom_io check
* fixup pci_add_device parameters
* use current->domain instead of hardware_domain
changes since cherry-pick:
* s/hardware_domain/d/ in some write_unlock calls in xen/drivers/passthrough/pci.c
--- xen/arch/x86/physdev.c | 9 ++++----- xen/drivers/passthrough/pci.c | 29 +++++++++++++++++++---------- xen/drivers/pci/physdev.c | 3 ++- xen/include/xen/pci.h | 5 +++-- 4 files changed, 28 insertions(+), 18 deletions(-)
xen/arch/x86/physdev.c +4 -5
@@ -446,8 +446,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)if (copy_from_guest (&manage_pci, arg,1 ) !=0 )break ; - ret =pci_add_device (0 , manage_pci.bus, manage_pci.devfn, -NULL ,NUMA_NO_NODE ); + ret =pci_add_device (0 , manage_pci.bus, manage_pci.devfn,NULL , +NUMA_NO_NODE , current->domain);break ; }
@@ -477,9 +477,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) pdev_info.is_virtfn = manage_pci_ext.is_virtfn; pdev_info.physfn.bus = manage_pci_ext.physfn.bus; pdev_info.physfn.devfn = manage_pci_ext.physfn.devfn; - ret =pci_add_device (0 , manage_pci_ext.bus, - manage_pci_ext.devfn, - &pdev_info,NUMA_NO_NODE ); + ret =pci_add_device (0 , manage_pci_ext.bus, manage_pci_ext.devfn, + &pdev_info,NUMA_NO_NODE , current->domain);break ; }
xen/drivers/passthrough/pci.c +19 -10
@@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,return is64bits ?2 :1 ; } -int pci_add_device (u16 seg,u8 bus,u8 devfn, -const struct pci_dev_info *info,nodeid_t node) +int pci_add_device (uint16_t seg,uint8_t bus,uint8_t devfn, +const struct pci_dev_info *info,nodeid_t node, +struct domain *d) {struct pci_seg *pseg;struct pci_dev *pdev;
@@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,const char *type;int ret; +if ( !d ) +return -EINVAL ; +if ( !info ) type ="device" ;else if ( info->is_virtfn )
@@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, ret =0 ;if ( !pdev->domain ) { - pdev->domain = hardware_domain; -write_lock (&hardware_domain->pci_lock); -list_add (&pdev->domain_list, &hardware_domain->pdev_list); + pdev->domain = d; +write_lock (&d->pci_lock); +list_add (&pdev->domain_list, &pdev->domain->pdev_list);/* * For devices not discovered by Xen during boot, add vPCI handlers
@@ -779,25 +783,30 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,if ( ret ) {list_del (&pdev->domain_list); -write_unlock (&hardware_domain->pci_lock); +write_unlock (&d->pci_lock); pdev->domain =NULL ;printk (XENLOG_ERR "Setup of vPCI failed: %d\n" , ret); goto out; } -write_unlock (&hardware_domain->pci_lock); +write_unlock (&d->pci_lock); ret =iommu_add_device (pdev);if ( ret ) { -write_lock (&hardware_domain->pci_lock); +write_lock (&d->pci_lock);vpci_deassign_device (pdev);list_del (&pdev->domain_list); -write_unlock (&hardware_domain->pci_lock); +write_unlock (&d->pci_lock); pdev->domain =NULL ; goto out; } } -else +else if ( pdev->domain == d )iommu_enable_device (pdev); +else + { + ret = -EINVAL ; + goto out; + }pci_enable_acs (pdev);
xen/drivers/pci/physdev.c +2 -1
@@ -50,7 +50,8 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) }#endif - ret =pci_add_device (add.seg, add.bus, add.devfn, &pdev_info, node); + ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node, current->domain);break ; }
xen/include/xen/pci.h +3 -2
@@ -226,8 +226,9 @@ void setup_hwdom_pci_devices(struct domain *d,int pci_release_devices (struct domain *d);int pci_add_segment (u16 seg);const unsigned long *pci_get_ro_map (u16 seg); -int pci_add_device (u16 seg,u8 bus,u8 devfn, -const struct pci_dev_info *info,nodeid_t node); +int pci_add_device (uint16_t seg,uint8_t bus,uint8_t devfn, +const struct pci_dev_info *info,nodeid_t node, +struct domain *d);int pci_remove_device (u16 seg,u8 bus,u8 devfn);int pci_ro_device (int seg,int bus,int devfn);int pci_hide_device (unsigned int seg,unsigned int bus,unsigned int devfn);
Signature · 1 line
2.34.1
v2 cover dom0less pci passthrough support on Arm part 1
This series adds basic PCI device enumeration in Xen. This will allow us to not rely on Dom0 enumeration for supported controllers, which will enable PCI passthrough for dom0less setups.
Enumeration is disabled by default and can be enabled with "pci-scan" cmdline option. Discovered devices are added to Xen and assigned to dom_io by default.
Edward Pickup (1):
arm/pci: Add pci-scan boot argument
Luca Fancellu (2):
xen/pci: modify pci_add_device to handle device add by Xen xen/pci: add discovered PCI device at boot
docs/misc/xen-command-line.pandoc | 7 ++ xen/arch/arm/include/asm/pci.h | 3 + xen/arch/arm/pci/pci-host-common.c | 1 + xen/arch/arm/pci/pci.c | 25 +++++++- xen/arch/x86/physdev.c | 9 ++- xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/pci.c | 71 ++++++++++++++------- xen/drivers/passthrough/vtd/iommu.c | 2 +- xen/drivers/pci/physdev.c | 3 +- xen/include/xen/pci.h | 10 +-- 10 files changed, 94 insertions(+), 39 deletions(-)
Signature · 1 line
2.34.1
PATCH v2 3/3 xen/pci: add discovered PCI device at boot
From: Luca Fancellu <[email protected]>
In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen. pci_add_device requires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so use dom_io as a temporary sentinel before devices are assigned to their target domains.
Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less handling to it.
In pci_add_device there is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain is dom_io, since it means the call is coming directly from Xen.
---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io
--- xen/arch/arm/pci/pci.c | 1 + xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/pci.c | 42 ++++++++++++++------- xen/drivers/passthrough/vtd/iommu.c | 2 +- xen/include/xen/pci.h | 5 +-- 5 files changed, 33 insertions(+), 19 deletions(-)
xen/arch/arm/pci/pci.c +1 -0
@@ -124,6 +124,7 @@ static int __init pci_init(void)if ( ret <0 )return ret; +setup_pci_devices (dom_io,NULL ); }return 0 ;
xen/drivers/passthrough/amd/pci_amd_iommu.c +1 -1
@@ -401,7 +401,7 @@ static void __hwdom_init cf_check amd_iommu_hwdom_init(struct domain *d)/* Make sure workarounds are applied (if needed) before adding devices. */ arch_iommu_hwdom_init (d); -setup_hwdom_pci_devices (d, amd_iommu_add_device); +setup_pci_devices (d, amd_iommu_add_device); } static void amd_iommu_disable_domain_device(const struct domain *domain,
xen/drivers/passthrough/pci.c +28 -14
103 lines not on this page · raw message
xen/drivers/passthrough/vtd/iommu.c +1 -1
@@ -1451,7 +1451,7 @@ static void __hwdom_init cf_check intel_iommu_hwdom_init(struct domain *d) {struct acpi_drhd_unit *drhd; -setup_hwdom_pci_devices (d, setup_hwdom_device); +setup_pci_devices (d, setup_hwdom_device);setup_hwdom_rmrr (d);/* Make sure workarounds are applied before enabling the IOMMU(s). */ arch_iommu_hwdom_init (d);
xen/include/xen/pci.h +2 -3
@@ -220,9 +220,8 @@ int scan_pci_devices(void);enum pdev_type pdev_type (u16 seg,u8 bus,u8 devfn);int find_upstream_bridge (u16 seg,u8 *bus,u8 *devfn,u8 *secbus); -void setup_hwdom_pci_devices (struct domain *d, -int (*handler)(uint8_t devfn, -struct pci_dev *pdev)); +void setup_pci_devices (struct domain *d,int (*handler)(uint8_t devfn, +struct pci_dev *pdev));int pci_release_devices (struct domain *d);int pci_add_segment (u16 seg);const unsigned long *pci_get_ro_map (u16 seg);
Signature · 1 line
2.34.1
Re: [PATCH v2 1/3] arm/pci: Add pci-scan boot argument
In reply to Mykyta PoturaiQuoted 10 lines · Mykyta Poturai
From: Edward Pickup <[email protected]>
This patch adds a Xen boot arguments that, if enabled, causes a call to existing code to scan pci devices enumerated by the firmware.
This will be needed ahead of dom0less support for pci passthrough on arm.
Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>
Quoted 122 lines
---
(cherry picked from commitbce463e1588a45e1bfdf59fc0d5f88b16604e439from
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
v1->v2:
* removeis_pci_scan_enabledwrapper
* makepci_scan_enabledro_after_init
* drop debug prints
* drop Edward's SOBchanges since cherry-pick:
* s/always_inline/inline/
* replace additional kconfig option with config DEBUGSigned-off-by Stewart Hildebrand <[email protected]>--- docs/misc/xen-command-line.pandoc | 7 +++++++ xen/arch/arm/include/asm/pci.h | 3 +++ xen/arch/arm/pci/pci-host-common.c | 1 + xen/arch/arm/pci/pci.c | 24 ++++++++++++++++++++++-- 4 files changed, 33 insertions(+), 2 deletions(-)docs/misc/xen-command-line.pandoc +7
-0@@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present). Flag to enable or disable support for PCI passthrough +### pci-scan (arm) +> `= <boolean>` + +> Default: `false` + +Flag to enable or disable Xen PCI scan at boot. + ### pcid (x86) > `= <boolean> | xpti=<bool>`xen/arch/arm/include/asm/pci.h +3
-0@@ -23,6 +23,7 @@#define pci_to_dev (pcidev) (&(pcidev)->arch.dev)extern bool pci_passthrough_enabled; +extern bool pci_scan_enabled;struct rangeset ;@@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);#else /*!CONFIG_HAS_PCI*/ +#define pci_scan_enabled false +struct pci_dev ;static inline void arch_pci_init_pdev (struct pci_dev *pdev) {}xen/arch/arm/pci/pci-host-common.c +1
-0@@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev, }pci_add_host_bridge (bridge); +pci_add_segment (bridge->segment);return bridge;xen/arch/arm/pci/pci.c +22
-2@@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)bool __read_mostly pci_passthrough_enabled;boolean_param ("pci-passthrough" , pci_passthrough_enabled); +/* By default pci scan is disabled. */ +bool __ro_after_init pci_scan_enabled; +boolean_param ("pci-scan" , pci_scan_enabled); +static int __init pci_init (void ) { +int ret; +/* * Enable PCI passthrough when has been enabled explicitly * (pci-passthrough=on). @@ -104,9 +110,23 @@ static int __init pci_init(void)panic ("Could not initialize PCI segment 0\n" );if ( acpi_disabled ) -return dt_pci_init (); + ret =dt_pci_init ();else -return acpi_pci_init (); + ret =acpi_pci_init (); + +if ( ret <0 ) +return ret; + +if ( pci_scan_enabled ) + { + ret =scan_pci_devices (); + +if ( ret <0 ) +return ret; + + } + +return 0 ; }__initcall (pci_init);Signature · 1 line
2.34.1
Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen
In reply to Mykyta PoturaiQuoted 11 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
Currently
pci_add_deviceis called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed thathardware_domainis the owner.On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.
Quoted 8 lines
Modify
pci_add_deviceto have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>
Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen
In reply to Mykyta PoturaiQuoted 24 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
Currently
pci_add_deviceis called from hypercalls requested by Dom0 to add pci devices and when the device has no domain associated with it, it is assumed thathardware_domainis the owner.On the dom0less scenario, the enumeration is performed by the firmware and Xen at boot time might want to assign some pci devices to guests, so it has to firstly add the device and then assign it to the final guest.
Modify
pci_add_deviceto have the owner domain passed as a parameter to the function, so that when it is called from the hypercall the owner would be the caller domain, otherwise when Xen is calling it, it would be another domain since hw domain could not be there (dom0less guests without Dom0 use case).Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>---
(cherry picked from commitf0c85d9043f7c9402e85b73361c8a13c683428cafrom
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
Quoted 4 lines
v1->v2:
* removedom_iocheck
* fixuppci_add_deviceparameters
* usecurrent->domaininstead ofhardware_domain
What I'm missing (as per my v1 comment) is discussion of the hardware_domain -> current->domain change, including the XSM aspect. Because of the XSM aspect, please also Cc the XSM maintainer going forward (I'm adding him here as well).
Quoted 15 lines
xen/drivers/passthrough/pci.c +3
-2@@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,return is64bits ?2 :1 ; } -int pci_add_device (u16 seg,u8 bus,u8 devfn, -const struct pci_dev_info *info,nodeid_t node) +int pci_add_device (uint16_t seg,uint8_t bus,uint8_t devfn, +const struct pci_dev_info *info,nodeid_t node, +struct domain *d) {struct pci_seg *pseg;struct pci_dev *pdev;
xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 668
Quoted 5 lines
xen/drivers/passthrough/pci.c +2
-0@@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,const char *type;int ret; +if ( !d ) +return -EINVAL ;
This should't be needed. Very remotely ASSERT(d) could be added here, but we don't normally do so elsewhere.
xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 776
Quoted 10 lines
xen/drivers/passthrough/pci.c +3
-3@@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, ret =0 ;if ( !pdev->domain ) { - pdev->domain = hardware_domain; -write_lock (&hardware_domain->pci_lock); -list_add (&pdev->domain_list, &hardware_domain->pdev_list); + pdev->domain = d; +write_lock (&d->pci_lock); +list_add (&pdev->domain_list, &pdev->domain->pdev_list);
Why pdev->domain instead of the shorter and more efficient d?
Jan
Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot
In reply to Mykyta PoturaiQuoted 19 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen.
pci_add_devicerequires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so usedom_ioas a temporary sentinel before devices are assigned to their target domains.Rename
setup_hwdom_pci_devicestosetup_pci_devicesand add dom0less handling to it.In
pci_add_devicethere is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain isdom_io, since it means the call is coming directly from Xen.Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>
Quoted 8 lines
---
(cherry picked from commiteff51e50021b75f5a50533f7de681b2ce044f5bdfrom
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
v1->v2:
* integrateadd_discovered_pci_devicesinto existing routines
* better explain the need fordom_io
What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the dom_io intermediary is necessary in the first place.
xen/drivers/passthrough/pci.c line 38
Quoted 7 lines
xen/drivers/passthrough/pci.c +1
-0@@ -35,6 +35,7 @@#include <xen/msi.h> #include <xsm/xsm.h> #include "ats.h" +#include "xen/dom0less-build.h"
No, please don't, at the very least not this way (using quotes rather than angle brackets). I may guess that it's for is_dom0less_mode(), but even then I wonder whether that declaration wouldn't better move elsewhere. It simply feels somewhat wrong to include this header here.
xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1194
Quoted 13 lines
xen/drivers/passthrough/pci.c +2
-2@@ -1181,19 +1185,21 @@ int __init scan_pci_devices(void)return ret; } -struct setup_hwdom { +struct setup_ctxt {struct domain *d;int (*handler)(uint8_t devfn,struct pci_dev *pdev); }; -static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, +static void __hwdom_init setup_one_pci_device(const struct setup_ctxt *ctxt,struct pci_dev *pdev)
Nit: Indentation then also needds to change on this following line.
xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1197
Quoted 4 lines
xen/drivers/passthrough/pci.c +1
-1{u8 devfn = pdev->devfn; -int err; +int err =0 ;
This doesn't suffice, as ...
xen/drivers/passthrough/pci.c int __init scan_pci_devices(void) · line 1205
Quoted 8 lines
xen/drivers/passthrough/pci.c +3
-1do { - err = ctxt->handler(devfn, pdev); + if ( ctxt->handler ) + err = ctxt->handler(devfn, pdev); + if ( err ) { printk(XENLOG_ERR "setup %pp for d%d failed (%d)\n",
... below here we may continue the loop even if we got an error. "err" needs setting unconditionally in the loop body, and hence maybe better with a conditional expression.
xen/drivers/passthrough/pci.c static int __hwdom_init cf_check _setup_hwdom_pci_devices( · line 1238
Quoted 5 lines
xen/drivers/passthrough/pci.c +1
-0@@ -1229,18 +1235,26 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices(if ( !pdev )continue ; +if (is_dom0less_mode () ) {
We're in a __hwdom_init function. You can't call an __init one from here.
Also nit (style): Brace placement.
xen/drivers/passthrough/pci.c static int __hwdom_init cf_check _setup_hwdom_pci_devices( · line 1241
Quoted 3 lines
xen/drivers/passthrough/pci.c +3
-0+int ret =pci_add_device (pdev->seg, pdev->bus, pdev->devfn,NULL , +NUMA_NO_NODE , ctxt->d); +if (ret)
Nit (style): Missing blanks.
xen/include/xen/pci.h int scan_pci_devices(void); · line 224
Quoted 11 lines
xen/include/xen/pci.h +2
-3@@ -220,9 +220,8 @@ int scan_pci_devices(void);enum pdev_type pdev_type (u16 seg,u8 bus,u8 devfn);int find_upstream_bridge (u16 seg,u8 *bus,u8 *devfn,u8 *secbus); -void setup_hwdom_pci_devices (struct domain *d, -int (*handler)(uint8_t devfn, -struct pci_dev *pdev)); +void setup_pci_devices (struct domain *d,int (*handler)(uint8_t devfn, +struct pci_dev *pdev));
I think in this case the 2nd parameter would better remain on the following line, to limit overall indentation.
Jan
Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot
Quoted 29 lines · Jan Beulich
Quoted 19 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen.
pci_add_devicerequires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so usedom_ioas a temporary sentinel before devices are assigned to their target domains.Rename
setup_hwdom_pci_devicestosetup_pci_devicesand add dom0less handling to it.In
pci_add_devicethere is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain isdom_io, since it means the call is coming directly from Xen.Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>Quoted 8 lines
---
(cherry picked from commiteff51e50021b75f5a50533f7de681b2ce044f5bdfrom
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
v1->v2:
* integrateadd_discovered_pci_devicesinto existing routines
* better explain the need fordom_io
Quoted 5 lines
What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the
dom_iointermediary is necessary in the first place.Jan
I've done some testing and indeed everything seems to work if we call pci_add_device directly during domain construction instead of reassigning them. Do you think this would be a better approach? If so then I guess this series needs to be dropped, and I will prepare a new one with direct device assignment to DomUs.
Signature · 1 line
Mykyta
Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot
Quoted 30 lines · Mykyta Poturai
Quoted 29 lines · Jan Beulich
Quoted 27 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen.
pci_add_devicerequires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so usedom_ioas a temporary sentinel before devices are assigned to their target domains.Rename
setup_hwdom_pci_devicestosetup_pci_devicesand add dom0less handling to it.In
pci_add_devicethere is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain isdom_io, since it means the call is coming directly from Xen.Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>---
(cherry picked from commiteff51e50021b75f5a50533f7de681b2ce044f5bdfrom
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
v1->v2:
* integrateadd_discovered_pci_devicesinto existing routines
* better explain the need fordom_io
Quoted 7 lines
Quoted 3 lines
What I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the
dom_iointermediary is necessary in the first place.I've done some testing and indeed everything seems to work if we call
pci_add_devicedirectly during domain construction instead of reassigning them. Do you think this would be a better approach?
I think so, yes, but first and foremost you'll need Arm maintainer buyoff on either approach (or yet another one).
Jan
Quoted 3 lines
If so
then I guess this series needs to be dropped, and I will prepare a new one with direct device assignment to DomUs.
Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot
Quoted 35 lines · Jan Beulich
Quoted 34 lines · Mykyta Poturai
Quoted 32 lines · Jan Beulich
Quoted 19 lines · Mykyta Poturai
From: Luca Fancellu <[email protected]>
In dom0less mode, there is no dom0 that can call PCI physdev ops to register PCI devices to iommu, so it needs to be done by Xen.
pci_add_devicerequires some default domain, we don't have hwdom, and the guests are not yet created during the PCI init phase, so usedom_ioas a temporary sentinel before devices are assigned to their target domains.Rename
setup_hwdom_pci_devicestosetup_pci_devicesand add dom0less handling to it.In
pci_add_devicethere is a call to xsm that doesn't consider the requester of the function to be Xen itself, so add a check to skip the call if the owner domain isdom_io, since it means the call is coming directly from Xen.Signed-off-by Luca Fancellu <[email protected]>Signed-off-by Mykyta Poturai <[email protected]>Quoted 8 lines
---
(cherry picked from commiteff51e50021b75f5a50533f7de681b2ce044f5bdfrom
the downstream branch poc/pci-passthrough from
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
v1->v2:
* integrateadd_discovered_pci_devicesinto existing routines
* better explain the need fordom_ioWhat I continue to miss is an explanation of why devices can't go to their ultimate "destination" domain right away (once those have been created), i.e. why the
dom_iointermediary is necessary in the first place.
Quoted 6 lines
Quoted 3 lines
I've done some testing and indeed everything seems to work if we call
pci_add_devicedirectly during domain construction instead of reassigning them. Do you think this would be a better approach?I think so, yes, but first and foremost you'll need Arm maintainer buyoff on either approach (or yet another one).
I am OK with that in principle