iommu groups + cleanup
v3 cover iommu groups + cleanup
This series is a mixture of tidying and some preparatory work for grouping PCI devices for the purposes of assignment.
Paul Durrant (4):
iommu / x86: move call to scan_pci_devices() out of vendor code
pci: add all-device iterator function...
iommu: introduce iommu_groups
iommu / pci: re-implement XEN_DOMCTL_get_device_group...
xen/drivers/passthrough/Makefile | 1 + xen/drivers/passthrough/amd/pci_amd_iommu.c | 3 +- xen/drivers/passthrough/groups.c | 137 ++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 172 +++++++++++----------------- xen/drivers/passthrough/vtd/iommu.c | 4 - xen/drivers/passthrough/x86/iommu.c | 14 ++- xen/include/xen/iommu.h | 10 ++ xen/include/xen/pci.h | 3 + 8 files changed, 236 insertions(+), 108 deletions(-) create mode 100644 xen/drivers/passthrough/groups.c
---
v2:
- Drop iommu_get_ops() move and add all-device iterator
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 1/4 iommu / x86: move call to scan_pci_devices() out of vendor code
It's not vendor specific so it doesn't really belong there.
Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused.
Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.
---
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Brian Woods <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
v2:
- Expanded commit comment.
- Moved PCI scan to before IOMMU initialization, rather than after it.
--- xen/drivers/passthrough/amd/pci_amd_iommu.c | 3 ++- xen/drivers/passthrough/vtd/iommu.c | 4 ---- xen/drivers/passthrough/x86/iommu.c | 6 ++++++ 3 files changed, 8 insertions(+), 5 deletions(-)
xen/drivers/passthrough/amd/pci_amd_iommu.c +2 -1
@@ -180,7 +180,8 @@ static int __init iov_detect(void)if ( !amd_iommu_perdev_intremap )printk (XENLOG_WARNING "AMD-Vi: Using global interrupt remap table is not recommended (see XSA-36)!\n" ); -return scan_pci_devices (); + +return 0 ; }int amd_iommu_alloc_root (struct domain_iommu *hd)
xen/drivers/passthrough/vtd/iommu.c +0 -4
@@ -2372,10 +2372,6 @@ static int __init vtd_setup(void)P (iommu_hap_pt_share,"Shared EPT tables" );#undef P - ret =scan_pci_devices (); -if ( ret ) - goto error; - ret =init_vtd_hw ();if ( ret ) goto error;
xen/drivers/passthrough/x86/iommu.c +6 -0
@@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops;int __init iommu_hardware_setup (void ) { +int rc; +if ( !iommu_init_ops )return -ENODEV ; + rc =scan_pci_devices (); +if ( rc ) +return rc; +if ( !iommu_ops.init ) iommu_ops = *iommu_init_ops->ops;else
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code
Quoted 10 lines
From: Paul Durrant [mailto:[email protected]]
Sent: Tuesday, July 16, 2019 6:17 PMIt's not vendor specific so it doesn't really belong there.
Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of
iommu_hardware_setup()but only places it there because the topology information would be otherwise unused.
Quoted 5 lines
Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.
Signed-off-by Paul Durrant <[email protected]>Reviewed-by "Roger Pau Monné" <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code
In reply to Paul DurrantQuoted 4 lines · Paul Durrant
xen/drivers/passthrough/x86/iommu.c +0
-0@@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops;
xen/drivers/passthrough/x86/iommu.c struct iommu_ops __read_mostly iommu_ops; · line 38
Quoted 10 lines
xen/drivers/passthrough/x86/iommu.c +5
-0int __init iommu_hardware_setup (void ) { +int rc; +if ( !iommu_init_ops )return -ENODEV ; + rc =scan_pci_devices (); +if ( rc ) +return rc;
From an abstract POV I'm not convinced failing IOMMU init because
a failed bus scan is appropriate. But the only currently possible failure is -ENOMEM, in which case we'd be in bigger trouble anyway. Therefore
The other question of course is in how far you can sensibly use the results of this (incomplete) bus scan later during IOMMU init. But hopefully that'll become clear from the subsequent patches.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code
Quoted 29 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 24 July 2019 14:41
To: Paul Durrant <[email protected]>
Cc: [email protected]; Brian Woods <[email protected]>; Suravee Suthikulpanit <[email protected]>; Andrew Cooper <[email protected]>; Kevin Tian <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v3 1/4] iommu / x86: move call toscan_pci_devices()out of vendor codeQuoted 4 lines · Paul Durrant
xen/drivers/passthrough/x86/iommu.c +0
-0@@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops;xen/drivers/passthrough/x86/iommu.c struct iommu_ops __read_mostly iommu_ops; · line 38
Quoted 10 lines
xen/drivers/passthrough/x86/iommu.c +5
-0int __init iommu_hardware_setup (void ) { +int rc; +if ( !iommu_init_ops )return -ENODEV ; + rc =scan_pci_devices (); +if ( rc ) +return rc;From an abstract POV I'm not convinced failing IOMMU init because
a failed bus scan is appropriate. But the only currently possible failure is-ENOMEM, in which case we'd be in bigger trouble anyway. Therefore
Quoted 5 lines
Acked-by Jan Beulich <[email protected]>The other question of course is in how far you can sensibly use the results of this (incomplete) bus scan later during IOMMU init. But hopefully that'll become clear from the subsequent patches.
The only use, at the moment, is for group assignment... but I do need to check that I haven't missed doing group assignment for subsequently added devices. I have a feeling I did miss it.
Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 2/4 pci: add all-device iterator function...
In reply to Paul Durrant...and use it for setup_hwdom_pci_devices() and dump_pci_devices().
The unlock/process-pending-softirqs/lock sequence that was in _setup_hwdom_pci_devices() is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case of dump_pci_devices(), since run_all_nonirq_keyhandlers() will call process_pending_softirqs() before invoking each key handler anyway, but this is not performance critical code.
The "==== segment XXXX ====" headline that was in _dump_pci_devices() has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
v3:
- Addressed review comments from Roger.
v2:
- New in v2.
--- xen/drivers/passthrough/pci.c | 121 ++++++++++++++++++++++++------------------ xen/include/xen/pci.h | 1 + 2 files changed, 69 insertions(+), 53 deletions(-)
xen/drivers/passthrough/pci.c +68 -53
@@ -1134,64 +1134,87 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, ctxt->d->domain_id, err); } -static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg) +static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg) {struct setup_hwdom *ctxt = arg; -int bus, devfn; +struct domain *d = ctxt->d; -for ( bus =0 ; bus <256 ; bus++ ) +if ( !pdev->domain ) + { + pdev->domain = d; +list_add (&pdev->domain_list, &d->pdev_list); +setup_one_hwdom_device (ctxt, pdev); + } +else if ( pdev->domain == dom_xen ) { -for ( devfn =0 ; devfn <256 ; devfn++ ) + pdev->domain = d; +setup_one_hwdom_device (ctxt, pdev); + pdev->domain = dom_xen; + } +else if ( pdev->domain != d ) +printk (XENLOG_WARNING "Dom%pd owning %04x:%02x:%02x.%u?\n" , + pdev->domain, pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), +PCI_FUNC (pdev->devfn)); + +return 0 ; +} + +struct psdi_ctxt { +int (*cb)(struct pci_dev *,void *); +void *arg; +}; + +static int pci_segment_devices_iterate (struct pci_seg *pseg,void *arg) +{ +struct psdi_ctxt *ctxt = arg; +unsigned int bus, devfn; +int rc =0 ; + +/* +* We don't iterate by walking pseg->alldevs_list here because that +* would make the pcidevs_unlock()/lock() sequence below unsafe. +*/ +for ( bus =0 ; !rc && bus <256 ; bus++ ) +for ( devfn =0 ; !rc && devfn <256 ; devfn++ ) {struct pci_dev *pdev =pci_get_pdev (pseg->nr, bus, devfn);if ( !pdev )continue ; -if ( !pdev->domain ) - { - pdev->domain = ctxt->d; -list_add (&pdev->domain_list, &ctxt->d->pdev_list); -setup_one_hwdom_device (ctxt, pdev); - } -else if ( pdev->domain == dom_xen ) - { - pdev->domain = ctxt->d; -setup_one_hwdom_device (ctxt, pdev); - pdev->domain = dom_xen; - } -else if ( pdev->domain != ctxt->d ) -printk (XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n" , - pdev->domain->domain_id, pseg->nr, bus, -PCI_SLOT (devfn),PCI_FUNC (devfn)); - -if ( iommu_verbose ) - { -pcidevs_unlock (); -process_pending_softirqs (); -pcidevs_lock (); - } - } + rc = ctxt->cb (pdev, ctxt->arg); -if ( !iommu_verbose ) - { +/* +* Err on the safe side and assume the callback has taken +* a significant amount of time. +*/ pcidevs_unlock ();process_pending_softirqs ();pcidevs_lock (); } - } -return 0 ; +return rc; +} + +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg) +{ +struct psdi_ctxt ctxt = { .cb = cb, .arg = arg }; +int rc; + +pcidevs_lock (); + rc =pci_segments_iterate (pci_segment_devices_iterate, &ctxt); +pcidevs_unlock (); + +return rc; } void __hwdom_init setup_hwdom_pci_devices(struct domain *d,int (*handler)(u8 devfn,struct pci_dev *)) {struct setup_hwdom ctxt = { .d = d, .handler = handler }; +int rc =pci_pdevs_iterate (setup_hwdom_pci_device, &ctxt); -pcidevs_lock (); -pci_segments_iterate (_setup_hwdom_pci_devices, &ctxt); -pcidevs_unlock (); +ASSERT (!rc); }#ifdef CONFIG_ACPI
@@ -1294,24 +1317,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) }#endif -static int _dump_pci_devices (struct pci_seg *pseg,void *arg) +static int dump_pci_device (struct pci_dev *pdev,void *arg) { -struct pci_dev *pdev;struct msi_desc *msi; -printk ("==== segment %04x ====\n" , pseg->nr); - - list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) - { -printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , - pseg->nr, pdev->bus, -PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn), - pdev->domain ? pdev->domain->domain_id :-1 , - (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); - list_for_each_entry ( msi, &pdev->msi_list, list ) -printk ("%d " , msi->irq); -printk (">\n" ); - } +printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , + pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), +PCI_FUNC (pdev->devfn), + pdev->domain ? pdev->domain->domain_id :-1 , + (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); + list_for_each_entry ( msi, &pdev->msi_list, list ) +printk ("%d " , msi->irq); +printk (">\n" );return 0 ; }
@@ -1319,9 +1336,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)static void dump_pci_devices (unsigned char ch) {printk ("==== PCI devices ====\n" ); -pcidevs_lock (); -pci_segments_iterate (_dump_pci_devices,NULL ); -pcidevs_unlock (); +pci_pdevs_iterate (dump_pci_device,NULL ); }static int __init setup_dump_pcidevs (void )
xen/include/xen/pci.h +1 -0
@@ -154,6 +154,7 @@ int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);struct pci_dev *pci_lock_pdev (int seg,int bus,int devfn);struct pci_dev *pci_lock_domain_pdev (struct domain *,int seg,int bus,int devfn); +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg);void setup_hwdom_pci_devices (struct domain *,int (*)(u8 devfn,struct pci_dev *));
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 2/4] pci: add all-device iterator function...
Quoted 58 lines · Paul Durrant
...and use it for
setup_hwdom_pci_devices()anddump_pci_devices().The unlock/process-pending-softirqs/lock sequence that was in
_setup_hwdom_pci_devices()is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case ofdump_pci_devices(), sincerun_all_nonirq_keyhandlers()will callprocess_pending_softirqs()before invoking each key handler anyway, but this is not performance critical code.The "==== segment XXXX ====" headline that was in
_dump_pci_devices()has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.Signed-off-by Paul Durrant <[email protected]>Reviewed-by "Roger Pau Monné" <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>v3:
- Addressed review comments from Roger.v2:
- New in v2.--- xen/drivers/passthrough/pci.c | 121 ++++++++++++++++++++++++------------------ xen/include/xen/pci.h | 1 + 2 files changed, 69 insertions(+), 53 deletions(-)xen/drivers/passthrough/pci.c +7
-3@@ -1134,64 +1134,87 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, ctxt->d->domain_id, err); } -static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg) +static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg) {struct setup_hwdom *ctxt = arg; -int bus, devfn; +struct domain *d = ctxt->d; -for ( bus =0 ; bus <256 ; bus++ ) +if ( !pdev->domain ) + { + pdev->domain = d; +list_add (&pdev->domain_list, &d->pdev_list); +setup_one_hwdom_device (ctxt, pdev);
xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1155
Quoted 10 lines
xen/drivers/passthrough/pci.c +8
-1+ } +else if ( pdev->domain == dom_xen ) { -for ( devfn =0 ; devfn <256 ; devfn++ ) + pdev->domain = d; +setup_one_hwdom_device (ctxt, pdev); + pdev->domain = dom_xen; + } +else if ( pdev->domain != d ) + printk(XENLOG_WARNING "Dom%pd owning %04x:%02x:%02x.%u?\n",
Just %pd. It takes care of rendering d%u or d[$name] for system domains.
Can be fixed on commit.
Quoted 116 lines
xen/drivers/passthrough/pci.c +44
-43+ pdev->domain, pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), + PCI_FUNC(pdev->devfn)); + +return 0 ; +} + +struct psdi_ctxt { +int (*cb)(struct pci_dev *,void *); +void *arg; +}; + +static int pci_segment_devices_iterate (struct pci_seg *pseg,void *arg) +{ +struct psdi_ctxt *ctxt = arg; +unsigned int bus, devfn; +int rc =0 ; + +/* +* We don't iterate by walking pseg->alldevs_list here because that +* would make the pcidevs_unlock()/lock() sequence below unsafe. +*/ +for ( bus =0 ; !rc && bus <256 ; bus++ ) +for ( devfn =0 ; !rc && devfn <256 ; devfn++ ) {struct pci_dev *pdev =pci_get_pdev (pseg->nr, bus, devfn);if ( !pdev )continue ; -if ( !pdev->domain ) - { - pdev->domain = ctxt->d; -list_add (&pdev->domain_list, &ctxt->d->pdev_list); -setup_one_hwdom_device (ctxt, pdev); - } -else if ( pdev->domain == dom_xen ) - { - pdev->domain = ctxt->d; -setup_one_hwdom_device (ctxt, pdev); - pdev->domain = dom_xen; - } -else if ( pdev->domain != ctxt->d ) -printk (XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n" , - pdev->domain->domain_id, pseg->nr, bus, -PCI_SLOT (devfn),PCI_FUNC (devfn)); - -if ( iommu_verbose ) - { -pcidevs_unlock (); -process_pending_softirqs (); -pcidevs_lock (); - } - } + rc = ctxt->cb (pdev, ctxt->arg); -if ( !iommu_verbose ) - { +/* +* Err on the safe side and assume the callback has taken +* a significant amount of time. +*/ pcidevs_unlock ();process_pending_softirqs ();pcidevs_lock (); } - } -return 0 ; +return rc; +} + +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg) +{ +struct psdi_ctxt ctxt = { .cb = cb, .arg = arg }; +int rc; + +pcidevs_lock (); + rc =pci_segments_iterate (pci_segment_devices_iterate, &ctxt); +pcidevs_unlock (); + +return rc; } void __hwdom_init setup_hwdom_pci_devices(struct domain *d,int (*handler)(u8 devfn,struct pci_dev *)) {struct setup_hwdom ctxt = { .d = d, .handler = handler }; +int rc =pci_pdevs_iterate (setup_hwdom_pci_device, &ctxt); -pcidevs_lock (); -pci_segments_iterate (_setup_hwdom_pci_devices, &ctxt); -pcidevs_unlock (); +ASSERT (!rc); }#ifdef CONFIG_ACPI @@ -1294,24 +1317,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) }#endif -static int _dump_pci_devices (struct pci_seg *pseg,void *arg) +static int dump_pci_device (struct pci_dev *pdev,void *arg) { -struct pci_dev *pdev;struct msi_desc *msi; -printk ("==== segment %04x ====\n" , pseg->nr); - - list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) - { -printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , - pseg->nr, pdev->bus, -PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn), - pdev->domain ? pdev->domain->domain_id :-1 , - (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); -list_for_each_entry ( msi, &pdev->msi_list, list )
xen/drivers/passthrough/pci.c bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) · line 1330
Quoted 10 lines
xen/drivers/passthrough/pci.c +7
-3-printk ("%d " , msi->irq); -printk (">\n" ); - } +printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , + pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), +PCI_FUNC (pdev->devfn), + pdev->domain ? pdev->domain->domain_id :-1 , + (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); + list_for_each_entry ( msi, &pdev->msi_list, list ) +printk ("%d " , msi->irq);
I know you're just copying the existing code, but this would be better to strip the space after the <, and use " %d" here, which will drop the final space before the > in the eventual output.
Again, can be fixed on commit.
~Andrew
Quoted 20 lines
xen/drivers/passthrough/pci.c +2
-3+printk (">\n" );return 0 ; }@@ -1319,9 +1336,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)static void dump_pci_devices (unsigned char ch) {printk ("==== PCI devices ====\n" ); -pcidevs_lock (); -pci_segments_iterate (_dump_pci_devices,NULL ); -pcidevs_unlock (); +pci_pdevs_iterate (dump_pci_device,NULL ); }static int __init setup_dump_pcidevs (void )xen/include/xen/pci.h +0
-0
xen/include/xen/pci.h int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus); · line 160
Quoted 7 lines
xen/include/xen/pci.h +1
-0@@ -154,6 +154,7 @@ int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);struct pci_dev *pci_lock_pdev (int seg,int bus,int devfn);struct pci_dev *pci_lock_domain_pdev (struct domain *,int seg,int bus,int devfn); +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg);void setup_hwdom_pci_devices (struct domain *,int (*)(u8 devfn,struct pci_dev *));
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 2/4] pci: add all-device iterator function...
Quoted 83 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 16 July 2019 11:25
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v3 2/4] pci: add all-device iterator function...xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom
Quoted 42 lines · Paul Durrant
...and use it for
setup_hwdom_pci_devices()anddump_pci_devices().The unlock/process-pending-softirqs/lock sequence that was in
_setup_hwdom_pci_devices()is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case ofdump_pci_devices(), sincerun_all_nonirq_keyhandlers()will callprocess_pending_softirqs()before invoking each key handler anyway, but this is not performance critical code.The "==== segment XXXX ====" headline that was in
_dump_pci_devices()has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.Signed-off-by Paul Durrant <[email protected]>Reviewed-by "Roger Pau Monné" <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>v3:
- Addressed review comments from Roger.v2:
- New in v2.--- xen/drivers/passthrough/pci.c | 121 ++++++++++++++++++++++++------------------ xen/include/xen/pci.h | 1 + 2 files changed, 69 insertions(+), 53 deletions(-)xen/drivers/passthrough/pci.c +0
-0@@ -1134,64 +1134,87 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom*ctxt,
Quoted 16 lines
xen/drivers/passthrough/pci.c +7
-3ctxt->d->domain_id, err); } -static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg) +static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg) {struct setup_hwdom *ctxt = arg; -int bus, devfn; +struct domain *d = ctxt->d; -for ( bus =0 ; bus <256 ; bus++ ) +if ( !pdev->domain ) + { + pdev->domain = d; +list_add (&pdev->domain_list, &d->pdev_list); +setup_one_hwdom_device (ctxt, pdev);xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1155
Quoted 10 lines
xen/drivers/passthrough/pci.c +8
-1+ } +else if ( pdev->domain == dom_xen ) { -for ( devfn =0 ; devfn <256 ; devfn++ ) + pdev->domain = d; +setup_one_hwdom_device (ctxt, pdev); + pdev->domain = dom_xen; + } +else if ( pdev->domain != d ) + printk(XENLOG_WARNING "Dom%pd owning %04x:%02x:%02x.%u?\n",Just %pd. It takes care of rendering d%u or d[$name] for system domains.
Quoted 2 lines
Can be fixed on commit.
Ok.
Quoted 131 lines
Quoted 116 lines
xen/drivers/passthrough/pci.c +44
-43+ pdev->domain, pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), + PCI_FUNC(pdev->devfn)); + +return 0 ; +} + +struct psdi_ctxt { +int (*cb)(struct pci_dev *,void *); +void *arg; +}; + +static int pci_segment_devices_iterate (struct pci_seg *pseg,void *arg) +{ +struct psdi_ctxt *ctxt = arg; +unsigned int bus, devfn; +int rc =0 ; + +/* +* We don't iterate by walking pseg->alldevs_list here because that +* would make the pcidevs_unlock()/lock() sequence below unsafe. +*/ +for ( bus =0 ; !rc && bus <256 ; bus++ ) +for ( devfn =0 ; !rc && devfn <256 ; devfn++ ) {struct pci_dev *pdev =pci_get_pdev (pseg->nr, bus, devfn);if ( !pdev )continue ; -if ( !pdev->domain ) - { - pdev->domain = ctxt->d; -list_add (&pdev->domain_list, &ctxt->d->pdev_list); -setup_one_hwdom_device (ctxt, pdev); - } -else if ( pdev->domain == dom_xen ) - { - pdev->domain = ctxt->d; -setup_one_hwdom_device (ctxt, pdev); - pdev->domain = dom_xen; - } -else if ( pdev->domain != ctxt->d ) -printk (XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n" , - pdev->domain->domain_id, pseg->nr, bus, -PCI_SLOT (devfn),PCI_FUNC (devfn)); - -if ( iommu_verbose ) - { -pcidevs_unlock (); -process_pending_softirqs (); -pcidevs_lock (); - } - } + rc = ctxt->cb (pdev, ctxt->arg); -if ( !iommu_verbose ) - { +/* +* Err on the safe side and assume the callback has taken +* a significant amount of time. +*/ pcidevs_unlock ();process_pending_softirqs ();pcidevs_lock (); } - } -return 0 ; +return rc; +} + +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg) +{ +struct psdi_ctxt ctxt = { .cb = cb, .arg = arg }; +int rc; + +pcidevs_lock (); + rc =pci_segments_iterate (pci_segment_devices_iterate, &ctxt); +pcidevs_unlock (); + +return rc; } void __hwdom_init setup_hwdom_pci_devices(struct domain *d,int (*handler)(u8 devfn,struct pci_dev *)) {struct setup_hwdom ctxt = { .d = d, .handler = handler }; +int rc =pci_pdevs_iterate (setup_hwdom_pci_device, &ctxt); -pcidevs_lock (); -pci_segments_iterate (_setup_hwdom_pci_devices, &ctxt); -pcidevs_unlock (); +ASSERT (!rc); }#ifdef CONFIG_ACPI @@ -1294,24 +1317,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) }#endif -static int _dump_pci_devices (struct pci_seg *pseg,void *arg) +static int dump_pci_device (struct pci_dev *pdev,void *arg) { -struct pci_dev *pdev;struct msi_desc *msi; -printk ("==== segment %04x ====\n" , pseg->nr); - - list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) - { -printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , - pseg->nr, pdev->bus, -PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn), - pdev->domain ? pdev->domain->domain_id :-1 , - (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); -list_for_each_entry ( msi, &pdev->msi_list, list )xen/drivers/passthrough/pci.c bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) · line 1330
Quoted 10 lines
xen/drivers/passthrough/pci.c +7
-3-printk ("%d " , msi->irq); -printk (">\n" ); - } +printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , + pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), +PCI_FUNC (pdev->devfn), + pdev->domain ? pdev->domain->domain_id :-1 , + (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); + list_for_each_entry ( msi, &pdev->msi_list, list ) +printk ("%d " , msi->irq);I know you're just copying the existing code, but this would be better to strip the space after the <, and use " %d" here, which will drop the final space before the > in the eventual output.
Quoted 1 line
Again, can be fixed on commit.
If you prefer that then that's fine.
Paul
Quoted 28 lines
~Andrew
Quoted 25 lines
xen/drivers/passthrough/pci.c +2
-3+printk (">\n" );return 0 ; }@@ -1319,9 +1336,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)static void dump_pci_devices (unsigned char ch) {printk ("==== PCI devices ====\n" ); -pcidevs_lock (); -pci_segments_iterate (_dump_pci_devices,NULL ); -pcidevs_unlock (); +pci_pdevs_iterate (dump_pci_device,NULL ); }static int __init setup_dump_pcidevs (void )xen/include/xen/pci.h +1
-0@@ -154,6 +154,7 @@ int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);struct pci_dev *pci_lock_pdev (int seg,int bus,int devfn);struct pci_dev *pci_lock_domain_pdev (struct domain *,int seg,int bus,int devfn); +int pci_pdevs_iterate (int (*cb)(struct pci_dev *,void *),void *arg);
Quoted 2 lines
xen/include/xen/pci.h int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus); · line 160
Quoted 2 lines
xen/include/xen/pci.h +0
-0void setup_hwdom_pci_devices (struct domain *,int (*)(u8 devfn,struct pci_dev *));
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 2/4] pci: add all-device iterator function...
In reply to Paul DurrantQuoted 9 lines · Paul Durrant
...and use it for
setup_hwdom_pci_devices()anddump_pci_devices().The unlock/process-pending-softirqs/lock sequence that was in
_setup_hwdom_pci_devices()is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case ofdump_pci_devices(), sincerun_all_nonirq_keyhandlers()will callprocess_pending_softirqs()before invoking each key handler anyway, but this is not performance critical code.
Quoted 4 lines
The "==== segment XXXX ====" headline that was in
_dump_pci_devices()has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.
For overall output volume it would perhaps be better to have the headline and drop the redundancy on every line. But that's just a side note, not something I expect you to change.
Quoted 43 lines
xen/drivers/passthrough/pci.c +13
-24+static int pci_segment_devices_iterate (struct pci_seg *pseg,void *arg) +{ +struct psdi_ctxt *ctxt = arg; +unsigned int bus, devfn; +int rc =0 ; + +/* +* We don't iterate by walking pseg->alldevs_list here because that +* would make the pcidevs_unlock()/lock() sequence below unsafe. +*/ +for ( bus =0 ; !rc && bus <256 ; bus++ ) +for ( devfn =0 ; !rc && devfn <256 ; devfn++ ) {struct pci_dev *pdev =pci_get_pdev (pseg->nr, bus, devfn);if ( !pdev )continue ; -if ( !pdev->domain ) - { - pdev->domain = ctxt->d; -list_add (&pdev->domain_list, &ctxt->d->pdev_list); -setup_one_hwdom_device (ctxt, pdev); - } -else if ( pdev->domain == dom_xen ) - { - pdev->domain = ctxt->d; -setup_one_hwdom_device (ctxt, pdev); - pdev->domain = dom_xen; - } -else if ( pdev->domain != ctxt->d ) -printk (XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n" , - pdev->domain->domain_id, pseg->nr, bus, -PCI_SLOT (devfn),PCI_FUNC (devfn)); - -if ( iommu_verbose ) - { -pcidevs_unlock (); -process_pending_softirqs (); -pcidevs_lock (); - } - } + rc = ctxt->cb (pdev, ctxt->arg);
xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1193
Quoted 10 lines
xen/drivers/passthrough/pci.c +4
-2-if ( !iommu_verbose ) - { +/* +* Err on the safe side and assume the callback has taken +* a significant amount of time. +*/ pcidevs_unlock ();process_pending_softirqs ();pcidevs_lock ();
This behavior is not generally acceptable to an arbitrary caller. Therefore I think a prominent notice is needed at the top of the function.
Even worse, the latest post-boot and outside of debug-key handling this isn't safe at all: You'd have to re-check that pseg hasn't gone away (this can't happen right now, but isn't impossible in principle), and more generally that the pseg tree hasn't changed. Since this would be a little difficult to arrange, I think doing so would better be left to the callback, or be controlled by an extra argument passed to pci_pdevs_iterate() (in both cases eliminating the need for a prominent warning at the top of the function).
Quoted 24 lines
xen/drivers/passthrough/pci.c +2
-15@@ -1294,24 +1317,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) }#endif -static int _dump_pci_devices (struct pci_seg *pseg,void *arg) +static int dump_pci_device (struct pci_dev *pdev,void *arg) { -struct pci_dev *pdev;struct msi_desc *msi; -printk ("==== segment %04x ====\n" , pseg->nr); - - list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) - { -printk ("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < " , - pseg->nr, pdev->bus, -PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn), - pdev->domain ? pdev->domain->domain_id :-1 , - (pdev->node !=NUMA_NO_NODE ) ? pdev->node :-1 ); - list_for_each_entry ( msi, &pdev->msi_list, list ) -printk ("%d " , msi->irq); -printk (">\n" ); - } + printk("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < ",
xen/drivers/passthrough/pci.c bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) · line 1334
Quoted 10 lines
xen/drivers/passthrough/pci.c +7
-0+ pdev->seg, pdev->bus,PCI_SLOT (pdev->devfn), +PCI_FUNC (pdev->devfn), + pdev->domain ? pdev->domain->domain_id :-1 , + (pdev->node != NUMA_NO_NODE) ? pdev->node : -1); + list_for_each_entry ( msi, &pdev->msi_list, list ) +printk ("%d " , msi->irq); +printk (">\n" );return 0 ; }
Seeing this code I don't think it would be difficult to arrange for the head line to be logged whenever the segment changes. You don't use "arg" right now, after all, which could point to a local variable ...
xen/drivers/passthrough/pci.c static int _dump_pci_devices(struct pci_seg *pseg, void *arg) · line 1339
Quoted 8 lines
xen/drivers/passthrough/pci.c +1
-3@@ -1319,9 +1336,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)static void dump_pci_devices (unsigned char ch) {printk ("==== PCI devices ====\n" ); -pcidevs_lock (); -pci_segments_iterate (_dump_pci_devices,NULL ); -pcidevs_unlock (); +pci_pdevs_iterate (dump_pci_device,NULL );
... here, initialized to e.g. UINT_MAX.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 3/4 iommu: introduce iommu_groups
In reply to Paul DurrantSome devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.
This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
v3:
- Fix signed vs. unsigned issue spotted by Roger.
v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.
--- xen/drivers/passthrough/Makefile | 1 + xen/drivers/passthrough/groups.c | 91 +++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/x86/iommu.c | 8 +++- xen/include/xen/iommu.h | 7 +++ xen/include/xen/pci.h | 2 + 5 files changed, 108 insertions(+), 1 deletion(-) create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86 subdir-$(CONFIG_ARM) += arm obj-y += iommu.o +obj-$(CONFIG_HAS_PCI) += groups.o obj-$(CONFIG_HAS_PCI) += pci.o obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
xen/drivers/passthrough/groups.c new file +91 -0
91 lines not on this page · raw message
xen/drivers/passthrough/x86/iommu.c +7 -1
@@ -43,7 +43,13 @@ int __init iommu_hardware_setup(void)/* x2apic setup may have previously initialised the struct. */ ASSERT (iommu_ops.init == iommu_init_ops->ops->init); -return iommu_init_ops->setup (); + rc = iommu_init_ops->setup (); +if ( rc ) +return rc; + +iommu_groups_init (); + +return pci_pdevs_iterate (iommu_group_assign,NULL ); }int iommu_enable_x2apic (void )
xen/include/xen/iommu.h +7 -0
@@ -317,6 +317,13 @@ DECLARE_PER_CPU(bool_t, iommu_dont_flush_iotlb);extern struct spinlock iommu_pt_cleanup_lock;extern struct page_list_head iommu_pt_cleanup_list; +#ifdef CONFIG_HAS_PCI + +void iommu_groups_init (void ); +int iommu_group_assign (struct pci_dev *pdev,void *arg); + +#endif /* CONFIG_HAS_PCI */ +#endif /* _IOMMU_H_ */ /*
xen/include/xen/pci.h +2 -0
@@ -79,6 +79,8 @@ struct pci_dev {struct list_head alldevs_list;struct list_head domain_list; +struct iommu_group *grp; +struct list_head msi_list;struct arch_msix *msix;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups
Quoted 9 lines · Paul Durrant
xen/drivers/passthrough/groups.c new file +9
-0+int iommu_group_assign (struct pci_dev *pdev,void *arg) +{ +const struct iommu_ops *ops =iommu_get_ops (); +int id; +struct iommu_group *grp; + +if ( !ops->get_device_group_id ) +return 0 ; +
xen/drivers/passthrough/groups.c line 75
Quoted 10 lines
xen/drivers/passthrough/groups.c new file +10
-0+ id = ops->get_device_group_id (pdev->seg, pdev->bus, pdev->devfn); +if ( id <0 ) +return -ENODATA ; + + grp =get_iommu_group (id); +if ( !grp ) +return -ENOMEM ; + +if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n",
No unadorned hex numbers please. This is a recipe for confusion during debugging.
Either %#x, or %u, and needs to be fixed on commit if we go with that route.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups
Quoted 24 lines · Andrew Cooper
Quoted 9 lines · Paul Durrant
xen/drivers/passthrough/groups.c new file +9
-0+int iommu_group_assign (struct pci_dev *pdev,void *arg) +{ +const struct iommu_ops *ops =iommu_get_ops (); +int id; +struct iommu_group *grp; + +if ( !ops->get_device_group_id ) +return 0 ; +xen/drivers/passthrough/groups.c line 75
Quoted 10 lines
xen/drivers/passthrough/groups.c new file +10
-0+ id = ops->get_device_group_id (pdev->seg, pdev->bus, pdev->devfn); +if ( id <0 ) +return -ENODATA ; + + grp =get_iommu_group (id); +if ( !grp ) +return -ENOMEM ; + +if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n",No unadorned hex numbers please. This is a recipe for confusion during debugging.
Quoted 1 line
Either %#x, or %u, and needs to be fixed on commit if we go with that route.
I assume (hope) that you mean this for the right most number only; it's not entirely unambiguous from your reply.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups
Quoted 24 lines · Jan Beulich
Quoted 23 lines · Andrew Cooper
Quoted 9 lines · Paul Durrant
xen/drivers/passthrough/groups.c new file +9
-0+int iommu_group_assign (struct pci_dev *pdev,void *arg) +{ +const struct iommu_ops *ops =iommu_get_ops (); +int id; +struct iommu_group *grp; + +if ( !ops->get_device_group_id ) +return 0 ; +xen/drivers/passthrough/groups.c line 75
Quoted 10 lines
xen/drivers/passthrough/groups.c new file +10
-0+ id = ops->get_device_group_id (pdev->seg, pdev->bus, pdev->devfn); +if ( id <0 ) +return -ENODATA ; + + grp =get_iommu_group (id); +if ( !grp ) +return -ENOMEM ; + +if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n",No unadorned hex numbers please. This is a recipe for confusion during debugging.
Quoted 3 lines
Quoted 1 line
Either %#x, or %u, and needs to be fixed on commit if we go with that route.
I assume (hope) that you mean this for the right most number only; it's not entirely unambiguous from your reply.
Oops yes - I only meant the IOMMU group formatter.
The PCI coordinates should get subsumed by %pp in fairly short order.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups
Quoted 25 lines · Andrew Cooper
Quoted 24 lines · Jan Beulich
Quoted 23 lines · Andrew Cooper
Quoted 9 lines · Paul Durrant
xen/drivers/passthrough/groups.c new file +9
-0+int iommu_group_assign (struct pci_dev *pdev,void *arg) +{ +const struct iommu_ops *ops =iommu_get_ops (); +int id; +struct iommu_group *grp; + +if ( !ops->get_device_group_id ) +return 0 ; +xen/drivers/passthrough/groups.c line 75
Quoted 10 lines
xen/drivers/passthrough/groups.c new file +10
-0+ id = ops->get_device_group_id (pdev->seg, pdev->bus, pdev->devfn); +if ( id <0 ) +return -ENODATA ; + + grp =get_iommu_group (id); +if ( !grp ) +return -ENOMEM ; + +if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n",No unadorned hex numbers please. This is a recipe for confusion during debugging.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Either %#x, or %u, and needs to be fixed on commit if we go with that route.
I assume (hope) that you mean this for the right most number only; it's not entirely unambiguous from your reply.
Oops yes - I only meant the IOMMU group formatter.
The PCI coordinates should get subsumed by %pp in fairly short order.
%op I (still) hope ;-)
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups
In reply to Paul Durrantxen/drivers/passthrough/Makefile subdir-$(CONFIG_X86) += x86 · line 7
Quoted 7 lines · Paul Durrant
xen/drivers/passthrough/Makefile +1
-0@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86 subdir-$(CONFIG_ARM) += arm obj-y += iommu.o +obj-$(CONFIG_HAS_PCI) += groups.o
I assume this dependency on PCI is temporary, as there's nothing inherently tying grouping of devices to PCI (afaict)?
xen/drivers/passthrough/Makefile subdir-$(CONFIG_X86) += x86
Quoted 8 lines
xen/drivers/passthrough/Makefile +8
-0+int iommu_group_assign(struct pci_dev *pdev, void *arg) +{ + const struct iommu_ops *ops = iommu_get_ops(); + int id; + struct iommu_group *grp; + + if ( !ops->get_device_group_id ) + return 0;
With you making groups mandatory (i.e. even solitary devices getting put in a group), shouldn't this be -EOPNOTSUPP, maybe accompanied by ASSERT_UNREACHABLE()?
xen/drivers/passthrough/Makefile
Quoted 12 lines
xen/drivers/passthrough/Makefile +12
-0+ id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn); + if ( id < 0 ) + return -ENODATA; + + grp = get_iommu_group(id); + if ( !grp ) + return -ENOMEM; + + if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n", + pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), grp->id);
I'm not overly happy about this new logging: On modern systems a debug level run is already rather verbose about PCI devices, simply because there are so many. If my hope to not see individual devices put in groups is not going to be fulfilled, can we at least try to come to some agreement that certain devices which can't sensibly be passed through won't be assigned groups (and hence won't produce output here)? A group-less device then would automatically be unable to have its owner changed.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 4/4 iommu / pci: re-implement XEN_DOMCTL_get_device_group...
In reply to Paul Durrant... using the new iommu_group infrastructure.
Because 'sibling' devices are now members of the same iommu_group, implement the domctl by looking up the iommu_group of the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.
---
Cc: Jan Beulich <[email protected]>
v3:
- Make 'max_sdevs' parameter in iommu_get_device_group() unsigned.
- Add missing check of max_sdevs to avoid buffer overflow.
v2:
- Re-implement in the absence of a per-group devs list.
- Make use of pci_sbdf_t.
--- xen/drivers/passthrough/groups.c | 46 ++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 51 ++-------------------------------------- xen/include/xen/iommu.h | 3 +++ 3 files changed, 51 insertions(+), 49 deletions(-)
xen/drivers/passthrough/groups.c +46 -0
@@ -12,8 +12,12 @@* GNU General Public License for more details. */ +#include <xen/guest_access.h> #include <xen/iommu.h> +#include <xen/pci.h> #include <xen/radix-tree.h> +#include <xen/sched.h> +#include <xsm/xsm.h> struct iommu_group {unsigned int id;
@@ -81,6 +85,48 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)return 0 ; } +int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs) +{ +struct iommu_group *grp =NULL ; +struct pci_dev *pdev; +unsigned int i =0 ; + +pcidevs_lock (); + + for_each_pdev ( d, pdev ) + { +if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; +break ; + } + } + +if ( !grp ) + goto out; + + for_each_pdev ( d, pdev ) + { +if (xsm_get_device_group (XSM_HOOK , pdev->sbdf.sbdf) || + pdev->grp != grp ) +continue ; + +if ( i < max_sdevs && +unlikely (copy_to_guest_offset (buf, i++, &pdev->sbdf.sbdf,1 )) ) + { +pcidevs_unlock (); +return -EFAULT ; + } + } + + out: +pcidevs_unlock (); + +return i; +} +/* * Local variables: * mode: C
xen/drivers/passthrough/pci.c +2 -49
66 lines not on this page · raw message
xen/include/xen/iommu.h +3 -0
@@ -321,6 +321,9 @@ extern struct page_list_head iommu_pt_cleanup_list;void iommu_groups_init (void );int iommu_group_assign (struct pci_dev *pdev,void *arg); +int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs);#endif /* CONFIG_HAS_PCI */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...
Quoted 64 lines · Paul Durrant
... using the new
iommu_groupinfrastructure.Because 'sibling' devices are now members of the same
iommu_group, implement the domctl by looking up theiommu_groupof the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>v3:
- Make 'max_sdevs' parameter iniommu_get_device_group()unsigned.
- Add missing check ofmax_sdevsto avoid buffer overflow.v2:
- Re-implement in the absence of a per-group devs list.
- Make use ofpci_sbdf_t.--- xen/drivers/passthrough/groups.c | 46 ++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 51 ++-------------------------------------- xen/include/xen/iommu.h | 3 +++ 3 files changed, 51 insertions(+), 49 deletions(-)xen/drivers/passthrough/groups.c +23
-0@@ -12,8 +12,12 @@* GNU General Public License for more details. */ +#include <xen/guest_access.h> #include <xen/iommu.h> +#include <xen/pci.h> #include <xen/radix-tree.h> +#include <xen/sched.h> +#include <xsm/xsm.h> struct iommu_group {unsigned int id;@@ -81,6 +85,48 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)return 0 ; } +int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs) +{ +struct iommu_group *grp =NULL ; +struct pci_dev *pdev; +unsigned int i =0 ; + +pcidevs_lock (); + + for_each_pdev ( d, pdev ) + { +if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; +break ; + } + } +
xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 116
Quoted 10 lines
xen/drivers/passthrough/groups.c +10
-0+if ( !grp ) + goto out; + + for_each_pdev ( d, pdev ) + { +if (xsm_get_device_group (XSM_HOOK , pdev->sbdf.sbdf) || + pdev->grp != grp ) +continue ; + + if ( i < max_sdevs &&
AFAICT you are adding the check here in order to keep current behaviour?
But isn't it wrong to not report to the caller that the buffer was smaller than required, and that the returned result is partial?
I don't see any way a caller can differentiate between a result that uses the full buffer and one that's actually partial due to smaller than required buffer provided. I think this function should return -ENOSPC for such case.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...
Quoted 77 lines
-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 16 July 2019 12:28
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implementXEN_DOMCTL_get_device_group...Quoted 69 lines · Paul Durrant
... using the new
iommu_groupinfrastructure.Because 'sibling' devices are now members of the same
iommu_group, implement the domctl by looking up theiommu_groupof the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>v3:
- Make 'max_sdevs' parameter iniommu_get_device_group()unsigned.
- Add missing check ofmax_sdevsto avoid buffer overflow.v2:
- Re-implement in the absence of a per-group devs list.
- Make use ofpci_sbdf_t.--- xen/drivers/passthrough/groups.c | 46 ++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 51 ++-------------------------------------- xen/include/xen/iommu.h | 3 +++ 3 files changed, 51 insertions(+), 49 deletions(-)xen/drivers/passthrough/groups.c +28
-0@@ -12,8 +12,12 @@* GNU General Public License for more details. */ +#include <xen/guest_access.h> #include <xen/iommu.h> +#include <xen/pci.h> #include <xen/radix-tree.h> +#include <xen/sched.h> +#include <xsm/xsm.h> struct iommu_group {unsigned int id;@@ -81,6 +85,48 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)return 0 ; } +int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs) +{ +struct iommu_group *grp =NULL ; +struct pci_dev *pdev; +unsigned int i =0 ; + +pcidevs_lock (); + + for_each_pdev ( d, pdev ) + { +if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; +break ; + } + } + +if ( !grp ) + goto out; + + for_each_pdev ( d, pdev ) + {
Quoted 8 lines
xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 116
Quoted 5 lines
xen/drivers/passthrough/groups.c +5
-0+if (xsm_get_device_group (XSM_HOOK , pdev->sbdf.sbdf) || + pdev->grp != grp ) +continue ; + + if ( i < max_sdevs &&AFAICT you are adding the check here in order to keep current behaviour?
Yes.
Quoted 2 lines
But isn't it wrong to not report to the caller that the buffer was smaller than required, and that the returned result is partial?
Given that there is zero documentation I think your guess is as good as mine as to what intention of the implementor was.
Quoted 5 lines
I don't see any way a caller can differentiate between a result that uses the full buffer and one that's actually partial due to smaller than required buffer provided. I think this function should return
-ENOSPCfor such case.
I'd prefer to stick to the principle of no change in behaviour. TBH I have not found any caller of xc_get_device_group() apart from a python binding and who knows what piece of antiquated code might sit on the other side of that. FWIW that code sets max_sdevs to 1024 so it's unlikely to run out of space so an ENOSPC might be ok. Still, I'd like to hear maintainer opinions on this.
Paul
Quoted 2 lines
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...
Quoted 55 lines · Paul Durrant
Quoted 32 lines
From: Roger Pau Monne <[email protected]>
Sent: 16 July 2019 12:28Quoted 28 lines · Paul Durrant
xen/drivers/passthrough/groups.c +24
-0@@ -81,6 +85,48 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)return 0 ; } +int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs) +{ +struct iommu_group *grp =NULL ; +struct pci_dev *pdev; +unsigned int i =0 ; + +pcidevs_lock (); + + for_each_pdev ( d, pdev ) + { +if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; +break ; + } + } + +if ( !grp ) + goto out; + + for_each_pdev ( d, pdev ) + {Quoted 8 lines
xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 116
Quoted 5 lines
xen/drivers/passthrough/groups.c +5
-0+if (xsm_get_device_group (XSM_HOOK , pdev->sbdf.sbdf) || + pdev->grp != grp ) +continue ; + + if ( i < max_sdevs &&AFAICT you are adding the check here in order to keep current behaviour?
Yes.
Quoted 2 lines
But isn't it wrong to not report to the caller that the buffer was smaller than required, and that the returned result is partial?
Given that there is zero documentation I think your guess is as good as mine as to what intention of the implementor was.
Quoted 5 lines
I don't see any way a caller can differentiate between a result that uses the full buffer and one that's actually partial due to smaller than required buffer provided. I think this function should return
-ENOSPCfor such case.
Quoted 6 lines
I'd prefer to stick to the principle of no change in behaviour. TBH I have not found any caller of
xc_get_device_group()apart from a python binding and who knows what piece of antiquated code might sit on the other side of that. FWIW that code setsmax_sdevsto 1024 so it's unlikely to run out of space so an ENOSPC might be ok. Still, I'd like to hear maintainer opinions on this.
How about we try to find a sufficiently backwards compatible solution which still allows recognizing insufficient buffer space? First of all the common null-handle approach could be used to get a total count. There's not much risk of this value getting stale between two successive domctl-s. And then returning -ENOBUFS upon exceeding the provided buffer shouldn't really break pre-existing code: It surely would misbehave anyway if the group was larger than what they think it is.
Another option would be to isolate all of this compatibility stuff into the libxc wrapper, making the hypercall itself return the actual count irrespective of the passed in buffer size (i.e. a generalization of the null-handle model).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...
In reply to Paul DurrantQuoted 20 lines · Paul Durrant
xen/drivers/passthrough/groups.c +20
-0+int iommu_get_device_group (struct domain *d,pci_sbdf_t sbdf, +XEN_GUEST_HANDLE_64 (uint32) buf, +unsigned int max_sdevs) +{ +struct iommu_group *grp =NULL ; +struct pci_dev *pdev; +unsigned int i =0 ; + +pcidevs_lock (); + + for_each_pdev ( d, pdev ) + { +if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; +break ; + } + } + +if ( !grp )
xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 117
Quoted 10 lines
xen/drivers/passthrough/groups.c +10
-0+ goto out; + + for_each_pdev ( d, pdev ) + { +if (xsm_get_device_group (XSM_HOOK , pdev->sbdf.sbdf) || + pdev->grp != grp ) +continue ; + +if ( i < max_sdevs && +unlikely (copy_to_guest_offset (buf, i++, &pdev->sbdf.sbdf,1 )) )
If you want to avoid breaking existing callers, you'll have to mimic here ...
Quoted 24 lines
xen/drivers/passthrough/pci.c +0
-24-static int iommu_get_device_group ( -struct domain *d,u16 seg,u8 bus,u8 devfn, -XEN_GUEST_HANDLE_64 (uint32) buf,int max_sdevs) -{ -const struct domain_iommu *hd =dom_iommu (d); -struct pci_dev *pdev; -int group_id, sdev_id; -u32 bdf; -int i =0 ; -const struct iommu_ops *ops = hd->platform_ops; - -if ( !iommu_enabled || !ops || !ops->get_device_group_id ) -return 0 ; - - group_id = ops->get_device_group_id (seg, bus, devfn); - -pcidevs_lock (); -for_each_pdev ( d, pdev ) - { -if ( (pdev->seg != seg) || - ((pdev->bus == bus) && (pdev->devfn == devfn)) ) -continue ; - -if (xsm_get_device_group (XSM_HOOK , (seg <<16 ) | (pdev->bus <<8 ) | pdev->devfn) )
xen/drivers/passthrough/pci.c int deassign_device(struct domain *d, u16 seg, u8 bus, u8 devfn) · line 1600
Quoted 10 lines
xen/drivers/passthrough/pci.c +0
-10-continue ; - - sdev_id = ops->get_device_group_id (seg, pdev->bus, pdev->devfn); -if ( (sdev_id == group_id) && (i < max_sdevs) ) - { - bdf =0 ; - bdf |= (pdev->bus &0xff ) <<16 ; - bdf |= (pdev->devfn &0xff ) <<8 ; - -if (unlikely (copy_to_guest_offset (buf, i, &bdf,1 )) )
... this rather odd organization of BDF. Omitting the segment is, I think, fine, as I don't expect groups to extend past segment boundaries (and iirc neither Intel's nor AMD's implementation have any means for this to happen).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel