Skip to contentSkip to conversation

iommu groups + cleanup

PATCH Superseded Paul Durrant 3 participants 16 messages Jul 15, 2019 – Jul 16, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover iommu groups + cleanup

This series is a mixture of tidying and some preparatory work for grouping PCI devices for the purposes of assignment.

Paul Durrant (4):
iommu / x86: move call to scan_pci_devices() out of vendor code
pci: add all-device iterator function...
iommu: introduce iommu_groups
iommu / pci: re-implement XEN_DOMCTL_get_device_group...

 xen/drivers/passthrough/Makefile            |   1 +
 xen/drivers/passthrough/amd/pci_amd_iommu.c |   3 +-
 xen/drivers/passthrough/groups.c            | 135 ++++++++++++++++++++++
 xen/drivers/passthrough/pci.c               | 171 +++++++++++-----------------
 xen/drivers/passthrough/vtd/iommu.c         |   4 -
 xen/drivers/passthrough/x86/iommu.c         |  14 ++-
 xen/include/xen/iommu.h                     |   9 ++
 xen/include/xen/pci.h                       |   3 +
 8 files changed, 232 insertions(+), 108 deletions(-)
 create mode 100644 xen/drivers/passthrough/groups.c

---
v2:
- Drop iommu_get_ops() move and add all-device iterator

Cc Andrew Cooper <[email protected]>
Cc Brian Woods <[email protected]>
Cc George Dunlap <[email protected]>
Cc Ian Jackson <[email protected]>
Cc Jan Beulich <[email protected]>
Cc Julien Grall <[email protected]>
Cc Kevin Tian <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc "Roger Pau Monné" <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Tim Deegan <[email protected]>
Cc Wei Liu <[email protected]>
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v2 1/4 iommu / x86: move call to scan_pci_devices() out of vendor code

It's not vendor specific so it doesn't really belong there.

Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused.

Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.

---
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Brian Woods <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Expanded commit comment.
- Moved PCI scan to before IOMMU initialization, rather than after it.

---
 xen/drivers/passthrough/amd/pci_amd_iommu.c | 3 ++-
 xen/drivers/passthrough/vtd/iommu.c         | 4 ----
 xen/drivers/passthrough/x86/iommu.c         | 6 ++++++
 3 files changed, 8 insertions(+), 5 deletions(-)
xen/drivers/passthrough/amd/pci_amd_iommu.c +2 -1
index 4afbcd1609..3338a8e0e8 100644
@@ -180,7 +180,8 @@ static int __init iov_detect(void)
 
     if ( !amd_iommu_perdev_intremap )
         printk(XENLOG_WARNING "AMD-Vi: Using global interrupt remap table is not recommended (see XSA-36)!\n");
-    return scan_pci_devices();
+
+    return 0;
 }
 
 int amd_iommu_alloc_root(struct domain_iommu *hd)
xen/drivers/passthrough/vtd/iommu.c +0 -4
index 8b27d7e775..b0e3bf26b5 100644
@@ -2372,10 +2372,6 @@ static int __init vtd_setup(void)
     P(iommu_hap_pt_share, "Shared EPT tables");
 #undef P
 
-    ret = scan_pci_devices();
-    if ( ret )
-        goto error;
-
     ret = init_vtd_hw();
     if ( ret )
         goto error;
xen/drivers/passthrough/x86/iommu.c +6 -0
index 0fa6dcc3fd..a7438c9c25 100644
@@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops;
 
 int __init iommu_hardware_setup(void)
 {
+    int rc;
+
     if ( !iommu_init_ops )
         return -ENODEV;
 
+    rc = scan_pci_devices();
+    if ( rc )
+        return rc;
+
     if ( !iommu_ops.init )
         iommu_ops = *iommu_init_ops->ops;
     else
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v2 1/4
#2 lore raw

Re: [Xen-devel] [PATCH v2 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code

Quoted 11 lines · Paul Durrant

It's not vendor specific so it doesn't really belong there.

Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused.

Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.

I would even consider moving the call to scan_pci_devices into pci_segments_init instead of doing it in the IOMMU code, as you suggest above.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH v2 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code

Quoted 17 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 15:39
To: Paul Durrant <[email protected]>
Cc: [email protected]; Suravee Suthikulpanit <[email protected]>; Brian Woods <[email protected]>; Kevin Tian <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v2 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code

Quoted 7 lines · Paul Durrant

It's not vendor specific so it doesn't really belong there.

Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused.

Quoted 7 lines
Quoted 4 lines

Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.

Thanks.

Quoted 3 lines

I would even consider moving the call to scan_pci_devices into pci_segments_init instead of doing it in the IOMMU code, as you suggest above.

Possibly, although without an IOMMU in the system I don't think there is currently a lot of point in populating the pdev list.

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by on v2 1/4
#4 lore raw

Re: [Xen-devel] [PATCH v2 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code

In reply to Paul Durrant
Quoted 9 lines · Paul Durrant

It's not vendor specific so it doesn't really belong there.

Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused.

Quoted 5 lines

Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization.

Quoted 70 lines

---
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Brian Woods <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Expanded commit comment.
- Moved PCI scan to before IOMMU initialization, rather than after it.
---

xen/drivers/passthrough/amd/pci_amd_iommu.c | 3 ++-
xen/drivers/passthrough/vtd/iommu.c         | 4 ----
xen/drivers/passthrough/x86/iommu.c         | 6 ++++++
3 files changed, 8 insertions(+), 5 deletions(-)
xen/drivers/passthrough/amd/pci_amd_iommu.c +2 -1
index 4afbcd1609..3338a8e0e8 100644
@@ -180,7 +180,8 @@ static int __init iov_detect(void)
 
    if ( !amd_iommu_perdev_intremap )
        printk(XENLOG_WARNING "AMD-Vi: Using global interrupt remap table is not recommended (see XSA-36)!\n");
-    return scan_pci_devices();
+
+    return 0;
 }
 
 int amd_iommu_alloc_root(struct domain_iommu *hd)
xen/drivers/passthrough/vtd/iommu.c +0 -4
index 8b27d7e775..b0e3bf26b5 100644
@@ -2372,10 +2372,6 @@ static int __init vtd_setup(void)
    P(iommu_hap_pt_share, "Shared EPT tables");
 #undef P
 
-    ret = scan_pci_devices();
-    if ( ret )
-        goto error;
-
    ret = init_vtd_hw();
    if ( ret )
        goto error;
xen/drivers/passthrough/x86/iommu.c +6 -1
index 0fa6dcc3fd..a7438c9c25 100644
@@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops;
 
 int __init iommu_hardware_setup(void)
 {
+    int rc;
+
    if ( !iommu_init_ops )
        return -ENODEV;
 
+    rc = scan_pci_devices();
+    if ( rc )
+        return rc;
+
    if ( !iommu_ops.init )
        iommu_ops = *iommu_init_ops->ops;
    else
--
Quoted 1 line

2.11.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

PATCH v2 2/4 pci: add all-device iterator function...

In reply to Paul Durrant

...and use it for setup_hwdom_pci_devices() and dump_pci_devices().

The unlock/process-pending-softirqs/lock sequence that was in _setup_hwdom_pci_devices() is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case of dump_pci_devices(), since run_all_nonirq_keyhandlers() will call process_pending_softirqs() before invoking each key handler anyway, but this is not performance critical code.

The "==== segment XXXX ====" headline that was in _dump_pci_devices() has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- New in v2.

---
 xen/drivers/passthrough/pci.c | 120 +++++++++++++++++++++++-------------------
 xen/include/xen/pci.h         |   1 +
 2 files changed, 68 insertions(+), 53 deletions(-)
xen/drivers/passthrough/pci.c +67 -53
index e88689425d..179cb7e17e 100644
@@ -1134,54 +1134,78 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
                ctxt->d->domain_id, err);
 }
 
-static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg)
+static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg)
 {
     struct setup_hwdom *ctxt = arg;
-    int bus, devfn;
+    struct domain *d = ctxt->d;
 
-    for ( bus = 0; bus < 256; bus++ )
+    if ( !pdev->domain )
     {
-        for ( devfn = 0; devfn < 256; devfn++ )
+        pdev->domain = d;
+        list_add(&pdev->domain_list, &d->pdev_list);
+        setup_one_hwdom_device(ctxt, pdev);
+    }
+    else if ( pdev->domain == dom_xen )
+    {
+        pdev->domain = d;
+        setup_one_hwdom_device(ctxt, pdev);
+        pdev->domain = dom_xen;
+    }
+    else if ( pdev->domain != d )
+        printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
+               pdev->domain->domain_id, pdev->seg, pdev->bus,
+               PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
+
+    return 0;
+}
+
+struct psdi_ctxt {
+    int (*cb)(struct pci_dev *, void *);
+    void *arg;
+};
+
+static int pci_segment_devices_iterate(struct pci_seg *pseg, void *arg)
+{
+    struct psdi_ctxt *ctxt = arg;
+    int bus, devfn;
+    int rc = 0;
+
+    /*
+     * We don't iterate by walking pseg->alldevs_list here because that
+     * would make the pcidevs_unlock()/lock() sequence below unsafe.
+     */
+    for ( bus = 0; !rc && bus < 256; bus++ )
+        for ( devfn = 0; !rc && devfn < 256; devfn++ )
         {
             struct pci_dev *pdev = pci_get_pdev(pseg->nr, bus, devfn);
 
             if ( !pdev )
                 continue;
 
-            if ( !pdev->domain )
-            {
-                pdev->domain = ctxt->d;
-                list_add(&pdev->domain_list, &ctxt->d->pdev_list);
-                setup_one_hwdom_device(ctxt, pdev);
-            }
-            else if ( pdev->domain == dom_xen )
-            {
-                pdev->domain = ctxt->d;
-                setup_one_hwdom_device(ctxt, pdev);
-                pdev->domain = dom_xen;
-            }
-            else if ( pdev->domain != ctxt->d )
-                printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
-                       pdev->domain->domain_id, pseg->nr, bus,
-                       PCI_SLOT(devfn), PCI_FUNC(devfn));
+            rc = ctxt->cb(pdev, ctxt->arg);
 
-            if ( iommu_verbose )
-            {
-                pcidevs_unlock();
-                process_pending_softirqs();
-                pcidevs_lock();
-            }
-        }
-
-        if ( !iommu_verbose )
-        {
+            /*
+             * Err on the safe side and assume the callback has taken
+             * a significant amount of time.
+             */
             pcidevs_unlock();
             process_pending_softirqs();
             pcidevs_lock();
         }
-    }
 
-    return 0;
+    return rc;
+}
+
+int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg)
+{
+    struct psdi_ctxt ctxt = { .cb = cb, .arg = arg };
+    int rc;
+
+    pcidevs_lock();
+    rc = pci_segments_iterate(pci_segment_devices_iterate, &ctxt);
+    pcidevs_unlock();
+
+    return rc;
 }
 
 void __hwdom_init setup_hwdom_pci_devices(
@@ -1189,9 +1213,7 @@ void __hwdom_init setup_hwdom_pci_devices(
 {
     struct setup_hwdom ctxt = { .d = d, .handler = handler };
 
-    pcidevs_lock();
-    pci_segments_iterate(_setup_hwdom_pci_devices, &ctxt);
-    pcidevs_unlock();
+    pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt);
 }
 
 #ifdef CONFIG_ACPI
@@ -1294,24 +1316,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev)
 }
 #endif
 
-static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
+static int dump_pci_device(struct pci_dev *pdev, void *arg)
 {
-    struct pci_dev *pdev;
     struct msi_desc *msi;
 
-    printk("==== segment %04x ====\n", pseg->nr);
-
-    list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
-    {
-        printk("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < ",
-               pseg->nr, pdev->bus,
-               PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),
-               pdev->domain ? pdev->domain->domain_id : -1,
-               (pdev->node != NUMA_NO_NODE) ? pdev->node : -1);
-        list_for_each_entry ( msi, &pdev->msi_list, list )
-               printk("%d ", msi->irq);
-        printk(">\n");
-    }
+    printk("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < ",
+           pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
+           PCI_FUNC(pdev->devfn),
+           pdev->domain ? pdev->domain->domain_id : -1,
+           (pdev->node != NUMA_NO_NODE) ? pdev->node : -1);
+    list_for_each_entry ( msi, &pdev->msi_list, list )
+        printk("%d ", msi->irq);
+    printk(">\n");
 
     return 0;
 }
@@ -1319,9 +1335,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 static void dump_pci_devices(unsigned char ch)
 {
     printk("==== PCI devices ====\n");
-    pcidevs_lock();
-    pci_segments_iterate(_dump_pci_devices, NULL);
-    pcidevs_unlock();
+    pci_pdevs_iterate(dump_pci_device, NULL);
 }
 
 static int __init setup_dump_pcidevs(void)
xen/include/xen/pci.h +1 -0
index 04a9f46cc3..79eb25417b 100644
@@ -154,6 +154,7 @@ int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);
 struct pci_dev *pci_lock_pdev(int seg, int bus, int devfn);
 struct pci_dev *pci_lock_domain_pdev(
     struct domain *, int seg, int bus, int devfn);
+int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg);
 
 void setup_hwdom_pci_devices(struct domain *,
                             int (*)(u8 devfn, struct pci_dev *));
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v2 2/4
#6 lore raw

Re: [Xen-devel] [PATCH v2 2/4] pci: add all-device iterator function...

Quoted 9 lines · Paul Durrant

...and use it for setup_hwdom_pci_devices() and dump_pci_devices().

The unlock/process-pending-softirqs/lock sequence that was in _setup_hwdom_pci_devices() is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case of dump_pci_devices(), since run_all_nonirq_keyhandlers() will call process_pending_softirqs() before invoking each key handler anyway, but this is not performance critical code.

Quoted 6 lines

The "==== segment XXXX ====" headline that was in _dump_pci_devices() has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.

Just some trivial comments.

Thanks.

Quoted 40 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- New in v2.

---
 xen/drivers/passthrough/pci.c | 120 +++++++++++++++++++++++-------------------
 xen/include/xen/pci.h         |   1 +
 2 files changed, 68 insertions(+), 53 deletions(-)
xen/drivers/passthrough/pci.c +6 -4
index e88689425d..179cb7e17e 100644
@@ -1134,54 +1134,78 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
                ctxt->d->domain_id, err);
 }
 
-static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg)
+static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg)
 {
     struct setup_hwdom *ctxt = arg;
-    int bus, devfn;
+    struct domain *d = ctxt->d;
 
-    for ( bus = 0; bus < 256; bus++ )
+    if ( !pdev->domain )
     {
-        for ( devfn = 0; devfn < 256; devfn++ )
+        pdev->domain = d;
+        list_add(&pdev->domain_list, &d->pdev_list);
+        setup_one_hwdom_device(ctxt, pdev);

xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1156

Quoted 10 lines
xen/drivers/passthrough/pci.c +10 -0
+    }
+    else if ( pdev->domain == dom_xen )
+    {
+        pdev->domain = d;
+        setup_one_hwdom_device(ctxt, pdev);
+        pdev->domain = dom_xen;
+    }
+    else if ( pdev->domain != d )
+        printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
+               pdev->domain->domain_id, pdev->seg, pdev->bus,

You can use %pd here to print the domain.

Quoted 4 lines
xen/drivers/passthrough/pci.c +4 -0
+               PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
+
+    return 0;
+}

xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1170

Quoted 10 lines
xen/drivers/passthrough/pci.c +10 -0
+
+struct psdi_ctxt {
+    int (*cb)(struct pci_dev *, void *);
+    void *arg;
+};
+
+static int pci_segment_devices_iterate(struct pci_seg *pseg, void *arg)
+{
+    struct psdi_ctxt *ctxt = arg;
+    int bus, devfn;

unsigned for both the above.

Quoted 70 lines
xen/drivers/passthrough/pci.c +26 -28
+    int rc = 0;
+
+    /*
+     * We don't iterate by walking pseg->alldevs_list here because that
+     * would make the pcidevs_unlock()/lock() sequence below unsafe.
+     */
+    for ( bus = 0; !rc && bus < 256; bus++ )
+        for ( devfn = 0; !rc && devfn < 256; devfn++ )
         {
             struct pci_dev *pdev = pci_get_pdev(pseg->nr, bus, devfn);
 
             if ( !pdev )
                 continue;
 
-            if ( !pdev->domain )
-            {
-                pdev->domain = ctxt->d;
-                list_add(&pdev->domain_list, &ctxt->d->pdev_list);
-                setup_one_hwdom_device(ctxt, pdev);
-            }
-            else if ( pdev->domain == dom_xen )
-            {
-                pdev->domain = ctxt->d;
-                setup_one_hwdom_device(ctxt, pdev);
-                pdev->domain = dom_xen;
-            }
-            else if ( pdev->domain != ctxt->d )
-                printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
-                       pdev->domain->domain_id, pseg->nr, bus,
-                       PCI_SLOT(devfn), PCI_FUNC(devfn));
+            rc = ctxt->cb(pdev, ctxt->arg);
 
-            if ( iommu_verbose )
-            {
-                pcidevs_unlock();
-                process_pending_softirqs();
-                pcidevs_lock();
-            }
-        }
-
-        if ( !iommu_verbose )
-        {
+            /*
+             * Err on the safe side and assume the callback has taken
+             * a significant amount of time.
+             */
             pcidevs_unlock();
             process_pending_softirqs();
             pcidevs_lock();
         }
-    }
 
-    return 0;
+    return rc;
+}
+
+int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg)
+{
+    struct psdi_ctxt ctxt = { .cb = cb, .arg = arg };
+    int rc;
+
+    pcidevs_lock();
+    rc = pci_segments_iterate(pci_segment_devices_iterate, &ctxt);
+    pcidevs_unlock();
+
+    return rc;
 }
 
 void __hwdom_init setup_hwdom_pci_devices(

xen/drivers/passthrough/pci.c void __hwdom_init setup_hwdom_pci_devices( · line 1216

Quoted 7 lines
xen/drivers/passthrough/pci.c +1 -3
@@ -1189,9 +1213,7 @@ void __hwdom_init setup_hwdom_pci_devices(
 {
     struct setup_hwdom ctxt = { .d = d, .handler = handler };
 
-    pcidevs_lock();
-    pci_segments_iterate(_setup_hwdom_pci_devices, &ctxt);
-    pcidevs_unlock();
+    pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt);

Since this now returns an error code it wold be good to handle it, even if it's just:

rc = pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt); if ( rc )

    ASSERT_UNREACHABLE();

setup_hwdom_pci_device will always return 0, hence a failure here is not possible AFAICT. Same with the other usage of pci_pdevs_iterate below.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH v2 2/4] pci: add all-device iterator function...

Quoted 20 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 16:21
To: Paul Durrant <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Tim (Xen.org) <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 2/4] pci: add all-device iterator function...

Quoted 9 lines · Paul Durrant

...and use it for setup_hwdom_pci_devices() and dump_pci_devices().

The unlock/process-pending-softirqs/lock sequence that was in _setup_hwdom_pci_devices() is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case of dump_pci_devices(), since run_all_nonirq_keyhandlers() will call process_pending_softirqs() before invoking each key handler anyway, but this is not performance critical code.

Quoted 9 lines
Quoted 6 lines

The "==== segment XXXX ====" headline that was in _dump_pci_devices() has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway.

Thanks.

Quoted 49 lines

Just some trivial comments.

Thanks.

xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom

Quoted 23 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- New in v2.

---
 xen/drivers/passthrough/pci.c | 120 +++++++++++++++++++++++-------------------
 xen/include/xen/pci.h         |   1 +
 2 files changed, 68 insertions(+), 53 deletions(-)
xen/drivers/passthrough/pci.c +0 -0
index e88689425d..179cb7e17e 100644
@@ -1134,54 +1134,78 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom

*ctxt,

Quoted 21 lines
xen/drivers/passthrough/pci.c +10 -4
                ctxt->d->domain_id, err);
 }
 
-static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg)
+static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg)
 {
     struct setup_hwdom *ctxt = arg;
-    int bus, devfn;
+    struct domain *d = ctxt->d;
 
-    for ( bus = 0; bus < 256; bus++ )
+    if ( !pdev->domain )
     {
-        for ( devfn = 0; devfn < 256; devfn++ )
+        pdev->domain = d;
+        list_add(&pdev->domain_list, &d->pdev_list);
+        setup_one_hwdom_device(ctxt, pdev);
+    }
+    else if ( pdev->domain == dom_xen )
+    {
+        pdev->domain = d;
Quoted 9 lines

xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1156

Quoted 6 lines
xen/drivers/passthrough/pci.c +6 -0
+        setup_one_hwdom_device(ctxt, pdev);
+        pdev->domain = dom_xen;
+    }
+    else if ( pdev->domain != d )
+        printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
+               pdev->domain->domain_id, pdev->seg, pdev->bus,

You can use %pd here to print the domain.

Oh, I didn't know about that one.

Quoted 8 lines
Quoted 8 lines
xen/drivers/passthrough/pci.c +8 -0
+               PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
+
+    return 0;
+}
+
+struct psdi_ctxt {
+    int (*cb)(struct pci_dev *, void *);
+    void *arg;
Quoted 9 lines

xen/drivers/passthrough/pci.c static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, · line 1170

Quoted 6 lines
xen/drivers/passthrough/pci.c +6 -0
+};
+
+static int pci_segment_devices_iterate(struct pci_seg *pseg, void *arg)
+{
+    struct psdi_ctxt *ctxt = arg;
+    int bus, devfn;

unsigned for both the above.

Ok.

Quoted 81 lines
Quoted 70 lines
xen/drivers/passthrough/pci.c +26 -28
+    int rc = 0;
+
+    /*
+     * We don't iterate by walking pseg->alldevs_list here because that
+     * would make the pcidevs_unlock()/lock() sequence below unsafe.
+     */
+    for ( bus = 0; !rc && bus < 256; bus++ )
+        for ( devfn = 0; !rc && devfn < 256; devfn++ )
         {
             struct pci_dev *pdev = pci_get_pdev(pseg->nr, bus, devfn);
 
             if ( !pdev )
                 continue;
 
-            if ( !pdev->domain )
-            {
-                pdev->domain = ctxt->d;
-                list_add(&pdev->domain_list, &ctxt->d->pdev_list);
-                setup_one_hwdom_device(ctxt, pdev);
-            }
-            else if ( pdev->domain == dom_xen )
-            {
-                pdev->domain = ctxt->d;
-                setup_one_hwdom_device(ctxt, pdev);
-                pdev->domain = dom_xen;
-            }
-            else if ( pdev->domain != ctxt->d )
-                printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n",
-                       pdev->domain->domain_id, pseg->nr, bus,
-                       PCI_SLOT(devfn), PCI_FUNC(devfn));
+            rc = ctxt->cb(pdev, ctxt->arg);
 
-            if ( iommu_verbose )
-            {
-                pcidevs_unlock();
-                process_pending_softirqs();
-                pcidevs_lock();
-            }
-        }
-
-        if ( !iommu_verbose )
-        {
+            /*
+             * Err on the safe side and assume the callback has taken
+             * a significant amount of time.
+             */
             pcidevs_unlock();
             process_pending_softirqs();
             pcidevs_lock();
         }
-    }
 
-    return 0;
+    return rc;
+}
+
+int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg)
+{
+    struct psdi_ctxt ctxt = { .cb = cb, .arg = arg };
+    int rc;
+
+    pcidevs_lock();
+    rc = pci_segments_iterate(pci_segment_devices_iterate, &ctxt);
+    pcidevs_unlock();
+
+    return rc;
 }
 
 void __hwdom_init setup_hwdom_pci_devices(

xen/drivers/passthrough/pci.c void __hwdom_init setup_hwdom_pci_devices( · line 1216

Quoted 7 lines
xen/drivers/passthrough/pci.c +1 -3
@@ -1189,9 +1213,7 @@ void __hwdom_init setup_hwdom_pci_devices(
 {
     struct setup_hwdom ctxt = { .d = d, .handler = handler };
 
-    pcidevs_lock();
-    pci_segments_iterate(_setup_hwdom_pci_devices, &ctxt);
-    pcidevs_unlock();
+    pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt);

Since this now returns an error code it wold be good to handle it, even if it's just:

Quoted 7 lines

rc = pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt); if ( rc )

    ASSERT_UNREACHABLE();

setup_hwdom_pci_device will always return 0, hence a failure here is not possible AFAICT. Same with the other usage of pci_pdevs_iterate below.

An ASSERT in setup_hwdom_pci_devices() sounds like a good idea. I think I'll leave one out of dump_pci_devices() though, since that's a debug-key handler.

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

PATCH v2 3/4 iommu: introduce iommu_groups

In reply to Paul Durrant

Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.

This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.

---
 xen/drivers/passthrough/Makefile    |  1 +
 xen/drivers/passthrough/groups.c    | 91 +++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/x86/iommu.c |  8 +++-
 xen/include/xen/iommu.h             |  7 +++
 xen/include/xen/pci.h               |  2 +
 5 files changed, 108 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
index d50ab188c8..8a77110179 100644
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86
 subdir-$(CONFIG_ARM) += arm
 
 obj-y += iommu.o
+obj-$(CONFIG_HAS_PCI) += groups.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 
xen/drivers/passthrough/groups.c new file +91 -0

91 lines not on this page · raw message

xen/drivers/passthrough/x86/iommu.c +7 -1
index a7438c9c25..90fc750456 100644
@@ -43,7 +43,13 @@ int __init iommu_hardware_setup(void)
         /* x2apic setup may have previously initialised the struct. */
         ASSERT(iommu_ops.init == iommu_init_ops->ops->init);
 
-    return iommu_init_ops->setup();
+    rc = iommu_init_ops->setup();
+    if ( rc )
+        return rc;
+
+    iommu_groups_init();
+
+    return pci_pdevs_iterate(iommu_group_assign, NULL);
 }
 
 int iommu_enable_x2apic(void)
xen/include/xen/iommu.h +7 -0
index 48f87480a7..c93f580fdc 100644
@@ -317,6 +317,13 @@ DECLARE_PER_CPU(bool_t, iommu_dont_flush_iotlb);
 extern struct spinlock iommu_pt_cleanup_lock;
 extern struct page_list_head iommu_pt_cleanup_list;
 
+#ifdef CONFIG_HAS_PCI
+
+void iommu_groups_init(void);
+int iommu_group_assign(struct pci_dev *pdev, void *arg);
+
+#endif /* CONFIG_HAS_PCI */
+
 #endif /* _IOMMU_H_ */
 
 /*
xen/include/xen/pci.h +2 -0
index 79eb25417b..e1f887af1c 100644
@@ -79,6 +79,8 @@ struct pci_dev {
     struct list_head alldevs_list;
     struct list_head domain_list;
 
+    struct iommu_group *grp;
+
     struct list_head msi_list;
 
     struct arch_msix *msix;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 17 lines · Paul Durrant

Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.

This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.

Quoted 1 line

LGTM, just two comments below.

Quoted 90 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.

---
 xen/drivers/passthrough/Makefile    |  1 +
 xen/drivers/passthrough/groups.c    | 91 +++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/x86/iommu.c |  8 +++-
 xen/include/xen/iommu.h             |  7 +++
 xen/include/xen/pci.h               |  2 +
 5 files changed, 108 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
index d50ab188c8..8a77110179 100644
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86
 subdir-$(CONFIG_ARM) += arm
 
 obj-y += iommu.o
+obj-$(CONFIG_HAS_PCI) += groups.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 
xen/drivers/passthrough/groups.c new file +47 -0
new file mode 100644
index 0000000000..1a2f461c87
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2019 Citrix Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/iommu.h>
+#include <xen/radix-tree.h>
+
+struct iommu_group {
+    unsigned int id;
+};
+
+static struct radix_tree_root iommu_groups;
+
+void __init iommu_groups_init(void)
+{
+    radix_tree_init(&iommu_groups);
+}
+
+static struct iommu_group *alloc_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = xzalloc(struct iommu_group);
+
+    if ( !grp )
+        return NULL;
+
+    grp->id = id;
+
+    if ( radix_tree_insert(&iommu_groups, id, grp) )
+    {
+        xfree(grp);
+        grp = NULL;
+    }
+
+    return grp;
+}
+
+static struct iommu_group *get_iommu_group(unsigned int id)

xen/drivers/passthrough/groups.c line 57

Quoted 10 lines
xen/drivers/passthrough/groups.c new file +10 -0
+{
+    struct iommu_group *grp = radix_tree_lookup(&iommu_groups, id);
+
+    if ( !grp )
+        grp = alloc_iommu_group(id);
+
+    return grp;
+}
+
+int iommu_group_assign(struct pci_dev *pdev, void *arg)

I'm not sure I see the point of the arg parameter, AFAICT it's completely unused.

xen/drivers/passthrough/groups.c line 66

Quoted 9 lines
xen/drivers/passthrough/groups.c new file +9 -0
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    unsigned int id;
+    struct iommu_group *grp;
+
+    if ( !ops->get_device_group_id )
+        return 0;
+
+    id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn);

I think I would prefer id to be of signed type here, then when you pass it to get_iommu_group it's already made unsigned.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 127 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 16:35
To: Paul Durrant <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Tim (Xen.org) <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 17 lines · Paul Durrant

Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.

This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.

Quoted 1 line

LGTM, just two comments below.

Quoted 95 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.

---
 xen/drivers/passthrough/Makefile    |  1 +
 xen/drivers/passthrough/groups.c    | 91 +++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/x86/iommu.c |  8 +++-
 xen/include/xen/iommu.h             |  7 +++
 xen/include/xen/pci.h               |  2 +
 5 files changed, 108 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
index d50ab188c8..8a77110179 100644
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86
 subdir-$(CONFIG_ARM) += arm
 
 obj-y += iommu.o
+obj-$(CONFIG_HAS_PCI) += groups.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 
xen/drivers/passthrough/groups.c new file +52 -0
new file mode 100644
index 0000000000..1a2f461c87
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2019 Citrix Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/iommu.h>
+#include <xen/radix-tree.h>
+
+struct iommu_group {
+    unsigned int id;
+};
+
+static struct radix_tree_root iommu_groups;
+
+void __init iommu_groups_init(void)
+{
+    radix_tree_init(&iommu_groups);
+}
+
+static struct iommu_group *alloc_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = xzalloc(struct iommu_group);
+
+    if ( !grp )
+        return NULL;
+
+    grp->id = id;
+
+    if ( radix_tree_insert(&iommu_groups, id, grp) )
+    {
+        xfree(grp);
+        grp = NULL;
+    }
+
+    return grp;
+}
+
+static struct iommu_group *get_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = radix_tree_lookup(&iommu_groups, id);
+
+    if ( !grp )
+        grp = alloc_iommu_group(id);
Quoted 8 lines

xen/drivers/passthrough/groups.c line 57

Quoted 5 lines
xen/drivers/passthrough/groups.c new file +5 -0
+
+    return grp;
+}
+
+int iommu_group_assign(struct pci_dev *pdev, void *arg)

I'm not sure I see the point of the arg parameter, AFAICT it's completely unused.

It needs to be there because it needs to conform to the all device iterator function callback prototype. It is indeed unused though.

Quoted 13 lines

xen/drivers/passthrough/groups.c line 66

Quoted 10 lines
xen/drivers/passthrough/groups.c new file +9 -0
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    unsigned int id;
+    struct iommu_group *grp;
+
+    if ( !ops->get_device_group_id )
+        return 0;
+
+    id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn);

I think I would prefer id to be of signed type here, then when you pass it to get_iommu_group it's already made unsigned.

That's a good catch. It's tested for < 0 below so it does need to be signed at this point.

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by
#11 lore raw

Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 136 lines · Paul Durrant
Quoted 127 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 16:35
To: Paul Durrant <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Tim (Xen.org) <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 17 lines · Paul Durrant

Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.

This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.

Quoted 1 line

LGTM, just two comments below.

Quoted 95 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.

---
 xen/drivers/passthrough/Makefile    |  1 +
 xen/drivers/passthrough/groups.c    | 91 +++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/x86/iommu.c |  8 +++-
 xen/include/xen/iommu.h             |  7 +++
 xen/include/xen/pci.h               |  2 +
 5 files changed, 108 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
index d50ab188c8..8a77110179 100644
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86
 subdir-$(CONFIG_ARM) += arm
 
 obj-y += iommu.o
+obj-$(CONFIG_HAS_PCI) += groups.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 
xen/drivers/passthrough/groups.c new file +52 -0
new file mode 100644
index 0000000000..1a2f461c87
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2019 Citrix Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/iommu.h>
+#include <xen/radix-tree.h>
+
+struct iommu_group {
+    unsigned int id;
+};
+
+static struct radix_tree_root iommu_groups;
+
+void __init iommu_groups_init(void)
+{
+    radix_tree_init(&iommu_groups);
+}
+
+static struct iommu_group *alloc_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = xzalloc(struct iommu_group);
+
+    if ( !grp )
+        return NULL;
+
+    grp->id = id;
+
+    if ( radix_tree_insert(&iommu_groups, id, grp) )
+    {
+        xfree(grp);
+        grp = NULL;
+    }
+
+    return grp;
+}
+
+static struct iommu_group *get_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = radix_tree_lookup(&iommu_groups, id);
+
+    if ( !grp )
+        grp = alloc_iommu_group(id);
Quoted 8 lines

xen/drivers/passthrough/groups.c line 57

Quoted 5 lines
xen/drivers/passthrough/groups.c new file +5 -0
+
+    return grp;
+}
+
+int iommu_group_assign(struct pci_dev *pdev, void *arg)

I'm not sure I see the point of the arg parameter, AFAICT it's completely unused.

Quoted 1 line

It needs to be there because it needs to conform to the all device iterator function callback prototype. It is indeed unused though.

Oh right, sorry for the noise.

Quoted 14 lines
Quoted 13 lines

xen/drivers/passthrough/groups.c line 66

Quoted 10 lines
xen/drivers/passthrough/groups.c new file +9 -0
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    unsigned int id;
+    struct iommu_group *grp;
+
+    if ( !ops->get_device_group_id )
+        return 0;
+
+    id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn);

I think I would prefer id to be of signed type here, then when you pass it to get_iommu_group it's already made unsigned.

Quoted 1 line

That's a good catch. It's tested for < 0 below so it does need to be signed at this point.

With that you can add my:

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 168 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 16 July 2019 09:57
To: Paul Durrant <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Tim (Xen.org) <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 16:35
To: Paul Durrant <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Wei Liu

<[email protected]>;

Quoted 2 lines
Quoted 2 lines

Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Tim (Xen.org)

<[email protected]>;

Quoted 129 lines
Quoted 120 lines

Julien Grall <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 3/4] iommu: introduce iommu_groups

Quoted 17 lines · Paul Durrant

Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS.

This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code.

Quoted 1 line

LGTM, just two comments below.

Quoted 95 lines

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Move code into new drivers/passthrough/groups.c
- Drop the group index.
- Handle failure to get group id.
- Drop the group devs list.

---
 xen/drivers/passthrough/Makefile    |  1 +
 xen/drivers/passthrough/groups.c    | 91 +++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/x86/iommu.c |  8 +++-
 xen/include/xen/iommu.h             |  7 +++
 xen/include/xen/pci.h               |  2 +
 5 files changed, 108 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/groups.c
xen/drivers/passthrough/Makefile +1 -0
index d50ab188c8..8a77110179 100644
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86
 subdir-$(CONFIG_ARM) += arm
 
 obj-y += iommu.o
+obj-$(CONFIG_HAS_PCI) += groups.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 
xen/drivers/passthrough/groups.c new file +52 -0
new file mode 100644
index 0000000000..1a2f461c87
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2019 Citrix Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/iommu.h>
+#include <xen/radix-tree.h>
+
+struct iommu_group {
+    unsigned int id;
+};
+
+static struct radix_tree_root iommu_groups;
+
+void __init iommu_groups_init(void)
+{
+    radix_tree_init(&iommu_groups);
+}
+
+static struct iommu_group *alloc_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = xzalloc(struct iommu_group);
+
+    if ( !grp )
+        return NULL;
+
+    grp->id = id;
+
+    if ( radix_tree_insert(&iommu_groups, id, grp) )
+    {
+        xfree(grp);
+        grp = NULL;
+    }
+
+    return grp;
+}
+
+static struct iommu_group *get_iommu_group(unsigned int id)
+{
+    struct iommu_group *grp = radix_tree_lookup(&iommu_groups, id);
+
+    if ( !grp )
+        grp = alloc_iommu_group(id);
Quoted 8 lines

xen/drivers/passthrough/groups.c line 57

Quoted 5 lines
xen/drivers/passthrough/groups.c new file +5 -0
+
+    return grp;
+}
+
+int iommu_group_assign(struct pci_dev *pdev, void *arg)

I'm not sure I see the point of the arg parameter, AFAICT it's completely unused.

Quoted 1 line

It needs to be there because it needs to conform to the all device iterator function callback

prototype. It is indeed unused though.

Oh right, sorry for the noise.

Quoted 14 lines
Quoted 13 lines

xen/drivers/passthrough/groups.c line 66

Quoted 10 lines
xen/drivers/passthrough/groups.c new file +9 -0
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    unsigned int id;
+    struct iommu_group *grp;
+
+    if ( !ops->get_device_group_id )
+        return 0;
+
+    id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn);

I think I would prefer id to be of signed type here, then when you pass it to get_iommu_group it's already made unsigned.

Quoted 5 lines
Quoted 1 line

That's a good catch. It's tested for < 0 below so it does need to be signed at this point.

With that you can add my:

Thanks :-)

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

PATCH v2 4/4 iommu / pci: re-implement XEN_DOMCTL_get_device_group...

In reply to Paul Durrant

... using the new iommu_group infrastructure.

Because 'sibling' devices are now members of the same iommu_group, implement the domctl by looking up the iommu_group of the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.

---
Cc: Jan Beulich <[email protected]>

v2:
- Re-implement in the absence of a per-group devs list.
- Make use of pci_sbdf_t.

---
 xen/drivers/passthrough/groups.c | 44 ++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/pci.c    | 51 ++--------------------------------------
 xen/include/xen/iommu.h          |  2 ++
 3 files changed, 48 insertions(+), 49 deletions(-)
xen/drivers/passthrough/groups.c +44 -0
index 1a2f461c87..6d8064e4f4 100644
@@ -12,8 +12,12 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/guest_access.h>
 #include <xen/iommu.h>
+#include <xen/pci.h>
 #include <xen/radix-tree.h>
+#include <xen/sched.h>
+#include <xsm/xsm.h>
 
 struct iommu_group {
     unsigned int id;
@@ -81,6 +85,46 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)
     return 0;
 }
 
+int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf,
+                           XEN_GUEST_HANDLE_64(uint32) buf, int max_sdevs)
+{
+    struct iommu_group *grp = NULL;
+    struct pci_dev *pdev;
+    unsigned int i = 0;
+
+    pcidevs_lock();
+
+    for_each_pdev ( d, pdev )
+    {
+        if ( pdev->sbdf.sbdf == sbdf.sbdf )
+        {
+            grp = pdev->grp;
+            break;
+        }
+    }
+
+    if ( !grp )
+        goto out;
+
+    for_each_pdev ( d, pdev )
+    {
+        if ( xsm_get_device_group(XSM_HOOK, pdev->sbdf.sbdf) ||
+             pdev->grp != grp )
+            continue;
+
+        if ( unlikely(copy_to_guest_offset(buf, i++, &pdev->sbdf.sbdf, 1)) )
+        {
+            pcidevs_unlock();
+            return -EFAULT;
+        }
+    }
+
+ out:
+    pcidevs_unlock();
+
+    return i;
+}
+
 /*
  * Local variables:
  * mode: C
xen/drivers/passthrough/pci.c +2 -49

66 lines not on this page · raw message

xen/include/xen/iommu.h +2 -0
index c93f580fdc..ac764b41f9 100644
@@ -321,6 +321,8 @@ extern struct page_list_head iommu_pt_cleanup_list;
 
 void iommu_groups_init(void);
 int iommu_group_assign(struct pci_dev *pdev, void *arg);
+int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf,
+                           XEN_GUEST_HANDLE_64(uint32) buf, int max_sdevs);
 
 #endif /* CONFIG_HAS_PCI */
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#14 lore raw

Re: [Xen-devel] [PATCH v2 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...

Quoted 38 lines · Paul Durrant

... using the new iommu_group infrastructure.

Because 'sibling' devices are now members of the same iommu_group, implement the domctl by looking up the iommu_group of the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.

---
Cc: Jan Beulich <[email protected]>

v2:
- Re-implement in the absence of a per-group devs list.
- Make use of pci_sbdf_t.

---
 xen/drivers/passthrough/groups.c | 44 ++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/pci.c    | 51 ++--------------------------------------
 xen/include/xen/iommu.h          |  2 ++
 3 files changed, 48 insertions(+), 49 deletions(-)
xen/drivers/passthrough/groups.c +4 -0
index 1a2f461c87..6d8064e4f4 100644
@@ -12,8 +12,12 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/guest_access.h>
 #include <xen/iommu.h>
+#include <xen/pci.h>
 #include <xen/radix-tree.h>
+#include <xen/sched.h>
+#include <xsm/xsm.h>
 
 struct iommu_group {
     unsigned int id;

xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 89

Quoted 5 lines
xen/drivers/passthrough/groups.c +2 -0
@@ -81,6 +85,46 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)
     return 0;
 }
 
+int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf,
+                           XEN_GUEST_HANDLE_64(uint32) buf, int max_sdevs)

max_sdevs should be unsigned AFAICT, but it seems to be completely unused. I think you want to do...

Quoted 11 lines
xen/drivers/passthrough/groups.c +11 -0
+{
+    struct iommu_group *grp = NULL;
+    struct pci_dev *pdev;
+    unsigned int i = 0;
+
+    pcidevs_lock();
+
+    for_each_pdev ( d, pdev )
+    {
+        if ( pdev->sbdf.sbdf == sbdf.sbdf )
+        {

xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 110

Quoted 10 lines
xen/drivers/passthrough/groups.c +10 -0
+            grp = pdev->grp;
+            break;
+        }
+    }
+
+    if ( !grp )
+        goto out;
+
+    for_each_pdev ( d, pdev )
+    {

if ( i == max_sdevs )

{
    pcidevs_unlock();
    return -ENOSPC;
}

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [Xen-devel] [PATCH v2 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...

Quoted 78 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 15 July 2019 16:46
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH v2 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group...

Quoted 38 lines · Paul Durrant

... using the new iommu_group infrastructure.

Because 'sibling' devices are now members of the same iommu_group, implement the domctl by looking up the iommu_group of the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group.

---
Cc: Jan Beulich <[email protected]>

v2:
- Re-implement in the absence of a per-group devs list.
- Make use of pci_sbdf_t.

---
 xen/drivers/passthrough/groups.c | 44 ++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/pci.c    | 51 ++--------------------------------------
 xen/include/xen/iommu.h          |  2 ++
 3 files changed, 48 insertions(+), 49 deletions(-)
xen/drivers/passthrough/groups.c +4 -0
index 1a2f461c87..6d8064e4f4 100644
@@ -12,8 +12,12 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/guest_access.h>
 #include <xen/iommu.h>
+#include <xen/pci.h>
 #include <xen/radix-tree.h>
+#include <xen/sched.h>
+#include <xsm/xsm.h>
 
 struct iommu_group {
     unsigned int id;

xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 89

Quoted 5 lines
xen/drivers/passthrough/groups.c +2 -0
@@ -81,6 +85,46 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg)
     return 0;
 }
 
+int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf,
+                           XEN_GUEST_HANDLE_64(uint32) buf, int max_sdevs)

max_sdevs should be unsigned AFAICT, but it seems to be completely unused. I think you want to do...

Quoted 11 lines
xen/drivers/passthrough/groups.c +11 -0
+{
+    struct iommu_group *grp = NULL;
+    struct pci_dev *pdev;
+    unsigned int i = 0;
+
+    pcidevs_lock();
+
+    for_each_pdev ( d, pdev )
+    {
+        if ( pdev->sbdf.sbdf == sbdf.sbdf )
+        {

xen/drivers/passthrough/groups.c int iommu_group_assign(struct pci_dev *pdev, void *arg) · line 110

Quoted 10 lines
xen/drivers/passthrough/groups.c +10 -0
+            grp = pdev->grp;
+            break;
+        }
+    }
+
+    if ( !grp )
+        goto out;
+
+    for_each_pdev ( d, pdev )
+    {

if ( i == max_sdevs )

Quoted 4 lines
{
    pcidevs_unlock();
    return -ENOSPC;
}

Oh, I'm sure I used to have that... I don't know how it got dropped. It certainly needs to be there.

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel