amd iommu: re-enable iommu msi if dom0 disabled it

PATCH Superseded Wei Wang 3 participants 6 messages Jun 8, 2012

Share & tools

Wei Wang

author

2012-06-08 13:28 UTC #0 lore raw

PATCH 1/1 amd iommu: re-enable iommu msi if dom0 disabled it

Hi Jan,
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case. Attached patch re-enables it in device assignment.

Thanks,
Wei

--
Advanced Micro Devices GmbH
Sitz: Dornach, Gemeinde Aschheim,
Landkreis München Registergericht München,
HRB Nr. 43632 WEEE Registrierungsnummer 129 19551
Geschäftsführer:
Alberto Bozzo

# HG changeset patch
# Parent f6bfaf9daa508c31b2bca0e461202db2759426fc
# User Wei Wang <[email protected]>
Re-enable iommu msi capability block if it is disabled by dom0

Signed-off-by Wei Wang <[email protected]>

xen/drivers/passthrough/amd/pci_amd_iommu.c +30 -0

diff -r f6bfaf9daa50 xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -81,6 +81,30 @@ static void disable_translation(u32 *dte
     dte[0] = entry;
 }
 
+static void iommu_msi_check_enable(struct amd_iommu *iommu)
+{
+    unsigned long flags;
+    uint16_t control;
+    uint8_t bus = PCI_BUS(iommu->bdf);
+    uint8_t dev = PCI_SLOT(iommu->bdf);
+    uint8_t func = PCI_FUNC(iommu->bdf);
+
+    ASSERT( iommu->msi_cap );
+    
+    spin_lock_irqsave(&iommu->lock, flags);
+
+    control = pci_conf_read16(iommu->seg, bus, dev, func,
+                              iommu->msi_cap + PCI_MSI_FLAGS);
+    if ( !(control & IOMMU_CONTROL_ENABLED) )
+    {    
+        control |= IOMMU_CONTROL_ENABLED;
+        pci_conf_write16(iommu->seg, bus, dev, func,
+                         iommu->msi_cap + PCI_MSI_FLAGS, control);
+    }
+
+    spin_unlock_irqrestore(&iommu->lock, flags);
+}
+
 static void amd_iommu_setup_domain_device(
     struct domain *domain, struct amd_iommu *iommu, int bdf)
 {

@@ -101,6 +125,12 @@ static void amd_iommu_setup_domain_devic
     if ( ats_enabled )
         dte_i = 1;
 
+    /* 
+     * In some cases, dom0 disables iommu msi capability, 
+     * re-enable it here.
+     */
+    iommu_msi_check_enable(iommu); 
+
     /* get device-table entry */
     req_id = get_dma_requestor_id(iommu->seg, bdf);
     dte = iommu->dev_table.buffer + (req_id * IOMMU_DEV_TABLE_ENTRY_SIZE);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Andrew Cooper

2012-06-08 13:31 UTC #1 lore raw

Quoted 4 lines · Wei Wang

Hi Jan,
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case. Attached patch re-enables it in device assignment.

Under which circumstances should dom0 able to play with the IOMMUs ? Surely the fact that dom0 can play with the IOMMUs is a bug in itself.

Quoted 11 lines

Thanks,
Wei
--
Advanced Micro Devices GmbH
Sitz: Dornach, Gemeinde Aschheim,
Landkreis München Registergericht München,
HRB Nr. 43632 WEEE Registrierungsnummer 129 19551
Geschäftsführer:
Alberto Bozzo

Signature · 2 lines

Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Wei Wang

author

2012-06-08 13:41 UTC #2 lore raw

Quoted 8 lines · Andrew Cooper

Quoted 4 lines · Wei Wang
Hi Jan,
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case. Attached patch re-enables it in device assignment.
Under which circumstances should dom0 able to play with the IOMMUs ? Surely the fact that dom0 can play with the IOMMUs is a bug in itself.

It looks like some generic msi/pci codes disable it, not the Linux iommu driver itself, which is only loaded on bare metal. AMD IOMMU expose interrupt capability as a normal msi block. So the general pci/msi layer of dom0 might touch it...

Thanks,
Wei

Quoted 12 lines

Quoted 11 lines
Thanks,
Wei
--
Advanced Micro Devices GmbH
Sitz: Dornach, Gemeinde Aschheim,
Landkreis München Registergericht München,
HRB Nr. 43632 WEEE Registrierungsnummer 129 19551
Geschäftsführer:
Alberto Bozzo

Jan Beulich

2012-06-08 14:08 UTC #3 lore raw

Quoted 7 lines

Quoted 1 line
Quoted 1 line · Wei Wang
Quoted 5 lines · Andrew Cooper
Quoted 3 lines · Wei Wang
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case. Attached patch re-enables it in device assignment.

Quoted 7 lines

Quoted 2 lines
Under which circumstances should dom0 able to play with the IOMMUs ? Surely the fact that dom0 can play with the IOMMUs is a bug in itself.
It looks like some generic msi/pci codes disable it, not the Linux iommu driver itself, which is only loaded on bare metal. AMD IOMMU expose interrupt capability as a normal msi block. So the general pci/msi layer of dom0 might touch it...

In that case it wouldn't be on pv-ops alone (which your patch says).

Also, your patch using IOMMU_CONTROL_ENABLED instead of PCI_MSI_FLAGS_ENABLE is quite misleading (as it hides the fact the what you play with is not IOMMU-specific).

Jan

Wei Wang

author

2012-06-08 14:16 UTC #4 lore raw

Quoted 10 lines · Jan Beulich

Quoted 10 lines
Quoted 1 line
Quoted 1 line · Wei Wang
Quoted 7 lines · Andrew Cooper
Quoted 3 lines · Wei Wang
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case. Attached patch re-enables it in device assignment.
Under which circumstances should dom0 able to play with the IOMMUs ? Surely the fact that dom0 can play with the IOMMUs is a bug in itself.

Quoted 7 lines

Quoted 4 lines
It looks like some generic msi/pci codes disable it, not the Linux iommu driver itself, which is only loaded on bare metal. AMD IOMMU expose interrupt capability as a normal msi block. So the general pci/msi layer of dom0 might touch it...
In that case it wouldn't be on pv-ops alone (which your patch says).

Yes, probably upstream Linux also has this issue. I will check that. But I did not see this issue on 3.2 pv_ops & 2.6 pv_ops.

Quoted 3 lines

Also, your patch using IOMMU_CONTROL_ENABLED instead of PCI_MSI_FLAGS_ENABLE is quite misleading (as it hides the fact the what you play with is not IOMMU-specific).

Thanks, I will use PCI_MSI_FLAGS_ENABLE in next post.

Wei

Quoted 3 lines

Jan

Jan Beulich

2012-06-08 13:47 UTC #5 lore raw

In reply to Wei Wang

Quoted 3 lines

Quoted 1 line
Quoted 1 line · Wei Wang
I found that recent dom0 (e.g 3.4 pv_ops) disables iommu msi capability for some reasons and iommu cannot generate any interrupts in this case.

That shouldn't happen in the first place, and hence should be fixed in the kernel.

Quoted 1 line

Attached patch re-enables it in device assignment.

I'm not really keen on taking this - it's silently working around a problem, and doesn't appear to cover the case where Dom0 would have its I/O got through the IOMMU as well.

If anything, I'd be much more in favor of hiding the entire config space of the IOMMU device from Dom0 (or at least making it read-only), which admittedly might be a little tricky. So first of all let's understand whether this can't reasonably be fixed in Linux.

Jan