Skip to contentSkip to conversation

pci: Reenable the AMD IOMMU if it's mysteriously vanished over suspend

PATCH In review Matthew Garrett 4 participants 5 messages Sep 30, 2010 – Oct 1, 2010 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pci: Reenable the AMD IOMMU if it's mysteriously vanished over suspend

AMD's reference BIOS code had a bug that could result in the firmware failing to reenable the iommu on resume. It transpires that this causes certain less than desirable behaviour when it comes to PCI accesses, to whit them ending up somewhere near Bristol when the more desirable outcome was Edinburgh. Sadness ensues, perhaps along with filesystem corruption. Let's make sure that it gets turned back on.

---
 drivers/pci/quirks.c |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)
drivers/pci/quirks.c +33 -0
index 89ed181..dcf9832 100644
@@ -2695,6 +2695,39 @@ DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_RICOH, PCI_DEVICE_ID_RICOH_R5C832,
 #endif /*CONFIG_MMC_RICOH_MMC*/
 
 
+#ifdef CONFIG_AMD_IOMMU
+
+/* Some AMD BIOSes fail to reenable the iommu on resume */
+
+static void amd_iommu_reenable(struct pci_dev *dev)
+{
+	u32 ioc_feature_control;
+	struct pci_dev *iommu;
+
+	iommu = pci_get_device(PCI_VENDOR_ID_ATI, 0x5a23, NULL);
+
+	if (!iommu)
+		return;
+
+	/* Select Northbridge indirect register 0x75 and enable writing */
+	pci_write_config_dword(dev, 0x60, 0x75 | (1 << 7));
+	pci_read_config_dword(dev, 0x64, &ioc_feature_control);
+
+	/* Enable the iommu if it's vanished */
+	if (!(ioc_feature_control & 0x1))
+		pci_write_config_dword(dev, 0x64, ioc_feature_control | 1);
+
+	/* Disable writing again */
+	pci_write_config_dword(dev, 0x60, 0x75);
+
+	pci_dev_put(iommu);
+}
+
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a10, amd_iommu_reenable);
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a12, amd_iommu_reenable);
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a13, amd_iommu_reenable);
+#endif
+
 static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f,
 			  struct pci_fixup *end)
 {
Signature · 1 line
1.7.3
#1 lore raw
Quoted 6 lines · Matthew Garrett

AMD's reference BIOS code had a bug that could result in the firmware failing to reenable the iommu on resume. It transpires that this causes certain less than desirable behaviour when it comes to PCI accesses, to whit them ending up somewhere near Bristol when the more desirable outcome was Edinburgh. Sadness ensues, perhaps along with filesystem corruption. Let's make sure that it gets turned back on.

Of course, this isn't actually sufficient - nothing in the iommu code appears to reprogram the BAR, for instance. Joerg, how is this meant to work?

Signature · 1 line
Matthew Garrett | [email protected]
#2 lore raw
Quoted 11 lines · Matthew Garrett
Quoted 6 lines · Matthew Garrett

AMD's reference BIOS code had a bug that could result in the firmware failing to reenable the iommu on resume. It transpires that this causes certain less than desirable behaviour when it comes to PCI accesses, to whit them ending up somewhere near Bristol when the more desirable outcome was Edinburgh. Sadness ensues, perhaps along with filesystem corruption. Let's make sure that it gets turned back on.

Of course, this isn't actually sufficient - nothing in the iommu code appears to reprogram the BAR, for instance. Joerg, how is this meant to work?

Enabling the IOMMU device is only part of what the BIOS is supposed to do to get the IOMMU ready for the OS. A handful of registers in the config space of the device and the IOMMU MMIO region must also be restored. Thats the simple part. The IOMMU device also spans two indirect register spaces to configure the IOMMU caches. I have no idea yet how this need to be configured. The plan is definitly to work-around a missing IOMMU device in the IOMMU driver. But I am still talking to some people to find out what exactly must be done, especially with the indirect register spaces.

Joerg

#3 lore raw
Quoted 9 lines · Joerg Roedel

Enabling the IOMMU device is only part of what the BIOS is supposed to do to get the IOMMU ready for the OS. A handful of registers in the config space of the device and the IOMMU MMIO region must also be restored. Thats the simple part. The IOMMU device also spans two indirect register spaces to configure the IOMMU caches. I have no idea yet how this need to be configured. The plan is definitly to work-around a missing IOMMU device in the IOMMU driver. But I am still talking to some people to find out what exactly must be done, especially with the indirect register spaces.

Chapter 8 of the register setup guide seems to have something on this - I'll try saving and restoring all of the setup it performs to see if it improves things.

Signature · 1 line
Matthew Garrett | [email protected]
#4 lore raw
In reply to Matthew Garrett
Quoted 31 lines · Matthew Garrett

AMD's reference BIOS code had a bug that could result in the firmware failing to reenable the iommu on resume. It transpires that this causes certain less than desirable behaviour when it comes to PCI accesses, to whit them ending up somewhere near Bristol when the more desirable outcome was Edinburgh. Sadness ensues, perhaps along with filesystem corruption. Let's make sure that it gets turned back on.

---
 drivers/pci/quirks.c |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)
drivers/pci/quirks.c +11 -0
index 89ed181..dcf9832 100644
@@ -2695,6 +2695,39 @@ DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_RICOH, PCI_DEVICE_ID_RICOH_R5C832,
 #endif /*CONFIG_MMC_RICOH_MMC*/
 
 
+#ifdef CONFIG_AMD_IOMMU
+
+/* Some AMD BIOSes fail to reenable the iommu on resume */
+
+static void amd_iommu_reenable(struct pci_dev *dev)
+{
+	u32 ioc_feature_control;
+	struct pci_dev *iommu;
+
+	iommu = pci_get_device(PCI_VENDOR_ID_ATI, 0x5a23, NULL);
+

drivers/pci/quirks.c DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_RICOH, PCI_DEVICE_ID_RICOH_R5C832, · line 2718

Quoted 10 lines
drivers/pci/quirks.c +10 -0
+	if (!iommu)
+		return;
+
+	/* Select Northbridge indirect register 0x75 and enable writing */
+	pci_write_config_dword(dev, 0x60, 0x75 | (1 << 7));
+	pci_read_config_dword(dev, 0x64, &ioc_feature_control);
+
+	/* Enable the iommu if it's vanished */
+	if (!(ioc_feature_control & 0x1))
+		pci_write_config_dword(dev, 0x64, ioc_feature_control | 1);

This sounds important and tricky enough that it'd be nice to have a note in dmesg about what we're doing.

Quoted 15 lines
drivers/pci/quirks.c +11 -0
+	/* Disable writing again */
+	pci_write_config_dword(dev, 0x60, 0x75);
+
+	pci_dev_put(iommu);
+}
+
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a10, amd_iommu_reenable);
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a12, amd_iommu_reenable);
+DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_ATI, 0x5a13, amd_iommu_reenable);
+#endif
+
 static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f,
 			  struct pci_fixup *end)
 {