Skip to contentSkip to conversation

PCI/iommu: Fix DMA alias problems

PATCH George Spelvin 4 participants 6 messages May 10, 2014 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Well, chalk up one test failure.

I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:

-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -dmar: DRHD: handling fault status reg 502
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 700
-dmar: DRHD: handling fault status reg 702
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 100
-ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -dmar: DRHD: handling fault status reg 102
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -ata8.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata8: limiting SATA link speed to 1.5 Gbps
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata7: limiting SATA link speed to 1.5 Gbps
-dmar: DRHD: handling fault status reg 202
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 400

Andrew's patch was on top of 3.14.3; my initial attempts to forward-port it to the 3.15 PCI changes (appended below) failed to boot.

commit 2323d20989aca44ce0e630e7d264b3973c1ee357

Author: Andrew Cooks <[email protected]>
Date:   Mon Jan 20 07:06:58 2014 -0500

drivers/pci: Add quirk for devices that use multiple functions.

Patch taken from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
and improved locally.

WARNING WARNING WARNING: This is a BROKEN attempt to make it work with 3.15; it DOES NOT BOOT in the form shown here.

drivers/iommu/intel-iommu.c +128 -0
index f256ffc02e..04d430d464 100644
@@ -1841,6 +1841,118 @@ static int domain_context_mapping_one(struct dmar_domain *domain,
 	return 0;
 }
 
+static void iommu_detach_dev(struct intel_iommu *iommu, u8 bus, u8 devfn);
+
+static void quirk_unmap_multi_requesters(struct pci_dev *pdev, u8 fn_map)
+{
+	int fn;
+	u8 bus, devfn;
+	struct intel_iommu *iommu = device_to_iommu(&pdev->dev, &bus, &devfn);
+
+	/* something must be seriously wrong if we can't lookup the iommu. */
+	BUG_ON(!iommu);
+
+	fn_map &= ~(1<<PCI_FUNC(devfn));	/* Skip the normal case */
+
+	for (fn = 0; fn_map >> fn; fn++) {
+		if (fn_map & (1<<fn)) {
+			iommu_detach_dev(iommu,
+					pdev->bus->number,
+					PCI_DEVFN(PCI_SLOT(devfn), fn));
+			dev_dbg(&pdev->dev,
+				"requester id quirk; ghost func %d unmapped",
+				fn);
+		}
+	}
+}
+
+/* For quirky devices that use multiple requester ids. */
+static int quirk_map_multi_requester_ids(struct dmar_domain *domain,
+		struct pci_dev *pdev,
+		int translation)
+{
+	int fn, err = 0;
+	u8 fn_map = pci_multi_requesters(pdev);
+	u8 bus, devfn;
+	struct intel_iommu *iommu;
+
+	/* this is the common, non-quirky case. */
+	if (!fn_map)
+		return 0;
+
+	fn_map &= ~(1<<PCI_FUNC(pdev->devfn));	/* Skip the normal case */
+
+	iommu = device_to_iommu(&pdev->dev, &bus, &devfn);
+
+	for (fn = 0; fn_map >> fn; fn++) {
+		if (fn_map & (1<<fn)) {
+			err = domain_context_mapping_one(domain, iommu, bus,
+					PCI_DEVFN(PCI_SLOT(pdev->devfn), fn),
+					translation);
+			if (err) {
+				dev_err(&pdev->dev,
+					"mapping ghost func %d failed", fn);
+				quirk_unmap_multi_requesters(pdev,
+					fn_map & ((1<<fn)-1));
+				return err;
+			}
+			dev_dbg(&pdev->dev,
+				"requester id quirk; ghost func %d mapped", fn);
+		}
+	}
+	return 0;
+}
+
+
+static void quirk_unmap_requester_id(struct pci_dev *pdev)
+{
+	u8 bus, devfn;
+	struct intel_iommu *iommu = device_to_iommu(&pdev->dev, &bus, &devfn);
+	u8 devfn2 = pci_requester(pdev);
+
+	/* something must be seriously wrong if we can't lookup the iommu. */
+	BUG_ON(!iommu);
+
+
+	if (devfn == devfn2)
+		return;
+
+	iommu_detach_dev(iommu,	bus, devfn2);
+	dev_dbg(&pdev->dev, "requester id quirk; bugged device unmapped");
+}
+
+static int quirk_map_requester_id(struct dmar_domain *domain,
+		struct pci_dev *pdev,
+		int translation)
+{
+	u8 bus, devfn;
+	struct intel_iommu *iommu = device_to_iommu(&pdev->dev, &bus, &devfn);
+	u8 devfn2 = pci_requester(pdev);
+	int err;
+
+	if (!iommu)
+		return -ENODEV;
+
+	dev_dbg(&pdev->dev,
+		"checking for incorrect pci requester id quirk...");
+
+	if (devfn == devfn2)
+		return 0;
+
+	err = domain_context_mapping_one(domain, iommu, bus, devfn2,
+			translation);
+	if (err) {
+		dev_err(&pdev->dev,
+			"requester id quirk: mapping dev %02x:%02x.%d failed",
+			bus, PCI_SLOT(devfn2), PCI_FUNC(devfn2));
+		return err;
+	}
+	dev_dbg(&pdev->dev,
+		"requester id quirk; dmar context entry added: %02x:%02x.%d",
+		bus, PCI_SLOT(devfn2), PCI_FUNC(devfn2));
+	return 0;
+}
+
 static int
 domain_context_mapping(struct dmar_domain *domain, struct device *dev,
 		       int translation)
@@ -1859,6 +1971,16 @@ domain_context_mapping(struct dmar_domain *domain, struct device *dev,
 	if (ret || !dev_is_pci(dev))
 		return ret;
 
+	/* quirk for devices using multiple pci requester ids */
+	ret = quirk_map_multi_requester_ids(domain, pdev, translation);
+	if (ret)
+		return ret;
+
+	/* quirk for devices single incorrect pci requester id */
+	ret = quirk_map_requester_id(domain, pdev, translation);
+	if (ret)
+		return ret;
+
 	/* dependent device mapping */
 	pdev = to_pci_dev(dev);
 	tmp = pci_find_upstream_pcie_bridge(pdev);
@@ -4076,12 +4198,18 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain,
 	list_for_each_entry_safe(info, tmp, &domain->devices, link) {
 		if (info->iommu == iommu && info->bus == bus &&
 		    info->devfn == devfn) {
+			struct pci_dev *pdev;
+
 			unlink_domain_info(info);
 			spin_unlock_irqrestore(&device_domain_lock, flags);
 
 			iommu_disable_dev_iotlb(info);
 			iommu_detach_dev(iommu, info->bus, info->devfn);
 			iommu_detach_dependent_devices(iommu, dev);
+			pdev = to_pci_dev(dev);
+			quirk_unmap_multi_requesters(pdev,
+						pci_multi_requesters(pdev));
+			quirk_unmap_requester_id(pdev);
 			free_devinfo_mem(info);
 
 			spin_lock_irqsave(&device_domain_lock, flags);
drivers/pci/quirks.c +144 -1

163 lines not on this page · raw message

include/linux/pci.h +10 -0
index aab57b4abe..63582ad30f 100644
@@ -1529,6 +1529,8 @@ enum pci_fixup_pass {
 #ifdef CONFIG_PCI_QUIRKS
 void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev);
 struct pci_dev *pci_get_dma_source(struct pci_dev *dev);
+u8 pci_multi_requesters(struct pci_dev *dev);
+u8 pci_requester(struct pci_dev *dev);
 int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags);
 void pci_dev_specific_enable_acs(struct pci_dev *dev);
 #else
@@ -1538,6 +1540,14 @@ static inline struct pci_dev *pci_get_dma_source(struct pci_dev *dev)
 {
 	return pci_dev_get(dev);
 }
+u8 pci_multi_requesters(struct pci_dev *dev)
+{
+	return 0;
+}
+u8 pci_requester(struct pci_dev *dev)
+{
+	return dev->devfn;
+}
 static inline int pci_dev_specific_acs_enabled(struct pci_dev *dev,
 					       u16 acs_flags)
 {
#1 lore raw
Quoted 12 lines · George Spelvin

Well, chalk up one test failure.

I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:

Review

Quoted 4 lines
(quoted diff) +0 -4
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
-dmar: DRHD: handling fault status reg 502
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000
-DMAR:[fault reason 02] Present bit in context entry is clear

When I checked, about 12 hours ago, the dma-alias branch on github didn't have all the v2 changes, so I applied the v2 patch set on 3.15-rc5. You may want to try that as well.

I didn't notice any problems on the board I used: GA-X79-UD5, with the same 88SE9172 controllers as above and with drives attached to each of the internal SATA ports.

a.

A message that is not in the archive (sent off-list or not yet received): 20140510121542.5359.qmail-HzZAx2gCgqrSUeElwK9/[email protected]

#3 lore raw
Quoted 7 lines · George Spelvin

Well, chalk up one test failure.

I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

What's the device ID of these devices?

Quoted 3 lines

With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:

The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,

Alex

Quoted 28 lines
(quoted diff) +0 -28
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
-dmar: DRHD: handling fault status reg 502
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DRHD: handling fault status reg 700
-dmar: DRHD: handling fault status reg 702
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DRHD: handling fault status reg 100
-ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
-ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
-dmar: DRHD: handling fault status reg 102
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-ata8.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
-ata8: limiting SATA link speed to 1.5 Gbps
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
-ata7: limiting SATA link speed to 1.5 Gbps
-dmar: DRHD: handling fault status reg 202
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 
-DMAR:[fault reason 02] Present bit in context entry is clear
-dmar: DRHD: handling fault status reg 400
#4 lore raw
Quoted 1 line

What's the device ID of these devices?

Oops! It's

05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 06:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 07:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)

Quoted 3 lines

The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,

Oh, okay! I got the patches from github because it was a lot easier to get the whole set that way. (I don't actually have an SMTP LKML subscription, so I have to copy them all from a web archive, then manually edit the headers into something RFC822-like enough for "git am" to accept.)

Mea culpa for forgetting to check the version. And thank you for the response!

#5 lore raw
Quoted 12 lines · George Spelvin
Quoted 1 line

What's the device ID of these devices?

Oops! It's

05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 06:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 07:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)

Quoted 3 lines

The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,

Quoted 7 lines

Oh, okay! I got the patches from github because it was a lot easier to get the whole set that way. (I don't actually have an SMTP LKML subscription, so I have to copy them all from a web archive, then manually edit the headers into something RFC822-like enough for "git am" to accept.)

Mea culpa for forgetting to check the version. And thank you for the response!

Hi George,

I'll post a v3 soon, I think we figured out the problem Andrew was having, a couple typos on my part. I'll push out a new branch for that so you don't need to piece it together yourself. Thanks,

Alex

#6 lore raw
Quoted 3 lines

I'll post a v3 soon, I think we figured out the problem Andrew was having, a couple typos on my part. I'll push out a new branch for that so you don't need to piece it together yourself. Thanks,

Don't worry; I imported v2, and have it running successfully right now. I ran e2fsck and hdparm -t on a couple of drives on the controller (including in parallel), with no problems.

Again, sorry for my earlier laziness.