PCI/iommu: Fix DMA alias problems
Well, chalk up one test failure.
I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -dmar: DRHD: handling fault status reg 502
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 700
-dmar: DRHD: handling fault status reg 702
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 100
-ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -dmar: DRHD: handling fault status reg 102
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -ata8.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata8: limiting SATA link speed to 1.5 Gbps
-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata7: limiting SATA link speed to 1.5 Gbps
-dmar: DRHD: handling fault status reg 202
-dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 400
Andrew's patch was on top of 3.14.3; my initial attempts to forward-port it to the 3.15 PCI changes (appended below) failed to boot.
commit 2323d20989aca44ce0e630e7d264b3973c1ee357
Author: Andrew Cooks <[email protected]> Date: Mon Jan 20 07:06:58 2014 -0500
drivers/pci: Add quirk for devices that use multiple functions.
Patch taken from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
and improved locally.
WARNING WARNING WARNING: This is a BROKEN attempt to make it work with 3.15; it DOES NOT BOOT in the form shown here.
drivers/iommu/intel-iommu.c +128 -0
@@ -1841,6 +1841,118 @@ static int domain_context_mapping_one(struct dmar_domain *domain,return 0 ; } +static void iommu_detach_dev (struct intel_iommu *iommu,u8 bus,u8 devfn); + +static void quirk_unmap_multi_requesters (struct pci_dev *pdev,u8 fn_map) +{ +int fn; +u8 bus, devfn; +struct intel_iommu *iommu =device_to_iommu (&pdev->dev, &bus, &devfn); + +/* something must be seriously wrong if we can't lookup the iommu. */ +BUG_ON (!iommu); + + fn_map &= ~(1 <<PCI_FUNC (devfn));/* Skip the normal case */ + +for (fn =0 ; fn_map >> fn; fn++) { +if (fn_map & (1 <<fn)) { +iommu_detach_dev (iommu, + pdev->bus->number, +PCI_DEVFN (PCI_SLOT (devfn), fn)); +dev_dbg (&pdev->dev, +"requester id quirk; ghost func %d unmapped" , + fn); + } + } +} + +/* For quirky devices that use multiple requester ids. */ +static int quirk_map_multi_requester_ids (struct dmar_domain *domain, +struct pci_dev *pdev, +int translation) +{ +int fn, err =0 ; +u8 fn_map =pci_multi_requesters (pdev); +u8 bus, devfn; +struct intel_iommu *iommu; + +/* this is the common, non-quirky case. */ +if (!fn_map) +return 0 ; + + fn_map &= ~(1 <<PCI_FUNC (pdev->devfn));/* Skip the normal case */ + + iommu =device_to_iommu (&pdev->dev, &bus, &devfn); + +for (fn =0 ; fn_map >> fn; fn++) { +if (fn_map & (1 <<fn)) { + err =domain_context_mapping_one (domain, iommu, bus, +PCI_DEVFN (PCI_SLOT (pdev->devfn), fn), + translation); +if (err) { +dev_err (&pdev->dev, +"mapping ghost func %d failed" , fn); +quirk_unmap_multi_requesters (pdev, + fn_map & ((1 <<fn)-1 )); +return err; + } +dev_dbg (&pdev->dev, +"requester id quirk; ghost func %d mapped" , fn); + } + } +return 0 ; +} + + +static void quirk_unmap_requester_id (struct pci_dev *pdev) +{ +u8 bus, devfn; +struct intel_iommu *iommu =device_to_iommu (&pdev->dev, &bus, &devfn); +u8 devfn2 =pci_requester (pdev); + +/* something must be seriously wrong if we can't lookup the iommu. */ +BUG_ON (!iommu); + + +if (devfn == devfn2) +return ; + +iommu_detach_dev (iommu, bus, devfn2); +dev_dbg (&pdev->dev,"requester id quirk; bugged device unmapped" ); +} + +static int quirk_map_requester_id (struct dmar_domain *domain, +struct pci_dev *pdev, +int translation) +{ +u8 bus, devfn; +struct intel_iommu *iommu =device_to_iommu (&pdev->dev, &bus, &devfn); +u8 devfn2 =pci_requester (pdev); +int err; + +if (!iommu) +return -ENODEV ; + +dev_dbg (&pdev->dev, +"checking for incorrect pci requester id quirk..." ); + +if (devfn == devfn2) +return 0 ; + + err =domain_context_mapping_one (domain, iommu, bus, devfn2, + translation); +if (err) { +dev_err (&pdev->dev, +"requester id quirk: mapping dev %02x:%02x.%d failed" , + bus,PCI_SLOT (devfn2),PCI_FUNC (devfn2)); +return err; + } +dev_dbg (&pdev->dev, +"requester id quirk; dmar context entry added: %02x:%02x.%d" , + bus,PCI_SLOT (devfn2),PCI_FUNC (devfn2)); +return 0 ; +} +static int domain_context_mapping (struct dmar_domain *domain,struct device *dev,int translation)
@@ -1859,6 +1971,16 @@ domain_context_mapping(struct dmar_domain *domain, struct device *dev,if (ret || !dev_is_pci (dev))return ret; +/* quirk for devices using multiple pci requester ids */ + ret =quirk_map_multi_requester_ids (domain, pdev, translation); +if (ret) +return ret; + +/* quirk for devices single incorrect pci requester id */ + ret =quirk_map_requester_id (domain, pdev, translation); +if (ret) +return ret; +/* dependent device mapping */ pdev =to_pci_dev (dev); tmp =pci_find_upstream_pcie_bridge (pdev);
@@ -4076,12 +4198,18 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain,list_for_each_entry_safe (info, tmp, &domain->devices, link) {if (info->iommu == iommu && info->bus == bus && info->devfn == devfn) { +struct pci_dev *pdev; +unlink_domain_info (info);spin_unlock_irqrestore (&device_domain_lock, flags);iommu_disable_dev_iotlb (info);iommu_detach_dev (iommu, info->bus, info->devfn);iommu_detach_dependent_devices (iommu, dev); + pdev =to_pci_dev (dev); +quirk_unmap_multi_requesters (pdev, +pci_multi_requesters (pdev)); +quirk_unmap_requester_id (pdev);free_devinfo_mem (info);spin_lock_irqsave (&device_domain_lock, flags);
drivers/pci/quirks.c +144 -1
163 lines not on this page · raw message
include/linux/pci.h +10 -0
@@ -1529,6 +1529,8 @@ enum pci_fixup_pass {#ifdef CONFIG_PCI_QUIRKS void pci_fixup_device (enum pci_fixup_pass pass,struct pci_dev *dev);struct pci_dev *pci_get_dma_source (struct pci_dev *dev); +u8 pci_multi_requesters (struct pci_dev *dev); +u8 pci_requester (struct pci_dev *dev);int pci_dev_specific_acs_enabled (struct pci_dev *dev,u16 acs_flags);void pci_dev_specific_enable_acs (struct pci_dev *dev);#else
@@ -1538,6 +1540,14 @@ static inline struct pci_dev *pci_get_dma_source(struct pci_dev *dev) {return pci_dev_get (dev); } +u8 pci_multi_requesters (struct pci_dev *dev) +{ +return 0 ; +} +u8 pci_requester (struct pci_dev *dev) +{ +return dev->devfn; +}static inline int pci_dev_specific_acs_enabled (struct pci_dev *dev,u16 acs_flags) {
Quoted 12 lines · George Spelvin
Well, chalk up one test failure.
I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:
Review
Quoted 4 lines
(quoted diff) +0
-4-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -dmar: DRHD: handling fault status reg 502 -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear
When I checked, about 12 hours ago, the dma-alias branch on github didn't have all the v2 changes, so I applied the v2 patch set on 3.15-rc5. You may want to try that as well.
I didn't notice any problems on the board I used: GA-X79-UD5, with the same 88SE9172 controllers as above and with drives attached to each of the internal SATA ports.
a.
A message that is not in the archive (sent off-list or not yet received): 20140510121542.5359.qmail-HzZAx2gCgqrSUeElwK9/[email protected]
Quoted 7 lines · George Spelvin
Well, chalk up one test failure.
I'm running a GA-X79-UP4 motherboard with VT-d enabled.
It has 3x Marvell SATA controllers:
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
What's the device ID of these devices?
Quoted 3 lines
With Andrew's patch from https://bugzilla.kernel.org/show_bug.cgi?id=42679#c22
the Marvell SATA ports work. With your dma-alias branch (rebased onto 3.15-rc5), the ports don't work:
The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,
Alex
Quoted 28 lines
(quoted diff) +0
-28-ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -dmar: DRHD: handling fault status reg 502 -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 700 -dmar: DRHD: handling fault status reg 702 -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 100 -ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) -dmar: DRHD: handling fault status reg 102 -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -ata8.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata8: limiting SATA link speed to 1.5 Gbps -ata7.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) -ata7: limiting SATA link speed to 1.5 Gbps -dmar: DRHD: handling fault status reg 202 -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffc0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DMAR:[DMA Write] Request device [05:00.1] fault addr fffe0000 -DMAR:[fault reason 02] Present bit in context entry is clear -dmar: DRHD: handling fault status reg 400
Quoted 1 line
What's the device ID of these devices?
Oops! It's
05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 06:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 07:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
Quoted 3 lines
The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,
Oh, okay! I got the patches from github because it was a lot easier to get the whole set that way. (I don't actually have an SMTP LKML subscription, so I have to copy them all from a web archive, then manually edit the headers into something RFC822-like enough for "git am" to accept.)
Mea culpa for forgetting to check the version. And thank you for the response!
Quoted 12 lines · George Spelvin
Quoted 1 line
What's the device ID of these devices?
Oops! It's
05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 06:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 07:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
Quoted 3 lines
The dma-alias branch only has the v1 patchset. There are several more Marvell devices included in the quirks in the v2 set. Please re-test with this code. Thanks,
Quoted 7 lines
Oh, okay! I got the patches from github because it was a lot easier to get the whole set that way. (I don't actually have an SMTP LKML subscription, so I have to copy them all from a web archive, then manually edit the headers into something RFC822-like enough for "git am" to accept.)
Mea culpa for forgetting to check the version. And thank you for the response!
Hi George,
I'll post a v3 soon, I think we figured out the problem Andrew was having, a couple typos on my part. I'll push out a new branch for that so you don't need to piece it together yourself. Thanks,
Alex
Quoted 3 lines
I'll post a v3 soon, I think we figured out the problem Andrew was having, a couple typos on my part. I'll push out a new branch for that so you don't need to piece it together yourself. Thanks,
Don't worry; I imported v2, and have it running successfully right now. I ran e2fsck and hdparm -t on a couple of drives on the controller (including in parallel), with no problems.
Again, sorry for my earlier laziness.