PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs
cover PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs
This solves system hang and file system corruption for Stoney Ridge systems (with built-in Radeon R4/R5 + discreet AMD Radeon R7 GPU) Such system example: HP laptop 17-ak041ur (more HW info at https://bugzilla.kernel.org/show_bug.cgi?id=194521)
Nikolai Kostrigin (1):
PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs
drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)
Signature · 1 line
2.21.0
PATCH RESEND 1/1 PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.
https://bugzilla.kernel.org/show_bug.cgi?id=194521
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)
drivers/pci/quirks.c +1 -0
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x98e4 , quirk_no_ats); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x6900 , quirk_no_ats);#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */
Signature · 1 line
2.21.0
[+cc Alex]
This claims to be a resend, but I don't see a previous posting.
There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:
Quoted 9 lines · Nikolai Kostrigin
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.https://bugzilla.kernel.org/show_bug.cgi?id=194521
Signed-off-by Nikolai Kostrigin <[email protected]>
Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.
Quoted 19 lines
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)drivers/pci/quirks.c +1
-0@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x98e4 , quirk_no_ats); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x6900 , quirk_no_ats);#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ Signature · 1 line
2.21.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
[+cc] linux-kernel@, linux-pci@
10.04.2019 10:26, Nikolai Kostrigin пишет:
Quoted 8 lines
Hello!
10.04.2019 00:59, Bjorn Helgaas пишет:
Quoted 3 lines
[+cc Alex]
This claims to be a resend, but I don't see a previous posting.
For some reason, unknown to me, my previous letter didn't appear neither in linux-kernel@, nor in linux-pci@ mailing lists.
I'm sorry. Now it's obvious I didn't manage to turn HTML formatting off before.
Quoted 52 lines
Quoted 6 lines
There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:
I'm aware of that and it was Joerg who pointed me to the above thread when I asked him for help.
In my case both devices mentioned in quirks are present in a laptop. Unfortunately I have no hardware with standalone AMD R7 GPU (0102:6900) to check whether unpatched kernel would fail the same way with both ATS and IOMMU enabled.Quoted 35 lines · Nikolai Kostrigin
Quoted 9 lines
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.https://bugzilla.kernel.org/show_bug.cgi?id=194521
Signed-off-by Nikolai Kostrigin <[email protected]>Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.
Quoted 19 lines
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)drivers/pci/quirks.c +1
-0@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x98e4 , quirk_no_ats); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x6900 , quirk_no_ats);#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ Signature · 1 line
2.21.0Signature · 2 lines
Best regards, Nikolai Kostrigin
Signature · 2 lines
Best regards, Nikolai Kostrigin
Quoted 11 lines · Nikolai Kostrigin
10.04.2019 10:26, Nikolai Kostrigin пишет:
Quoted 8 lines
Hello!
10.04.2019 00:59, Bjorn Helgaas пишет:
Quoted 3 lines
[+cc Alex]
This claims to be a resend, but I don't see a previous posting.
For some reason, unknown to me, my previous letter didn't appear neither in linux-kernel@, nor in linux-pci@ mailing lists.
I'm sorry. Now it's obvious I didn't manage to turn HTML formatting off before.
No worries, happens to me all the time since gmail is so unfriendly about text-only email!
Bjorn
Quoted 18 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Tuesday, April 9, 2019 5:59 PM
To: Nikolai Kostrigin <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs[+cc Alex]
This claims to be a resend, but I don't see a previous posting.
There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:
Quoted 11 lines
https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0
@BN6PR12MB1652.namprd12.prod.outlook.com/Quoted 7 lines · Nikolai Kostrigin
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and system failure. Disable ATS on these devices to make them usable again with IOMMU enabled Thanks to Joerg Roedel <[email protected]> for help.
+ a few AMD people
Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.
Alex
Quoted 28 lines
Quoted 1 line
Signed-off-by Nikolai Kostrigin <[email protected]>Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.
drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4879
Quoted 13 lines
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0
-0
4700d24e5d55..abb2532e16bf100644drivers/pci/quirks.c +1
-0@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x98e4 , quirk_no_ats); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900,
quirk_no_ats);Quoted 6 lines
drivers/pci/quirks.c +0
-1#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ --2.21.0
Quoted 49 lines
-----Original Message-----
From: Deucher, Alexander
Sent: Wednesday, April 10, 2019 10:47 AM
To: Bjorn Helgaas <[email protected]>; Nikolai Kostrigin <[email protected]>; Suthikulpanit, Suravee ([email protected]) <[email protected]>; Lendacky, Thomas <[email protected]>; Kuehling, Felix ([email protected]) <[email protected]>; Koenig, Christian ([email protected]) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon R7 GPUsQuoted 10 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Tuesday, April 9, 2019 5:59 PM
To: Nikolai Kostrigin <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon
R7 GPUsQuoted 10 lines
[+cc Alex]
This claims to be a resend, but I don't see a previous posting.
There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:
https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0
Quoted 6 lines
@BN6PR12MB1652.namprd12.prod.outlook.com/
Quoted 3 lines · Nikolai Kostrigin
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and system failure. Disable ATS on these devices to make them usable again with IOMMU enabled Thanks
to
Quoted 4 lines
Quoted 4 lines
Joerg Roedel <[email protected]> for help.
+ a few AMD people
Quoted 14 lines
Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (
0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.
Reading through these bugs again it seems to be an issue with Stoney APUs, not the dGPU specifically. I think it would be better to disable ATS in general if a stoney based platform was detected rather than adding ATS quirks for devices then someone may put in a Stoney based platform. It also seems to be related to runtime pm on the dGPU. Disabling runtime pm also seem to fix the issue. On these systems runtime pm for the dGPU is controlled via ACPI (either ATPX or _PR3 depending on the platform). Maybe something doesn't get restored properly on runtime resume which cases the ATS issues?
Alex
Quoted 32 lines
Alex
Quoted 12 lines
Quoted 1 line
Signed-off-by Nikolai Kostrigin <[email protected]>Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.
Quoted 4 lines
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)Quoted 8 lines
drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4878
Quoted 8 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0
-0
4700d24e5d55..abb2532e16bf100644drivers/pci/quirks.c +0
-0@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4,
quirk_no_ats);Quoted 8 lines
drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4879
Quoted 1 line
drivers/pci/quirks.c +1
-0+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900,
quirk_no_ats);Quoted 6 lines
drivers/pci/quirks.c +0
-1#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ --2.21.0
Quoted 17 lines · Alexander
Quoted 16 lines
+ a few AMD people
Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (
0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.
Quoted 9 lines
Reading through these bugs again it seems to be an issue with Stoney APUs, not the dGPU specifically. I think it would be better to disable ATS in general if a stoney based platform was detected rather than adding ATS quirks for devices then someone may put in a Stoney based platform. It also seems to be related to runtime pm on the dGPU. Disabling runtime pm also seem to fix the issue. On these systems runtime pm for the dGPU is controlled via ACPI (either ATPX or
_PR3depending on the platform). Maybe something doesn't get restored properly on runtime resume which cases the ATS issues?
This seems all pretty much possible, but we lack the ability to debug this further on our side. So until we have a real root-cause with a more specific quirk that only targets systems with a broken sbios or whatever, we need to catch-all approach.
We can remove these quirks again when AMD sends more specific quirks upstream.
Regards,
Joerg
Quoted 9 lines · Nikolai Kostrigin
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.https://bugzilla.kernel.org/show_bug.cgi?id=194521
Signed-off-by Nikolai Kostrigin <[email protected]>
I helped Nikolai with tracking this issue down to ATS handling and suggested adding the quirk.
[+cc Alex, Suravee, Thomas, Felix, Christian]
Quoted 9 lines · Nikolai Kostrigin
ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.https://bugzilla.kernel.org/show_bug.cgi?id=194521
Signed-off-by Nikolai Kostrigin <[email protected]>
I applied this to pci/virtualization for v5.2 with Joerg's ack, a stable tag, and a note about Alex's suspicion that this may be system-specific and may also affect other devices.
Quoted 19 lines
--- drivers/pci/quirks.c | 1 + 1 file changed, 1 insertion(+)drivers/pci/quirks.c +1
-0@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)/* AMD Stoney platform GPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x98e4 , quirk_no_ats); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x6900 , quirk_no_ats);#endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ Signature · 1 line
2.21.0