Skip to contentSkip to conversation

PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

PATCH RESEND In review Nikolai Kostrigin 5 participants 10 messages Apr 8, 2019 – Apr 11, 2019 1 message not in the archive PCI

lore
Share & tools
lore
#0 lore raw

cover PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

This solves system hang and file system corruption for Stoney Ridge systems (with built-in Radeon R4/R5 + discreet AMD Radeon R7 GPU) Such system example: HP laptop 17-ak041ur (more HW info at https://bugzilla.kernel.org/show_bug.cgi?id=194521)

Nikolai Kostrigin (1):
PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
Signature · 1 line
2.21.0
#1 lore raw

PATCH RESEND 1/1 PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/quirks.c +1 -0
index 4700d24e5d55..abb2532e16bf 100644
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4, quirk_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900, quirk_no_ats);
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
Signature · 1 line
2.21.0
#2 lore raw

[+cc Alex]

This claims to be a resend, but I don't see a previous posting.

There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:

https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0@BN6PR12MB1652.namprd12.prod.outlook.com/

Quoted 9 lines · Nikolai Kostrigin

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.

Quoted 19 lines
---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/quirks.c +1 -0
index 4700d24e5d55..abb2532e16bf 100644
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4, quirk_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900, quirk_no_ats);
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
Signature · 1 line
2.21.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

[+cc] linux-kernel@, linux-pci@

10.04.2019 10:26, Nikolai Kostrigin пишет:

Quoted 8 lines

Hello!

10.04.2019 00:59, Bjorn Helgaas пишет:

Quoted 3 lines

[+cc Alex]

This claims to be a resend, but I don't see a previous posting.

For some reason, unknown to me, my previous letter didn't appear neither in linux-kernel@, nor in linux-pci@ mailing lists.

I'm sorry. Now it's obvious I didn't manage to turn HTML formatting off before.

Quoted 52 lines
Quoted 6 lines

There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:

https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0@BN6PR12MB1652.namprd12.prod.outlook.com/

I'm aware of that and it was Joerg who pointed me to the above thread when I asked him for help.
In my case both devices mentioned in quirks are present in a laptop. Unfortunately I have no hardware with standalone AMD R7 GPU (0102:6900) to check whether unpatched kernel would fail the same way with both ATS and IOMMU enabled.

Quoted 35 lines · Nikolai Kostrigin
Quoted 9 lines

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.

Quoted 19 lines
---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/quirks.c +1 -0
index 4700d24e5d55..abb2532e16bf 100644
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4, quirk_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900, quirk_no_ats);
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
Signature · 1 line
2.21.0
Signature · 2 lines
Best regards,
Nikolai Kostrigin
Signature · 2 lines
Best regards,
Nikolai Kostrigin
#5 lore raw
Quoted 11 lines · Nikolai Kostrigin

10.04.2019 10:26, Nikolai Kostrigin пишет:

Quoted 8 lines

Hello!

10.04.2019 00:59, Bjorn Helgaas пишет:

Quoted 3 lines

[+cc Alex]

This claims to be a resend, but I don't see a previous posting.

For some reason, unknown to me, my previous letter didn't appear neither in linux-kernel@, nor in linux-pci@ mailing lists.

I'm sorry. Now it's obvious I didn't manage to turn HTML formatting off before.

No worries, happens to me all the time since gmail is so unfriendly about text-only email!

Bjorn

#6 lore raw
In reply to Bjorn Helgaas
Quoted 18 lines

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Tuesday, April 9, 2019 5:59 PM
To: Nikolai Kostrigin <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

[+cc Alex]

This claims to be a resend, but I don't see a previous posting.

There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:

Quoted 11 lines

https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0
@BN6PR12MB1652.namprd12.prod.outlook.com/

Quoted 7 lines · Nikolai Kostrigin

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and system failure. Disable ATS on these devices to make them usable again with IOMMU enabled Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

+ a few AMD people

Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.

Alex

Quoted 28 lines
Quoted 1 line

Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.

drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4879

Quoted 13 lines
---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0

4700d24e5d55..abb2532e16bf 100644

drivers/pci/quirks.c +1 -0
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4, quirk_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900,

quirk_no_ats);

Quoted 6 lines
drivers/pci/quirks.c +0 -1
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
--

2.21.0

#7 lore raw
Quoted 49 lines

-----Original Message-----
From: Deucher, Alexander
Sent: Wednesday, April 10, 2019 10:47 AM
To: Bjorn Helgaas <[email protected]>; Nikolai Kostrigin <[email protected]>; Suthikulpanit, Suravee ([email protected]) <[email protected]>; Lendacky, Thomas <[email protected]>; Kuehling, Felix ([email protected]) <[email protected]>; Koenig, Christian ([email protected]) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon R7 GPUs

Quoted 10 lines

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Tuesday, April 9, 2019 5:59 PM
To: Nikolai Kostrigin <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH RESEND 1/1] PCI: Add ATS-disable quirk for AMD Radeon
R7 GPUs

Quoted 10 lines

[+cc Alex]

This claims to be a resend, but I don't see a previous posting.

There *was* discussion when the quirk was added two years ago for a different device. As part of that, Alex thought only that device would be affected and ATS was validated on other GPUs:

https://lore.kernel.org/lkml/BN6PR12MB165278346BE8A76B1E4412AFF7EA0

Quoted 6 lines

@BN6PR12MB1652.namprd12.prod.outlook.com/

Quoted 3 lines · Nikolai Kostrigin

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and system failure. Disable ATS on these devices to make them usable again with IOMMU enabled Thanks

to

Quoted 4 lines
Quoted 4 lines

Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

+ a few AMD people

Quoted 14 lines

Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.

Reading through these bugs again it seems to be an issue with Stoney APUs, not the dGPU specifically. I think it would be better to disable ATS in general if a stoney based platform was detected rather than adding ATS quirks for devices then someone may put in a Stoney based platform. It also seems to be related to runtime pm on the dGPU. Disabling runtime pm also seem to fix the issue. On these systems runtime pm for the dGPU is controlled via ACPI (either ATPX or _PR3 depending on the platform). Maybe something doesn't get restored properly on runtime resume which cases the ATS issues?

Alex

Quoted 32 lines

Alex

Quoted 12 lines
Quoted 1 line

Joerg, I'm happy to merge this if you would review or ack it. I don't know enough to conclude that this is the root cause. It'd be nice to have an actual AMD erratum. Maybe it would even have a list of affected devices so we could get them all at once so people wouldn't have to trip over them one by one.

Quoted 4 lines
---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 8 lines

drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4878

Quoted 8 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0

4700d24e5d55..abb2532e16bf 100644

drivers/pci/quirks.c +0 -0
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4,

quirk_no_ats);

Quoted 8 lines

drivers/pci/quirks.c static void quirk_no_ats(struct pci_dev *pdev) · line 4879

Quoted 1 line
drivers/pci/quirks.c +1 -0
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900,

quirk_no_ats);

Quoted 6 lines
drivers/pci/quirks.c +0 -1
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
--

2.21.0

#8 lore raw
Quoted 17 lines · Alexander
Quoted 16 lines

+ a few AMD people

Seeing this bug makes it more clear. I don't think this is a problem with the GPU. I think it's a problem with either the sbios or iommu. I think the original quirk added for stoney (0x98e4) is probably wrong as well. I suspect we need a quirk for a particular laptop or sbios versions. We validated ATS extensively with Carrizo based systems (the system in the bug report above is Carrizo based) since it is the basis of our ROCm support on APUs. We have also been involved in tons of Linux OEM preloads with both Carrizo and Stoney based APUs in combination with TOPAZ dGPUs (0x6900) and haven't seen this issue in those programs. We also have TOPAZ dGPUs used in OEM programs with Intel chipsets and haven't seen the issue. I suspect since windows does not use the IOMMU by default, the sbios settings may not be well validated on certain windows only skus. I'd rather make these DMI matches or something like that for the platform or at the very least match the SSIDs as well.

Quoted 9 lines

Reading through these bugs again it seems to be an issue with Stoney APUs, not the dGPU specifically. I think it would be better to disable ATS in general if a stoney based platform was detected rather than adding ATS quirks for devices then someone may put in a Stoney based platform. It also seems to be related to runtime pm on the dGPU. Disabling runtime pm also seem to fix the issue. On these systems runtime pm for the dGPU is controlled via ACPI (either ATPX or _PR3 depending on the platform). Maybe something doesn't get restored properly on runtime resume which cases the ATS issues?

This seems all pretty much possible, but we lack the ability to debug this further on our side. So until we have a real root-cause with a more specific quirk that only targets systems with a broken sbios or whatever, we need to catch-all approach.

We can remove these quirks again when AMD sends more specific quirks upstream.

Regards,

Joerg

Acked-by on v1 1/1
#9 lore raw
In reply to Nikolai Kostrigin
Quoted 9 lines · Nikolai Kostrigin

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

I helped Nikolai with tracking this issue down to ATS handling and suggested adding the quirk.

#10 lore raw
In reply to Nikolai Kostrigin

[+cc Alex, Suravee, Thomas, Felix, Christian]

Quoted 9 lines · Nikolai Kostrigin

ATS is broken on this hardware (at least for Stoney Ridge based laptop) and causes IOMMU stalls and
system failure. Disable ATS on these devices to make them usable again with IOMMU enabled
Thanks to Joerg Roedel <[email protected]> for help.

https://bugzilla.kernel.org/show_bug.cgi?id=194521

I applied this to pci/virtualization for v5.2 with Joerg's ack, a stable tag, and a note about Alex's suspicion that this may be system-specific and may also affect other devices.

Quoted 19 lines
---
 drivers/pci/quirks.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/quirks.c +1 -0
index 4700d24e5d55..abb2532e16bf 100644
@@ -4876,6 +4876,7 @@ static void quirk_no_ats(struct pci_dev *pdev)
 
 /* AMD Stoney platform GPU */
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x98e4, quirk_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x6900, quirk_no_ats);
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */
Signature · 1 line
2.21.0