PCI: Blacklist AMD Stoney GPU devices for ATS
PATCH 1/1 PCI: Blacklist AMD Stoney GPU devices for ATS
From: Joerg Roedel <[email protected]>
ATS is broken on these devices. Under invalidation load, the GPU does not reply to invalidations anymore, causing Completion-wait loop timeouts on the AMD IOMMU driver side. Fix it by not enabling ATS on these devices.
Note that below mentioned commit is not broken, it just triggers the issue because it might cause invalidation storms on devices.
--- drivers/pci/ats.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/pci/ats.c +8 -0
@@ -17,10 +17,18 @@#include "pci.h" +static const struct pci_device_id broken_ats_tbl[] = { + {PCI_DEVICE (PCI_VENDOR_ID_AMD ,0x98e4 ) },/* AMD Stoney GPU part */ + {0 } +}; +void pci_ats_init (struct pci_dev *dev) {int pos; +if (pci_match_id (broken_ats_tbl, dev)) +return ; + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_ATS );if (!pos)return ;
Signature · 1 line
1.9.1
Quoted 5 lines
-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 8:17 AM
To: Bjorn Helgaas
Cc: [email protected]; [email protected]; Joerg Roedel=
;
Quoted 14 lines
Daniel Drake; Deucher, Alexander
Subject: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATS =20
From: Joerg Roedel <[email protected]>
=20
ATS is broken on these devices. Under invalidation load, the GPU does not reply to invalidations anymore, causing Completion-wait loop timeouts on the AMD IOMMU driver side. Fix it by not enabling ATS on these devices.
=20
Note that below mentioned commit is not broken, it just triggers the issue because it might cause invalidation storms on devices.
=20
Quoted 5 lines
Fixes b1516a14657a ('iommu/amd: Implement flush queue')Reported-by Daniel Drake <[email protected]>Cc Daniel Drake <[email protected]>Cc Alexander Deucher <[email protected]>Signed-off-by Joerg Roedel <[email protected]>
Did you see Arindam's patch from yesterday[1]? Not sure which is the prope= r fix, maybe both?
Alex
[1] - https://lists.freedesktop.org/archives/amd-gfx/2017-March/006862.html
Quoted 30 lines
--- drivers/pci/ats.c | 8 ++++++++ 1 file changed, 8 insertions(+)=20
drivers/pci/ats.c +0
-0@@ -17,10 +17,18 @@=20
#include "pci.h"
=20drivers/pci/ats.c +2
-0+static const struct pci_device_id broken_ats_tbl[] =3D { + {PCI_DEVICE (PCI_VENDOR_ID_AMD ,0x98e4 ) },/* AMD Stoney GPU part */
drivers/pci/ats.c +3
-0+ { 0 } +}; +void pci_ats_init (struct pci_dev *dev) {int pos;=20
drivers/pci/ats.c +3
-1+if (pci_match_id (broken_ats_tbl, dev)) +return ; + pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ATS);if (!pos)return ; --1.9.1
Quoted 19 lines · Alexander
Quoted 19 lines
-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 8:17 AM
To: Bjorn Helgaas
Cc: [email protected]; [email protected]; Joerg Roedel; Daniel Drake; Deucher, Alexander
Subject: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATSFrom: Joerg Roedel <[email protected]>
ATS is broken on these devices. Under invalidation load, the GPU does not reply to invalidations anymore, causing Completion-wait loop timeouts on the AMD IOMMU driver side. Fix it by not enabling ATS on these devices.
Note that below mentioned commit is not broken, it just triggers the issue because it might cause invalidation storms on devices.
Quoted 7 lines
Quoted 5 lines
Fixes b1516a14657a ('iommu/amd: Implement flush queue')Reported-by Daniel Drake <[email protected]>Cc Daniel Drake <[email protected]>Cc Alexander Deucher <[email protected]>Signed-off-by Joerg Roedel <[email protected]>Did you see Arindam's patch from yesterday[1]? Not sure which is the proper fix, maybe both?
Arindam's patch makes sense on its own, but not as a fix for this issue. It lowers the invalidation load on the GPU, but there are still ways to trigger a high invalidation rate on the device. So it might hide the issue, but not fix it.
We need to disable ATS on the device if it doesn't work reliably.
Joerg
Quoted 36 lines
-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 4:29 PM
To: Deucher, Alexander
Cc: 'Joerg Roedel'; Bjorn Helgaas; [email protected]; linux- [email protected]; Daniel Drake; Nath, Arindam
Subject: Re: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATS =20Quoted 5 lines · Alexander
Quoted 5 lines
-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 8:17 AM
To: Bjorn Helgaas
Cc: [email protected]; [email protected]; JoergRoedel;
Quoted 14 lines
Quoted 14 lines
Daniel Drake; Deucher, Alexander
Subject: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATSFrom: Joerg Roedel <[email protected]>
ATS is broken on these devices. Under invalidation load, the GPU does not reply to invalidations anymore, causing Completion-wait loop timeouts on the AMD IOMMU driver side. Fix it by not enabling ATS on these devices.
Note that below mentioned commit is not broken, it just triggers the issue because it might cause invalidation storms on devices.
Quoted 7 lines
Quoted 5 lines
Fixes b1516a14657a ('iommu/amd: Implement flush queue')Reported-by Daniel Drake <[email protected]>Cc Daniel Drake <[email protected]>Cc Alexander Deucher <[email protected]>Signed-off-by Joerg Roedel <[email protected]>Did you see Arindam's patch from yesterday[1]? Not sure which is the
Quoted 8 lines
proper fix, maybe both?
=20
Arindam's patch makes sense on its own, but not as a fix for this issue. It lowers the invalidation load on the GPU, but there are still ways to trigger a high invalidation rate on the device. So it might hide the issue, but not fix it.
=20
We need to disable ATS on the device if it doesn't work reliably.
The question is, could the problem stem from flushing an entity that didn't=
request it, or should that not matter? I guess it shouldn't matter otherw=
ise we'd see this on other platforms like Carrizo as well.
Alex
Quoted 4 lines · Alexander
The question is, could the problem stem from flushing an entity that didn't request it, or should that not matter? I guess it shouldn't matter otherwise we'd see this on other platforms like Carrizo as well.
What do you mean by "didn't request it"? The IOMMU driver only sends io/tlb invalidations to devices it enabled ATS on.
Joerg
Quoted 9 lines
-----Original Message-----
From: 'Joerg Roedel' [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 4:56 PM
To: Deucher, Alexander
Cc: 'Joerg Roedel'; Bjorn Helgaas; [email protected]; linux- [email protected]; Daniel Drake; Nath, Arindam
Subject: Re: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATS =20
On Tue, Mar 28, 2017 at 08:37:01PM +0000, Deucher, Alexander wrote:
Quoted 7 lines
Quoted 4 lines
The question is, could the problem stem from flushing an entity that didn't request it, or should that not matter? I guess it shouldn't matter otherwise we'd see this on other platforms like Carrizo as well.
=20
What do you mean by "didn't request it"? The IOMMU driver only sends io/tlb invalidations to devices it enabled ATS on.
If I understand Arindam's patch correctly, it only flushes TLB entries for = domains in the flush queue whereas the previous behavior was to flush all d= omains. If there was no TLB flush in the queue for that domain, could flus= hing it cause a problem? Sorry, I'm not too familiar with the IOMMU code o= r ATS.
Alex
Quoted 4 lines · Alexander
If I understand Arindam's patch correctly, it only flushes TLB entries for domains in the flush queue whereas the previous behavior was to flush all domains. If there was no TLB flush in the queue for that domain, could flushing it cause a problem?
No, that can't cause a problem. An io/tlb flush for the device is just a message that the device should invalidate its own tlb. The device can't know and doesn't need to know whether the page-tables it used to fill the tlb really changed.
As it looks, the problem we are seeing here is that we are sending a large amount of these requests to the GPU device, and wait for its completion every time. This shouldn't be a problem for ATS devices, but the GPU here seems to fail at some point and doesn't answer to the invalidation request anymore, causing the completion-wait loop timeouts.
Arindam's patch makes the high flush-frequency less likely, but it can still happen, depending on how the GPU is used. So its the best to keep ATS disabled on the device as it doesn't work correctly and we risk running in the same problem again when we leave it enabled and just make the trigger less likely.
Joerg
Quoted 14 lines
-----Original Message-----
From: 'Joerg Roedel' [mailto:[email protected]]
Sent: Wednesday, March 29, 2017 3:56 AM
To: Deucher, Alexander
Cc: 'Joerg Roedel'; Bjorn Helgaas; [email protected]; linux- [email protected]; Daniel Drake; Nath, Arindam
Subject: Re: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATSQuoted 4 lines · Alexander
If I understand Arindam's patch correctly, it only flushes TLB entries for domains in the flush queue whereas the previous behavior was to flush all domains. If there was no TLB flush in the queue for that domain, could flushing it cause a problem?
Quoted 4 lines
No, that can't cause a problem. An io/tlb flush for the device is just a message that the device should invalidate its own tlb. The device can't know and doesn't need to know whether the page-tables it used to fill the tlb really changed.
Joerg, as per my limited understanding of ATS, the ATC will respond to inva= lidation requests after making sure there are no in-flight DMA transactions=
with the address requested by IOMMU to be invalidated. Now since the IOMMU= was sending invalidate command to GPU even though there was no explicit pa=
ge unmapping request from the graphics subsystem, we _might_ end up in a si= tuation where the ATC takes longer than the invalidation timeout to respond=
to IOMMU.
With the patch I provided, since only those domains who have actually reque= sted for unmapping pages have been added to the flush queue, we send TLB in= validation commands to only those specific domains. This avoids sending inv= alidation command to GPU ATC every single flush.
I do agree that the Stoney might have some issue which causes it not to be = able complete the invalidation command in time since we are not observing t= he issue on CZ and other ASICs.
Thanks,
Arindam
Quoted 15 lines
As it looks, the problem we are seeing here is that we are sending a large amount of these requests to the GPU device, and wait for its completion every time. This shouldn't be a problem for ATS devices, but the GPU here seems to fail at some point and doesn't answer to the invalidation request anymore, causing the completion-wait loop timeouts.
Arindam's patch makes the high flush-frequency less likely, but it can still happen, depending on how the GPU is used. So its the best to keep ATS disabled on the device as it doesn't work correctly and we risk running in the same problem again when we leave it enabled and just make the trigger less likely.
Joerg
Hi Arindam,
Quoted 7 lines · Arindam
Joerg, as per my limited understanding of ATS, the ATC will respond to invalidation requests after making sure there are no in-flight DMA transactions with the address requested by IOMMU to be invalidated. Now since the IOMMU was sending invalidate command to GPU even though there was no explicit page unmapping request from the graphics subsystem, we _might_ end up in a situation where the ATC takes longer than the invalidation timeout to respond to IOMMU.
The maximum wait-time in the loop is 100ms. This should be more than enough for the ATC to complete any in-flight transaction and flush its internal TLB.
If that is not enough, there is almost certainly something wrong with the hardware.
Joerg
Quoted 20 lines
-----Original Message-----
From: 'Joerg Roedel' [mailto:[email protected]]
Sent: Wednesday, March 29, 2017 3:12 PM
To: Nath, Arindam
Cc: Deucher, Alexander; 'Joerg Roedel'; Bjorn Helgaas; linux- [email protected]; [email protected]; Daniel Drake
Subject: Re: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATSHi Arindam,
Quoted 4 lines · Arindam
Joerg, as per my limited understanding of ATS, the ATC will respond to invalidation requests after making sure there are no in-flight DMA transactions with the address requested by IOMMU to be invalidated. Now since the IOMMU was sending invalidate command to GPU even
though
Quoted 3 lines
there was no explicit page unmapping request from the graphics subsystem, we _might_ end up in a situation where the ATC takes longer than the invalidation timeout to respond to IOMMU.
Quoted 6 lines
The maximum wait-time in the loop is 100ms. This should be more than enough for the ATC to complete any in-flight transaction and flush its internal TLB.
If that is not enough, there is almost certainly something wrong with the hardware.
Actually what you said is correct. Before creating the patch, I had experim= ented with 1s and 10s invalidation timeout values, but none of them helped.
Thanks,
Arindam
Quoted 4 lines
Joerg
Deucher, Alexander
Acked-byQuoted 22 lines
-----Original Message-----
From: 'Joerg Roedel' [mailto:[email protected]]
Sent: Tuesday, March 28, 2017 6:26 PM
To: Deucher, Alexander
Cc: 'Joerg Roedel'; Bjorn Helgaas; [email protected]; linux- [email protected]; Daniel Drake; Nath, Arindam
Subject: Re: [PATCH] PCI: Blacklist AMD Stoney GPU devices for ATS =20Quoted 4 lines · Alexander
If I understand Arindam's patch correctly, it only flushes TLB entries for domains in the flush queue whereas the previous behavior was to flush all domains. If there was no TLB flush in the queue for that domain, could flushing it cause a problem?
=20
No, that can't cause a problem. An io/tlb flush for the device is just a message that the device should invalidate its own tlb. The device can't know and doesn't need to know whether the page-tables it used to fill the tlb really changed.
=20
As it looks, the problem we are seeing here is that we are sending a large amount of these requests to the GPU device, and wait for its completion every time. This shouldn't be a problem for ATS devices, but
Quoted 8 lines
the GPU here seems to fail at some point and doesn't answer to the invalidation request anymore, causing the completion-wait loop timeouts. =20
Arindam's patch makes the high flush-frequency less likely, but it can still happen, depending on how the GPU is used. So its the best to keep ATS disabled on the device as it doesn't work correctly and we risk running in the same problem again when we leave it enabled and just make the trigger less likely.
Thanks for clarifying. The patch is:
Hi Joerg,
Quoted 29 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
ATS is broken on these devices. Under invalidation load, the GPU does not reply to invalidations anymore, causing Completion-wait loop timeouts on the AMD IOMMU driver side. Fix it by not enabling ATS on these devices.
Note that below mentioned commit is not broken, it just triggers the issue because it might cause invalidation storms on devices.
Fixes b1516a14657a ('iommu/amd: Implement flush queue')Reported-by Daniel Drake <[email protected]>Cc Daniel Drake <[email protected]>Cc Alexander Deucher <[email protected]>Signed-off-by Joerg Roedel <[email protected]>--- drivers/pci/ats.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/pci/ats.c +1
-0@@ -17,10 +17,18 @@#include "pci.h" +static const struct pci_device_id broken_ats_tbl[] = {
drivers/pci/ats.c line 30
Quoted 10 lines
drivers/pci/ats.c +6
-0+ { PCI_DEVICE(PCI_VENDOR_ID_AMD, 0x98e4) }, /* AMD Stoney GPU part */ + { 0 } +}; +void pci_ats_init (struct pci_dev *dev) {int pos; +if (pci_match_id (broken_ats_tbl, dev)) +return ;
This is fine functionally, but from a stylistic point of view, I guess I would prefer to have it implemented in drivers/pci/quirks.c just to have some consistency in how we work around device defects.
Quoted 6 lines
drivers/pci/ats.c +0
-0pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_ATS );if (!pos)return ;Signature · 1 line
1.9.1
Hi Bjorn,
Quoted 2 lines · Bjorn Helgaas
drivers/pci/ats.c line 21
Quoted 2 lines
drivers/pci/ats.c +2
-0+static const struct pci_device_id broken_ats_tbl[] = { + {PCI_DEVICE (PCI_VENDOR_ID_AMD ,0x98e4 ) },/* AMD Stoney GPU part */
Just found out that the affected GPU uses PCI_VENDOR_ID_ATI, fixed that too.
Quoted 5 lines
Quoted 5 lines
drivers/pci/ats.c +3
-0+ { 0 } +}; +void pci_ats_init (struct pci_dev *dev)
Quoted 8 lines
drivers/pci/ats.c line 30
Quoted 4 lines
drivers/pci/ats.c +2
-0int pos; +if (pci_match_id (broken_ats_tbl, dev)) +return ;This is fine functionally, but from a stylistic point of view, I guess I would prefer to have it implemented in
drivers/pci/quirks.cjust to have some consistency in how we work around device defects.
Done, I send a new patch shortly.
Joerg