PCI: Reprogram bridge prefetch registers on resume
Subsystems (3)
PATCH v2 1/1 PCI: Reprogram bridge prefetch registers on resume
On 38+ Intel-based Asus products, the nvidia GPU becomes unusable after S3 suspend/resume. The affected products include multiple generations of nvidia GPUs and Intel SoCs. After resume, nouveau logs many errors such as:
fifo: fault 00 [READ] at 0000005555555000 engine 00 [GR] client 04 [HUB/FE] reason 4a [] on channel -1 [007fa91000 unknown]
DRM: failed to idle channel 0 [DRM]
Similarly, the nvidia proprietary driver also fails after resume (black screen, 100% CPU usage in Xorg process). We shipped a sample to Nvidia for diagnosis, and their response indicated that it's a problem with the parent PCI bridge (on the Intel SoC), not the GPU.
Runtime suspend/resume works fine, only S3 suspend is affected.
We found a workaround: on resume, rewrite the Intel PCI bridge 'Prefetchable Base Upper 32 Bits' register (PCI_PREF_BASE_UPPER32). In the cases that I checked, this register has value 0 and we just have to rewrite that value.
Linux already saves and restores PCI config space during suspend/resume, but this register was being skipped because upon resume, it already has value 0 (the correct, pre-suspend value).
Intel appear to have previously acknowledged this behaviour and the requirement to rewrite this register.
https://bugzilla.kernel.org/show_bug.cgi?id=116851#c23
Based on that, rewrite the prefetch register values even when that appears unnecessary.
We have confirmed this solution on all the affected models we have in-hands (X542UQ, UX533FD, X530UN, V272UN).
Additionally, this solves an issue where r8169 MSI-X interrupts were broken after S3 suspend/resume on Asus X441UAR. This issue was recently worked around in commit 7bb05b85bc2d ("r8169: don't use MSI-X on RTL8106e"). It also fixes the same issue on RTL6186evl/8111evl on an Aimfor-tech laptop that we had not yet patched. I suspect it will also fix the issue that was worked around in commit 7c53a722459c ("r8169:
don't use MSI-X on RTL8168g").
Thomas Martitz reports that this change also solves an issue where the AMD Radeon Polaris 10 GPU on the HP Zbook 14u G5 is unresponsive after S3 suspend/resume.
---
Notes:
Replaces patch:
PCI: add prefetch quirk to work around Asus/Nvidia suspend issues
Some of the more verbose info was moved to bugzilla:
https://bugzilla.kernel.org/show_bug.cgi?id=201069
This patch is aimed at v4.19 (and maybe v4.18-stable); we may follow up with more intrusive improvements for v4.20+.
v2: reimplement the register restore within the existing
pci_restore_config_space() code.
drivers/pci/pci.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)
drivers/pci/pci.c +21 -10
@@ -1289,13 +1289,15 @@ int pci_save_state(struct pci_dev *dev)EXPORT_SYMBOL (pci_save_state);static void pci_restore_config_dword (struct pci_dev *pdev,int offset, -u32 saved_val,int retry) +u32 saved_val,int retry,bool force) {u32 val; -pci_read_config_dword (pdev, offset, &val); -if (val == saved_val) -return ; +if (!force) { +pci_read_config_dword (pdev, offset, &val); +if (val == saved_val) +return ; + }for (;;) { pci_dbg(pdev, "restoring config space at offset %#x (was %#x, writing %#x)\n",
@@ -1313,25 +1315,34 @@ static void pci_restore_config_dword(struct pci_dev *pdev, int offset, }static void pci_restore_config_space_range (struct pci_dev *pdev, -int start,int end,int retry) +int start,int end,int retry, +bool force) {int index;for (index = end; index >= start; index--)pci_restore_config_dword (pdev,4 * index, pdev->saved_config_space[index], - retry); + retry, force); }static void pci_restore_config_space (struct pci_dev *pdev) {if (pdev->hdr_type ==PCI_HEADER_TYPE_NORMAL ) { -pci_restore_config_space_range (pdev,10 ,15 ,0 ); +pci_restore_config_space_range (pdev,10 ,15 ,0 , false);/* Restore BARs before the command register. */ -pci_restore_config_space_range (pdev,4 ,9 ,10 ); -pci_restore_config_space_range (pdev,0 ,3 ,0 ); +pci_restore_config_space_range (pdev,4 ,9 ,10 , false); +pci_restore_config_space_range (pdev,0 ,3 ,0 , false); + }else if (pdev->hdr_type ==PCI_HEADER_TYPE_BRIDGE ) { +pci_restore_config_space_range (pdev,12 ,15 ,0 , false); +/* Force rewriting of prefetch registers to avoid +* S3 resume issues on Intel PCI bridges that occur when +* these registers are not explicitly written. +*/ +pci_restore_config_space_range (pdev,9 ,11 ,0 , true); +pci_restore_config_space_range (pdev,0 ,8 ,0 , false); }else { -pci_restore_config_space_range (pdev,0 ,15 ,0 ); +pci_restore_config_space_range (pdev,0 ,15 ,0 , false); } }
Signature · 6 lines
2.17.1 _______________________________________________ Nouveau mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/nouveau
A message that is not in the archive (sent off-list or not yet received): 20180912064523.9599-1-drake-6IF/jdPJHihWk0Htik3J/[email protected]
Rafael J. Wysocki
Reviewed-byQuoted 123 lines · Daniel Drake
On 38+ Intel-based Asus products, the nvidia GPU becomes unusable after S3 suspend/resume. The affected products include multiple generations of nvidia GPUs and Intel SoCs. After resume, nouveau logs many errors such as:
fifo: fault 00 [READ] at 0000005555555000 engine 00 [GR] client 04 [HUB/FE] reason 4a [] on channel -1 [
007fa91000unknown]
DRM: failed to idle channel 0 [DRM]Similarly, the nvidia proprietary driver also fails after resume (black screen, 100% CPU usage in Xorg process). We shipped a sample to Nvidia for diagnosis, and their response indicated that it's a problem with the parent PCI bridge (on the Intel SoC), not the GPU.
Runtime suspend/resume works fine, only S3 suspend is affected.
We found a workaround: on resume, rewrite the Intel PCI bridge 'Prefetchable Base Upper 32 Bits' register (
PCI_PREF_BASE_UPPER32). In the cases that I checked, this register has value 0 and we just have to rewrite that value.Linux already saves and restores PCI config space during suspend/resume, but this register was being skipped because upon resume, it already has value 0 (the correct, pre-suspend value).
Intel appear to have previously acknowledged this behaviour and the requirement to rewrite this register.
https://bugzilla.kernel.org/show_bug.cgi?id=116851#c23Based on that, rewrite the prefetch register values even when that appears unnecessary.
We have confirmed this solution on all the affected models we have in-hands (X542UQ, UX533FD, X530UN, V272UN).
Additionally, this solves an issue where r8169 MSI-X interrupts were broken after S3 suspend/resume on Asus X441UAR. This issue was recently worked around in commit
7bb05b85bc2d("r8169: don't use MSI-X on RTL8106e"). It also fixes the same issue on RTL6186evl/8111evl on an Aimfor-tech laptop that we had not yet patched. I suspect it will also fix the issue that was worked around in commit7c53a722459c("r8169:
don't use MSI-X on RTL8168g").Thomas Martitz reports that this change also solves an issue where the AMD Radeon Polaris 10 GPU on the HP Zbook 14u G5 is unresponsive after S3 suspend/resume.
Signed-off-by Daniel Drake <[email protected]>---
Notes:
Replaces patch:
PCI: add prefetch quirk to work around Asus/Nvidia suspend issuesSome of the more verbose info was moved to bugzilla:
https://bugzilla.kernel.org/show_bug.cgi?id=201069This patch is aimed at v4.19 (and maybe v4.18-stable); we may follow up with more intrusive improvements for v4.20+.
v2: reimplement the register restore within the existing
pci_restore_config_space()code.drivers/pci/pci.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)drivers/pci/pci.c +17
-9@@ -1289,13 +1289,15 @@ int pci_save_state(struct pci_dev *dev)EXPORT_SYMBOL (pci_save_state);static void pci_restore_config_dword (struct pci_dev *pdev,int offset, -u32 saved_val,int retry) +u32 saved_val,int retry,bool force) {u32 val; -pci_read_config_dword (pdev, offset, &val); -if (val == saved_val) -return ; +if (!force) { +pci_read_config_dword (pdev, offset, &val); +if (val == saved_val) +return ; + }for (;;) { pci_dbg(pdev, "restoring config space at offset %#x (was %#x, writing %#x)\n",@@ -1313,25 +1315,34 @@ static void pci_restore_config_dword(struct pci_dev *pdev, int offset, }static void pci_restore_config_space_range (struct pci_dev *pdev, -int start,int end,int retry) +int start,int end,int retry, +bool force) {int index;for (index = end; index >= start; index--)pci_restore_config_dword (pdev,4 * index, pdev->saved_config_space[index], - retry); + retry, force); }static void pci_restore_config_space (struct pci_dev *pdev) {if (pdev->hdr_type ==PCI_HEADER_TYPE_NORMAL ) { -pci_restore_config_space_range (pdev,10 ,15 ,0 ); +pci_restore_config_space_range (pdev,10 ,15 ,0 , false);/* Restore BARs before the command register. */ -pci_restore_config_space_range (pdev,4 ,9 ,10 ); -pci_restore_config_space_range (pdev,0 ,3 ,0 ); +pci_restore_config_space_range (pdev,4 ,9 ,10 , false); +pci_restore_config_space_range (pdev,0 ,3 ,0 , false); + }else if (pdev->hdr_type ==PCI_HEADER_TYPE_BRIDGE ) { +pci_restore_config_space_range (pdev,12 ,15 ,0 , false); +/* Force rewriting of prefetch registers to avoid +* S3 resume issues on Intel PCI bridges that occur when +* these registers are not explicitly written.
drivers/pci/pci.c static void pci_restore_config_dword(struct pci_dev *pdev, int offset,
Quoted 10 lines
drivers/pci/pci.c +4
-2+*/ +pci_restore_config_space_range (pdev,9 ,11 ,0 , true); +pci_restore_config_space_range (pdev,0 ,8 ,0 , false); }else { -pci_restore_config_space_range (pdev,0 ,15 ,0 ); +pci_restore_config_space_range (pdev,0 ,15 ,0 , false); } } --
Passing the extra bool around is somewhat clumsy, but I haven't found a cleaner way to do it, so
_______________________________________________
Nouveau mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau
Quoted 2 lines · Rafael J. Wysocki
Passing the extra bool around is somewhat clumsy, but I haven't found a cleaner way to do it, so
Thanks, according to your suggestion (which I plan to follow after this) we will remove this parameter for v4.20+ and have all the register writes be forced regardless of current value.
Daniel
Quoted 80 lines · Daniel Drake
On 38+ Intel-based Asus products, the nvidia GPU becomes unusable after S3 suspend/resume. The affected products include multiple generations of nvidia GPUs and Intel SoCs. After resume, nouveau logs many errors such as:
fifo: fault 00 [READ] at 0000005555555000 engine 00 [GR] client 04 [HUB/FE] reason 4a [] on channel -1 [
007fa91000unknown]
DRM: failed to idle channel 0 [DRM]Similarly, the nvidia proprietary driver also fails after resume (black screen, 100% CPU usage in Xorg process). We shipped a sample to Nvidia for diagnosis, and their response indicated that it's a problem with the parent PCI bridge (on the Intel SoC), not the GPU.
Runtime suspend/resume works fine, only S3 suspend is affected.
We found a workaround: on resume, rewrite the Intel PCI bridge 'Prefetchable Base Upper 32 Bits' register (
PCI_PREF_BASE_UPPER32). In the cases that I checked, this register has value 0 and we just have to rewrite that value.Linux already saves and restores PCI config space during suspend/resume, but this register was being skipped because upon resume, it already has value 0 (the correct, pre-suspend value).
Intel appear to have previously acknowledged this behaviour and the requirement to rewrite this register.
https://bugzilla.kernel.org/show_bug.cgi?id=116851#c23Based on that, rewrite the prefetch register values even when that appears unnecessary.
We have confirmed this solution on all the affected models we have in-hands (X542UQ, UX533FD, X530UN, V272UN).
Additionally, this solves an issue where r8169 MSI-X interrupts were broken after S3 suspend/resume on Asus X441UAR. This issue was recently worked around in commit
7bb05b85bc2d("r8169: don't use MSI-X on RTL8106e"). It also fixes the same issue on RTL6186evl/8111evl on an Aimfor-tech laptop that we had not yet patched. I suspect it will also fix the issue that was worked around in commit7c53a722459c("r8169:
don't use MSI-X on RTL8168g").Thomas Martitz reports that this change also solves an issue where the AMD Radeon Polaris 10 GPU on the HP Zbook 14u G5 is unresponsive after S3 suspend/resume.
Signed-off-by Daniel Drake <[email protected]>---
Notes:
Replaces patch:
PCI: add prefetch quirk to work around Asus/Nvidia suspend issuesSome of the more verbose info was moved to bugzilla:
https://bugzilla.kernel.org/show_bug.cgi?id=201069This patch is aimed at v4.19 (and maybe v4.18-stable); we may follow up with more intrusive improvements for v4.20+.
v2: reimplement the register restore within the existing
pci_restore_config_space()code.drivers/pci/pci.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)drivers/pci/pci.c +1
-2@@ -1289,13 +1289,15 @@ int pci_save_state(struct pci_dev *dev)EXPORT_SYMBOL (pci_save_state);static void pci_restore_config_dword (struct pci_dev *pdev,int offset, -u32 saved_val,int retry) +u32 saved_val,int retry,bool force) {u32 val; -pci_read_config_dword (pdev, offset, &val);
drivers/pci/pci.c int pci_save_state(struct pci_dev *dev) · line 1303
Quoted 10 lines
drivers/pci/pci.c +5
-2-if (val == saved_val) -return ; +if (!force) { +pci_read_config_dword (pdev, offset, &val); +if (val == saved_val) +return ; + }for (;;) { pci_dbg(pdev, "restoring config space at offset %#x (was %#x, writing %#x)\n",
"val" is uninitialized and used in this debug message.
To avoid adding another parameter and changing unrelated lines below, what about:
force = pdev->header_type == PCI_HEADER_TYPE_BRIDGE &&
0x24 <= start && start <= 0x2c;(or start == 0x28 || start == 0x24 || start == 0x28 if you want to be very explicit). This would reduce in a smaller diff for this temporary solution.
Kind regards,
Peter
Quoted 44 lines
drivers/pci/pci.c +15
-6@@ -1313,25 +1315,34 @@ static void pci_restore_config_dword(struct pci_dev *pdev, int offset, }static void pci_restore_config_space_range (struct pci_dev *pdev, -int start,int end,int retry) +int start,int end,int retry, +bool force) {int index;for (index = end; index >= start; index--)pci_restore_config_dword (pdev,4 * index, pdev->saved_config_space[index], - retry); + retry, force); }static void pci_restore_config_space (struct pci_dev *pdev) {if (pdev->hdr_type ==PCI_HEADER_TYPE_NORMAL ) { -pci_restore_config_space_range (pdev,10 ,15 ,0 ); +pci_restore_config_space_range (pdev,10 ,15 ,0 , false);/* Restore BARs before the command register. */ -pci_restore_config_space_range (pdev,4 ,9 ,10 ); -pci_restore_config_space_range (pdev,0 ,3 ,0 ); +pci_restore_config_space_range (pdev,4 ,9 ,10 , false); +pci_restore_config_space_range (pdev,0 ,3 ,0 , false); + }else if (pdev->hdr_type ==PCI_HEADER_TYPE_BRIDGE ) { +pci_restore_config_space_range (pdev,12 ,15 ,0 , false); +/* Force rewriting of prefetch registers to avoid +* S3 resume issues on Intel PCI bridges that occur when +* these registers are not explicitly written. +*/ +pci_restore_config_space_range (pdev,9 ,11 ,0 , true); +pci_restore_config_space_range (pdev,0 ,8 ,0 , false); }else { -pci_restore_config_space_range (pdev,0 ,15 ,0 ); +pci_restore_config_space_range (pdev,0 ,15 ,0 , false); } }Signature · 1 line
2.17.1