pci: prevent putting nvidia GPUs into lower device states on certain intel bridges
PATCH v5 1/1 pci: prevent putting nvidia GPUs into lower device states on certain intel bridges
Fixes state transitions of Nvidia Pascal GPUs from D3cold into higher device states.
v2: convert to pci_dev quirk
put a proper technical explanation of the issue as a in-code comment
v3: disable it only for certain combinations of intel and nvidia hardware
v4: simplify quirk by setting flag on the GPU itself
v5: restructure quirk to make it easier to add new IDs
fix whitespace issues
fix potential NULL pointer access
update the quirk documentation
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=205623
--- drivers/pci/pci.c | 7 ++++++ drivers/pci/quirks.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 1 + 3 files changed, 59 insertions(+)
drivers/pci/pci.c +7 -0
@@ -850,6 +850,13 @@ static int pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state) || (state == PCI_D2 && !dev->d2_support))return -EIO ; +/* +* Check if we have a bad combination of bridge controller and nvidia +* GPU, see quirk_broken_nv_runpm for more info +*/ +if (state !=PCI_D0 && dev->broken_nv_runpm) +return 0 ; +pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);/*
drivers/pci/quirks.c +51 -0
54 lines not on this page · raw message
include/linux/pci.h +1 -0
@@ -416,6 +416,7 @@ struct pci_dev { unsigned int __aer_firmware_first_valid:1; unsigned int __aer_firmware_first:1; unsigned int broken_intx_masking:1; /* INTx masking can't be used */ + unsigned int broken_nv_runpm:1; /* some combinations of intel bridge controller and nvidia GPUs break rtd3 */ unsigned int io_window_1k:1; /* Intel bridge 1K I/O windows */ unsigned int irq_managed:1; unsigned int has_secondary_link:1;
Signature · 6 lines
2.23.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 34 lines · Karol Herbst
Fixes state transitions of Nvidia Pascal GPUs from D3cold into higher device states.
v2: convert to
pci_devquirk
put a proper technical explanation of the issue as a in-code comment
v3: disable it only for certain combinations of intel and nvidia hardware
v4: simplify quirk by setting flag on the GPU itself
v5: restructure quirk to make it easier to add new IDs
fix whitespace issues
fix potential NULL pointer access
update the quirk documentationSigned-off-by Karol Herbst <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Lyude Paul <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Mika Westerberg <[email protected]>Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=205623
--- drivers/pci/pci.c | 7 ++++++ drivers/pci/quirks.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 1 + 3 files changed, 59 insertions(+)drivers/pci/pci.c +0
-0
drivers/pci/pci.c static int pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state) · line 858
Quoted 9 lines
drivers/pci/pci.c +6
-0@@ -850,6 +850,13 @@ static int pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state) || (state == PCI_D2 && !dev->d2_support))return -EIO ; +/* +* Check if we have a bad combination of bridge controller and nvidia +* GPU, see quirk_broken_nv_runpm for more info +*/ +if (state !=PCI_D0 && dev->broken_nv_runpm) +return 0 ;
The result of this change in the suspend-to-idle path will be leaving the device and its PCIe port in D0 while suspended, unless the device itself has power management methods in the ACPI tables (according to Mika that is not the case).
I don't think that this is desirable.
Quoted 9 lines
drivers/pci/pci.c +1
-0+pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);/* drivers/pci/quirks.c +0
-0
drivers/pci/quirks.c static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev) · line 5274
Quoted 7 lines
drivers/pci/quirks.c +4
-0@@ -5268,3 +5268,54 @@ static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev)DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,0x13b1 ,PCI_CLASS_DISPLAY_VGA ,8 , quirk_reset_lenovo_thinkpad_p50_nvgpu); + +/* +* Some Intel PCIe bridge controllers cause devices to not reappear doing a +* D0 -> D3hot -> D3cold -> D0 sequence.
This is inaccurate and not entirely fair AFAICS.
First off, what is a "PCIe bridge controller"? A PCIe root complex?
Second, I don't think that you really can blame hardware here, because the problem is related to AML (forcing a different code path in AML makes it go away, so the same hardware with different AML would work).
More precisely, the behavior of the kernel is not what is expected by AML associated with the PCIe port holding the device.
Quoted 2 lines
Skipping the intermediate D3hot step
+ * seems to make it work again.
Yes, but the change would need to cover both the PM-runtime and suspend-to-idle code paths.
Also it may be driver-induced rather than quirk-based.
Quoted 3 lines
+ *
+ * This leads to various manifestations of this issue:
+ * - AIML code execution hits an infinite loop (as the coe waits on device
Typo: coe -> code
Quoted 1 line
+ * memory to change).
Which AML code is this, the power-off part or power-on part? Is this AML code associated with the GPU or with the PCIe port holding it (I guess the latter from what Mika said)?
Also IIRC ACPICA has a mechanism to break infinite loops in AML by aborting the looping method after a timeout.
Quoted 4 lines
+ * - kernel crashes, as all PCI reads return -1, which most code isn't able
+ * to handle well enough.
+ * - sudden shutdowns, as the kernel identified an unrecoverable error after
+ * userspace tries to access the GPU.
IMO it would be enough to say that the GPU is not accessible after an attempt to remove power from it.
Quoted 6 lines
+ *
+ * In all cases dmesg will contain at least one line like this:
+ * 'nouveau 0000:01:00.0: Refused to change power state, currently in D3'
+ * followed by a lot of nouveau timeouts.
+ *
+ * ACPI code
Which ACPI code?
Quoted 1 line
writes bit
0x80to the not documented PCI register0x248of the
0x248 relative to what? A PCI bar (if so then which one) or the PCI config space (and which part of it if so)?
Quoted 1 line
+ * Intel PCIe bridge controller (
0x1901) in order to power down the GPU.
This doesn't seem accurate. It rather writes to this register to change the state of the PCIe link between the GPU and the PCIe port holding it, which is not the same as powering off.
Quoted 2 lines
+ * Nonetheless, there are other code paths inside the ACPI firmware which use
+ * other registers, which seem to work fine:
The meaning of the above is unclear.
Does "other" mean "alternative"?
Quoted 1 line
+ * -
0xbcbit0x20(publicly available documentation claims 'reserved')
A pointer to that documentation, please (yes, it should be included in the comment).
Quoted 3 lines
+ * -
0xb0bit0x10(link disable)
+ * Changing the conditions inside the firmware by poking into the relevant
+ * addresses does resolve the issue,
What does this mean, precisely?
Quoted 10 lines
but it seemed to be ACPI private memory
+ * and not any device accessible memory at all, so there is no portable way of
+ * changing the conditions.
+ *
+ * The only systems where this behavior can be seen are hybrid graphics laptops
+ * with a secondary Nvidia Maxwell, Pascal or Turing GPU. It cannot be ruled
+ * out that this issue only occurs in combination with listed Intel PCIe
+ * bridge controllers and the mentioned GPUs or if it's only a hw bug in the
+ * bridge controller.
+ */
So again, this is AML-related, so talking about hardware bugs at this point is premature at best.
Quoted 21 lines
drivers/pci/quirks.c +16
-0+ +static void quirk_broken_nv_runpm (struct pci_dev *dev) +{ +struct pci_dev *bridge =pci_upstream_bridge (dev); + +if (!bridge || bridge->vendor !=PCI_VENDOR_ID_INTEL ) +return ; + +switch (bridge->device) { +case 0x1901 : + dev->broken_nv_runpm =1 ; + } +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_BASE_CLASS_DISPLAY ,16 , + quirk_broken_nv_runpm);include/linux/pci.h +0
-0
include/linux/pci.h struct pci_dev {
Quoted 8 lines
include/linux/pci.h +1
-1@@ -416,6 +416,7 @@ struct pci_dev { unsigned int __aer_firmware_first_valid:1; unsigned int __aer_firmware_first:1; unsigned int broken_intx_masking:1; /* INTx masking can't be used */ + unsigned int broken_nv_runpm:1; /* some combinations of intel bridge controller and nvidia GPUs break rtd3 */ unsigned int io_window_1k:1; /* Intel bridge 1K I/O windows */ unsigned int irq_managed:1; unsigned int has_secondary_link:1; --
Based of what has been uncovered so far, I would propose the following way to address this issue:
(1) Introduce a "parent_d3cold" PCI device flag that would indicate to the PCI layer to leave the given device in D0 during suspend (both system-wide and runtime) and power-manage the parent instead as long as the parent PCIe port can be put into D3cold via ACPI (and as long as the downstream device itself has no ACPI PM).
(2) Set this new flag for the GPUs in question from the nouveau driver.
So what about this?