PCI/ASPM: Add back L1 PM Substate save and restore
PATCH 1/1 PCI/ASPM: Add back L1 PM Substate save and restore
Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.
The original revert mentioned that we restore L1 PM substate configuration even though ASPM L1 may already be enabled. This is due the fact that the pci_restore_aspm_l1ss_state() was called before pci_restore_pcie_state().
Try to enable this functionality again by:
1) Moving the restore happen after PCIe capability is restored
following PCIe r6.0, sec 5.5.4.
2) Following the PCIe spec more closely to restore L1 PM substate configuration (program enable bits last).
3) Adding denylist that skips the restoring on the ASUS system where the original regression happened, just in case.
---
Hi,
There are several systems out there that are not entering lower package C-states after the first suspend/resume cycle because we do not restore L1SS configuration as can be seen in the linked bugzilla entry. Although they are functioning fine they still empty the battery faster than the user may expect.
Apologies sending this out during merge window but I will be starting my vacation next week so wanted to get this out before, and possibly incorporate any changes during the week still.
drivers/pci/pci.c | 7 ++++ drivers/pci/pci.h | 4 +++ drivers/pci/pcie/aspm.c | 76 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+)
drivers/pci/pci.c +7 -0
@@ -1679,6 +1679,7 @@ int pci_save_state(struct pci_dev *dev)pci_save_dpc_state (dev);pci_save_aer_state (dev);pci_save_ptm_state (dev); +pci_save_aspm_l1ss_state (dev);return pci_save_vc_state (dev); }EXPORT_SYMBOL (pci_save_state);
@@ -1790,6 +1791,7 @@ void pci_restore_state(struct pci_dev *dev)pci_restore_vc_state (dev);pci_restore_rebar_state (dev);pci_restore_dpc_state (dev); +pci_restore_aspm_l1ss_state (dev);pci_restore_ptm_state (dev);pci_aer_clear_status (dev);
@@ -3474,6 +3476,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)if (error)pci_err (dev,"unable to allocate suspend buffer for LTR\n" ); + error =pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_L1SS , +2 *sizeof (u32)); +if (error) +pci_err (dev,"unable to allocate suspend buffer for ASPM-L1SS\n" ); +pci_allocate_vc_save_buffers (dev); }
drivers/pci/pci.h +4 -0
@@ -560,10 +560,14 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active);void pcie_aspm_init_link_state (struct pci_dev *pdev);void pcie_aspm_exit_link_state (struct pci_dev *pdev);void pcie_aspm_powersave_config_link (struct pci_dev *pdev); +void pci_save_aspm_l1ss_state (struct pci_dev *pdev); +void pci_restore_aspm_l1ss_state (struct pci_dev *pdev);#else static inline void pcie_aspm_init_link_state (struct pci_dev *pdev) { }static inline void pcie_aspm_exit_link_state (struct pci_dev *pdev) { }static inline void pcie_aspm_powersave_config_link (struct pci_dev *pdev) { } +static inline void pci_save_aspm_l1ss_state (struct pci_dev *pdev) { } +static inline void pci_restore_aspm_l1ss_state (struct pci_dev *pdev) { }#endif #ifdef CONFIG_PCIE_ECRC
drivers/pci/pcie/aspm.c +76 -0
88 lines not on this page · raw message
Signature · 1 line
2.40.1
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.
Hello Mika,
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in <https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)
Best Regards,
Thomas
Hi Thomas,
Quoted 8 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.
Quoted 6 lines
Hello Mika,
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)
Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.Hello Mika,
Quoted 8 lines
Quoted 4 lines
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.
To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.
If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."
Bjorn
Hi Bjorn,
Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.Hello Mika,
Quoted 8 lines
Quoted 4 lines
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.
Quoted 8 lines
To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.
If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."
Fair enough.
I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to /sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).
So when forcing S3 this means the system suspend and resume is done through the BIOS who is supposed to enable wakes and program the devices accordingly during and after S3 before the OS is given back the control, it might very well be that it already did something for the L1 configuration here before Linux (with this patch) reconfigured them and this is causing the problem.
@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.
Quoted 3 lines · Mika Westerberg
@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.
In fact, there is: Entering suspend-to-ram without setting /sys/power/mem_sleep to "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:
I just retested your patch with setting mem_sleep to "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.
I attached the output of dmidecode to the bugzilla entry mentioned above: <https://bugzilla.kernel.org/attachment.cgi?id=304494>
Best regards,
Thomas
Hi Thomas,
Quoted 11 lines · Thomas Witt
Quoted 3 lines · Mika Westerberg
@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.
In fact, there is: Entering suspend-to-ram without setting
/sys/power/mem_sleepto "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:
Thanks for the clarification.
Quoted 3 lines
I just retested your patch with setting
mem_sleepto "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.
I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
Quoted 2 lines
I attached the output of dmidecode to the bugzilla entry mentioned above:
<https://bugzilla.kernel.org/attachment.cgi?id=304494>
Thanks!
Quoted 17 lines · Mika Westerberg
Hi Thomas,
Quoted 11 lines · Thomas Witt
Quoted 3 lines · Mika Westerberg
@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.
In fact, there is: Entering suspend-to-ram without setting
/sys/power/mem_sleepto "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:Thanks for the clarification.
Quoted 8 lines
Quoted 3 lines
I just retested your patch with setting
mem_sleepto "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
One possibility is to check the package C-state residency like:
# cat /sys/kernel/debug/pmc_core/package_cstate_show
after system sleep and see if there is a change with the patch applied, as done here:
Quoted 4 lines · Mika Westerberg
I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With mem_sleep set to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.
So I guess I want to keep the "deep" setting in my cmdline.
BR
Thomas
Quoted 6 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg
I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
Quoted 6 lines
No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With
mem_sleepset to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.So I guess I want to keep the "deep" setting in my cmdline.
Okay thanks a lot for checking! Back to drawing board then...
Quoted 6 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg
I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
Quoted 6 lines
No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With
mem_sleepset to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.So I guess I want to keep the "deep" setting in my cmdline.
It's likely something is not entering the right state. You can try the following script to check what the cause may be.
https://github.com/intel/S0ixSelftestTool
David
Quoted 4 lines
BR
Thomas
Quoted 12 lines · David E. Box
Quoted 11 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg
I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?
No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With
mem_sleepset to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.
Quoted 6 lines
Quoted 1 line
So I guess I want to keep the "deep" setting in my cmdline.
It's likely something is not entering the right state. You can try the following script to check what the cause may be.
That would be useful insights, indeed.
@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):
- Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
- Program L1 SS before L1 enables
- Program L1 SS enables after rest of the fields in the capability
Applies on top of v6.4.
------8<-----8<-----8<-----8<-----
drivers/pci/pci.c +16 -2
@@ -1545,7 +1545,7 @@ static void pci_restore_pcie_state(struct pci_dev *dev) {int i =0 ;struct pci_cap_saved_state *save_state; -u16 *cap; +u16 *cap, val; save_state =pci_find_saved_cap (dev,PCI_CAP_ID_EXP );if (!save_state)
@@ -1560,7 +1560,14 @@ static void pci_restore_pcie_state(struct pci_dev *dev) cap = (u16 *)&save_state->cap.data[0 ];pcie_capability_write_word (dev,PCI_EXP_DEVCTL , cap[i++]); -pcie_capability_write_word (dev,PCI_EXP_LNKCTL , cap[i++]); +/* +* Restoring ASPM L1 substates has special requirements +* according to the PCIe spec 6.0. So we restore here only the +* LNKCTL register with the ASPM control field clear. ASPM will +* be restored in pci_restore_aspm_state(). +*/ + val = cap[i++] & ~PCI_EXP_LNKCTL_ASPMC ; +pcie_capability_write_word (dev,PCI_EXP_LNKCTL , val);pcie_capability_write_word (dev,PCI_EXP_SLTCTL , cap[i++]);pcie_capability_write_word (dev,PCI_EXP_RTCTL , cap[i++]);pcie_capability_write_word (dev,PCI_EXP_DEVCTL2 , cap[i++]);
@@ -1671,6 +1678,7 @@ int pci_save_state(struct pci_dev *dev)pci_save_ltr_state (dev);pci_save_dpc_state (dev);pci_save_aer_state (dev); +pci_save_aspm_state (dev);pci_save_ptm_state (dev);return pci_save_vc_state (dev); }
@@ -1784,6 +1792,7 @@ void pci_restore_state(struct pci_dev *dev)pci_restore_rebar_state (dev);pci_restore_dpc_state (dev);pci_restore_ptm_state (dev); +pci_restore_aspm_state (dev);pci_aer_clear_status (dev);pci_restore_aer_state (dev);
@@ -3467,6 +3476,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)if (error)pci_err (dev,"unable to allocate suspend buffer for LTR\n" ); + error =pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_L1SS , +2 *sizeof (u32)); +if (error) +pci_err (dev,"unable to allocate suspend buffer for ASPM-L1SS\n" ); +pci_allocate_vc_save_buffers (dev); }
drivers/pci/pci.h +4 -0
@@ -567,10 +567,14 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active);void pcie_aspm_init_link_state (struct pci_dev *pdev);void pcie_aspm_exit_link_state (struct pci_dev *pdev);void pcie_aspm_powersave_config_link (struct pci_dev *pdev); +void pci_save_aspm_state (struct pci_dev *pdev); +void pci_restore_aspm_state (struct pci_dev *pdev);#else static inline void pcie_aspm_init_link_state (struct pci_dev *pdev) { }static inline void pcie_aspm_exit_link_state (struct pci_dev *pdev) { }static inline void pcie_aspm_powersave_config_link (struct pci_dev *pdev) { } +static inline void pci_save_aspm_state (struct pci_dev *pdev) { } +static inline void pci_restore_aspm_state (struct pci_dev *pdev) { }#endif #ifdef CONFIG_PCIE_ECRC
drivers/pci/pcie/aspm.c +109 -0
121 lines not on this page · raw message
Signature · 1 line
2.39.2
Quoted 10 lines · Mika Westerberg
@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):
- Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
- Program L1 SS before L1 enables
- Program L1 SS enables after rest of the fields in the capability
Sadly, same as before. With s2idle, power consumption stays up, but suspend/resume works, with deep it does not correctly suspend the PCI devices.
Hi Thomas,
Quoted 7 lines · Thomas Witt
Quoted 7 lines · Mika Westerberg
Quoted 6 lines
Quoted 6 lines
@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
- Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
- Program L1 SS before L1 enables
- Program L1 SS enables after rest of the fields in the capabilitySadly, same as before. With s2idle, power consumption stays up, but suspend/resume works, with deep it does not correctly suspend the PCI devices.
Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.
The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.
https://bugzilla.kernel.org/attachment.cgi?id=304553
Please let me know if it works. Thanks.
David
Quoted 6 lines · David E. Box
Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.
Quoted 8 lines
The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.
https://bugzilla.kernel.org/attachment.cgi?id=304553
Please let me know if it works. Thanks.
David
Hi David,
I tried your patch, and I see no difference from Mika's. Still not coming back from suspend.
Thomas
Hi Thomas,
Quoted 12 lines · Thomas Witt
Quoted 11 lines · David E. Box
Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.
The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.
Quoted 8 lines
Quoted 3 lines
Please let me know if it works. Thanks.
David
Hi David,
I tried your patch, and I see no difference from Mika's. Still not coming back from suspend.
Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.
Quoted 5 lines · Mika Westerberg
Hi Thomas,
Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.
Hi Mika,
I did, with -s it gives
Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.
with -r on
Your system did not achieve the runtime PC10 state during screen ON
additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")
with -r off, it tries xset which fails due to a lack of xserver.
Thomas
Hi Thomas,
Quoted 11 lines · Thomas Witt
Quoted 5 lines · Mika Westerberg
Hi Thomas,
Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.
Hi Mika,
I did, with -s it gives
Quoted 7 lines
Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.with -r on
Your system did not achieve the runtime PC10 state during screen ON
Thanks for trying. Did you change the "mem_sleep" back to "s2idle" before you run the script?
Quoted 3 lines
additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")
@David, do you know what might be the issue?
Quoted 1 line
with -r off, it tries xset which fails due to a lack of xserver.
You do have graphics running right? I mean i915 driver is enabled and all the firmwares are in place (should come with the distro). I'm asking because s2idle typically requires that graphics and pretty much all the devices on the SoC have a driver and the accompanying firmwares, and that they enter D3 properly.
Quoted 18 lines · Mika Westerberg
Hi Thomas,
Quoted 15 lines · Thomas Witt
Quoted 5 lines · Mika Westerberg
Hi Thomas,
Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.
Hi Mika,
I did, with -s it gives
Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.
Quoted 6 lines
Quoted 3 lines
with -r on
Your system did not achieve the runtime PC10 state during screen ON
Thanks for trying. Did you change the "
mem_sleep" back to "s2idle" before you run the script?
The script checks the FADT to determine if the system supports S0ix and it found that it didn't which is weird since Thomas is setting "mem_sleep" to "deep" from the default "s2idle" which is based on this bit.
Here are the commands to check it.
sudo acpidump -n FADT -b
iasl -d facp.dat
grep "Low Power S0 Idle" facp.dsl
Thomas, can you confirm what the value of mem_sleep is when you boot and run the above to confirm what your hardware supports?
Quoted 6 lines
Quoted 4 lines
additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")@David, do you know what might be the issue?
Yes. The latest kernel changes the output of the ltr_show command by adding a PMC number prefix (since Meteor Lake has more than one PMC now). The script is erring on the unexpected colon. We'll get this fixed.
David
Quoted 8 lines
Quoted 2 lines
with -r off, it tries xset which fails due to a lack of xserver.
You do have graphics running right? I mean i915 driver is enabled and all the firmwares are in place (should come with the distro). I'm asking because s2idle typically requires that graphics and pretty much all the devices on the SoC have a driver and the accompanying firmwares, and that they enter D3 properly.
Quoted 33 lines · Mika Westerberg
Hi Bjorn,
Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.Hello Mika,
Quoted 8 lines
Quoted 4 lines
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.
Quoted 8 lines
To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.
If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."
Quoted 8 lines
Fair enough.
I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to
/sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).
JFYI actually it is a mainline kernel parameter.
Reference the documentation here:
https://www.kernel.org/doc/html/v6.4/admin-guide/kernel-parameters.html?highlight=mem_sleep_default
Quoted 12 lines
So when forcing S3 this means the system suspend and resume is done through the BIOS who is supposed to enable wakes and program the devices accordingly during and after S3 before the OS is given back the control, it might very well be that it already did something for the L1 configuration here before Linux (with this patch) reconfigured them and this is causing the problem.
@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.
Quoted 43 lines · Mario Limonciello
Quoted 33 lines · Mika Westerberg
Hi Bjorn,
Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg
Commit
a7152be79b62("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.Hello Mika,
Quoted 8 lines
Quoted 4 lines
I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.
Quoted 8 lines
To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.
If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."
Quoted 8 lines
Fair enough.
I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to
/sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).
Quoted 4 lines
JFYI actually it is a mainline kernel parameter.
Reference the documentation here:
https://www.kernel.org/doc/html/v6.4/admin-guide/kernel-parameters.html?highlight=mem_sleep_default
Indeed it is. Thanks for correcting me. I grepped this from my source tree but somehow screwed it up and it did not show up anything. Now that I checked again it is there.