Skip to contentSkip to conversation

PCI/ASPM: Add back L1 PM Substate save and restore

PATCH Superseded Mika Westerberg 6 participants 21 messages Jun 27, 2023 – Aug 10, 2023 PCI

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/ASPM: Add back L1 PM Substate save and restore

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

The original revert mentioned that we restore L1 PM substate configuration even though ASPM L1 may already be enabled. This is due the fact that the pci_restore_aspm_l1ss_state() was called before pci_restore_pcie_state().

Try to enable this functionality again by:

1) Moving the restore happen after PCIe capability is restored

    following PCIe r6.0, sec 5.5.4.

2) Following the PCIe spec more closely to restore L1 PM substate configuration (program enable bits last).
3) Adding denylist that skips the restoring on the ASUS system where the original regression happened, just in case.

---
Hi,

There are several systems out there that are not entering lower package C-states after the first suspend/resume cycle because we do not restore L1SS configuration as can be seen in the linked bugzilla entry. Although they are functioning fine they still empty the battery faster than the user may expect.

Apologies sending this out during merge window but I will be starting my vacation next week so wanted to get this out before, and possibly incorporate any changes during the week still.

 drivers/pci/pci.c       |  7 ++++
 drivers/pci/pci.h       |  4 +++
 drivers/pci/pcie/aspm.c | 76 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 87 insertions(+)
drivers/pci/pci.c +7 -0
index 578bf0d3ec3c..bf30561aa32c 100644
@@ -1679,6 +1679,7 @@ int pci_save_state(struct pci_dev *dev)
 	pci_save_dpc_state(dev);
 	pci_save_aer_state(dev);
 	pci_save_ptm_state(dev);
+	pci_save_aspm_l1ss_state(dev);
 	return pci_save_vc_state(dev);
 }
 EXPORT_SYMBOL(pci_save_state);
@@ -1790,6 +1791,7 @@ void pci_restore_state(struct pci_dev *dev)
 	pci_restore_vc_state(dev);
 	pci_restore_rebar_state(dev);
 	pci_restore_dpc_state(dev);
+	pci_restore_aspm_l1ss_state(dev);
 	pci_restore_ptm_state(dev);
 
 	pci_aer_clear_status(dev);
@@ -3474,6 +3476,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)
 	if (error)
 		pci_err(dev, "unable to allocate suspend buffer for LTR\n");
 
+	error = pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_L1SS,
+					    2 * sizeof(u32));
+	if (error)
+		pci_err(dev, "unable to allocate suspend buffer for ASPM-L1SS\n");
+
 	pci_allocate_vc_save_buffers(dev);
 }
 
drivers/pci/pci.h +4 -0
index d09e8f39e429..f34b55701d63 100644
@@ -560,10 +560,14 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active);
 void pcie_aspm_init_link_state(struct pci_dev *pdev);
 void pcie_aspm_exit_link_state(struct pci_dev *pdev);
 void pcie_aspm_powersave_config_link(struct pci_dev *pdev);
+void pci_save_aspm_l1ss_state(struct pci_dev *pdev);
+void pci_restore_aspm_l1ss_state(struct pci_dev *pdev);
 #else
 static inline void pcie_aspm_init_link_state(struct pci_dev *pdev) { }
 static inline void pcie_aspm_exit_link_state(struct pci_dev *pdev) { }
 static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { }
+static inline void pci_save_aspm_l1ss_state(struct pci_dev *pdev) { }
+static inline void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) { }
 #endif
 
 #ifdef CONFIG_PCIE_ECRC
drivers/pci/pcie/aspm.c +76 -0

88 lines not on this page · raw message

Signature · 1 line
2.40.1
#1 lore raw
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Hello Mika,

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in <https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Best Regards,
Thomas

#2 lore raw

Hi Thomas,

Quoted 8 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Quoted 6 lines

Hello Mika,

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.

#3 lore raw
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Hello Mika,

Quoted 8 lines
Quoted 4 lines

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.

To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.

If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."

Bjorn

#4 lore raw

Hi Bjorn,

Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Hello Mika,

Quoted 8 lines
Quoted 4 lines

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.

Quoted 8 lines

To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.

If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."

Fair enough.

I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to /sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).

So when forcing S3 this means the system suspend and resume is done through the BIOS who is supposed to enable wakes and program the devices accordingly during and after S3 before the OS is given back the control, it might very well be that it already did something for the L1 configuration here before Linux (with this patch) reconfigured them and this is causing the problem.

@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.

#5 lore raw
Quoted 3 lines · Mika Westerberg

@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.

In fact, there is: Entering suspend-to-ram without setting /sys/power/mem_sleep to "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:

<https://www.tuxedocomputers.com/en/Infos/Help-Support/Instructions/Fine-tuning-of-power-management-with-suspend-standby.tuxedo#>

I just retested your patch with setting mem_sleep to "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.

I attached the output of dmidecode to the bugzilla entry mentioned above: <https://bugzilla.kernel.org/attachment.cgi?id=304494>

Best regards,
Thomas

#6 lore raw

Hi Thomas,

Quoted 11 lines · Thomas Witt
Quoted 3 lines · Mika Westerberg

@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.

In fact, there is: Entering suspend-to-ram without setting /sys/power/mem_sleep to "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:

<https://www.tuxedocomputers.com/en/Infos/Help-Support/Instructions/Fine-tuning-of-power-management-with-suspend-standby.tuxedo#>

Thanks for the clarification.

Quoted 3 lines

I just retested your patch with setting mem_sleep to "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

Quoted 2 lines

I attached the output of dmidecode to the bugzilla entry mentioned above:
<https://bugzilla.kernel.org/attachment.cgi?id=304494>

Thanks!

#7 lore raw
Quoted 17 lines · Mika Westerberg

Hi Thomas,

Quoted 11 lines · Thomas Witt
Quoted 3 lines · Mika Westerberg

@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.

In fact, there is: Entering suspend-to-ram without setting /sys/power/mem_sleep to "deep", my laptop consumes about the same power as it would idling online. The manufacturer suggests setting that commandline parameter:

<https://www.tuxedocomputers.com/en/Infos/Help-Support/Instructions/Fine-tuning-of-power-management-with-suspend-standby.tuxedo#>

Thanks for the clarification.

Quoted 8 lines
Quoted 3 lines

I just retested your patch with setting mem_sleep to "s2idle", and it no longer triggers the loss of PCI devices. I guess that could be the indicator that Björn asked for.

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

One possibility is to check the package C-state residency like:

  # cat /sys/kernel/debug/pmc_core/package_cstate_show

after system sleep and see if there is a change with the patch applied, as done here:

https://bugzilla.kernel.org/show_bug.cgi?id=217321

#8 lore raw
In reply to Mika Westerberg
Quoted 4 lines · Mika Westerberg

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With mem_sleep set to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.

So I guess I want to keep the "deep" setting in my cmdline.

BR
Thomas

#9 lore raw
Quoted 6 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

Quoted 6 lines

No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With mem_sleep set to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.

So I guess I want to keep the "deep" setting in my cmdline.

Okay thanks a lot for checking! Back to drawing board then...

#10 lore raw
In reply to Thomas Witt
Quoted 6 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

Quoted 6 lines

No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With mem_sleep set to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.

So I guess I want to keep the "deep" setting in my cmdline.

It's likely something is not entering the right state. You can try the following script to check what the cause may be.

https://github.com/intel/S0ixSelftestTool

David

Quoted 4 lines

BR
Thomas

#11 lore raw
Quoted 12 lines · David E. Box
Quoted 11 lines · Thomas Witt
Quoted 4 lines · Mika Westerberg

I wonder if the patch actually helps here now because the reason we want to add it back is that it allows the CPU to enter lower power states and thus reducing the power consumption in S2idle too. Do you observe that when you have the patch applied?

No joy. I did not check what its actually doing, but the computer takes about 150mA over the charger at idle with screen off and 140mA in suspend. With mem_sleep set to "deep", it takes about 20mA in suspend. All with the battery at 100%, at 19.7V.

Quoted 6 lines
Quoted 1 line

So I guess I want to keep the "deep" setting in my cmdline.

It's likely something is not entering the right state. You can try the following script to check what the cause may be.

https://github.com/intel/S0ixSelftestTool

That would be useful insights, indeed.

@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):

- Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
- Program L1 SS before L1 enables
- Program L1 SS enables after rest of the fields in the capability

Applies on top of v6.4.

------8<-----8<-----8<-----8<-----

drivers/pci/pci.c +16 -2
index 5ede93222bc1..2e947fea5afc 100644
@@ -1545,7 +1545,7 @@ static void pci_restore_pcie_state(struct pci_dev *dev)
 {
 	int i = 0;
 	struct pci_cap_saved_state *save_state;
-	u16 *cap;
+	u16 *cap, val;
 
 	save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP);
 	if (!save_state)
@@ -1560,7 +1560,14 @@ static void pci_restore_pcie_state(struct pci_dev *dev)
 
 	cap = (u16 *)&save_state->cap.data[0];
 	pcie_capability_write_word(dev, PCI_EXP_DEVCTL, cap[i++]);
-	pcie_capability_write_word(dev, PCI_EXP_LNKCTL, cap[i++]);
+	/*
+	 * Restoring ASPM L1 substates has special requirements
+	 * according to the PCIe spec 6.0. So we restore here only the
+	 * LNKCTL register with the ASPM control field clear. ASPM will
+	 * be restored in pci_restore_aspm_state().
+	 */
+	val = cap[i++] & ~PCI_EXP_LNKCTL_ASPMC;
+	pcie_capability_write_word(dev, PCI_EXP_LNKCTL, val);
 	pcie_capability_write_word(dev, PCI_EXP_SLTCTL, cap[i++]);
 	pcie_capability_write_word(dev, PCI_EXP_RTCTL, cap[i++]);
 	pcie_capability_write_word(dev, PCI_EXP_DEVCTL2, cap[i++]);
@@ -1671,6 +1678,7 @@ int pci_save_state(struct pci_dev *dev)
 	pci_save_ltr_state(dev);
 	pci_save_dpc_state(dev);
 	pci_save_aer_state(dev);
+	pci_save_aspm_state(dev);
 	pci_save_ptm_state(dev);
 	return pci_save_vc_state(dev);
 }
@@ -1784,6 +1792,7 @@ void pci_restore_state(struct pci_dev *dev)
 	pci_restore_rebar_state(dev);
 	pci_restore_dpc_state(dev);
 	pci_restore_ptm_state(dev);
+	pci_restore_aspm_state(dev);
 
 	pci_aer_clear_status(dev);
 	pci_restore_aer_state(dev);
@@ -3467,6 +3476,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)
 	if (error)
 		pci_err(dev, "unable to allocate suspend buffer for LTR\n");
 
+	error = pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_L1SS,
+					    2 * sizeof(u32));
+	if (error)
+		pci_err(dev, "unable to allocate suspend buffer for ASPM-L1SS\n");
+
 	pci_allocate_vc_save_buffers(dev);
 }
 
drivers/pci/pci.h +4 -0
index 2475098f6518..47dbbc006884 100644
@@ -567,10 +567,14 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active);
 void pcie_aspm_init_link_state(struct pci_dev *pdev);
 void pcie_aspm_exit_link_state(struct pci_dev *pdev);
 void pcie_aspm_powersave_config_link(struct pci_dev *pdev);
+void pci_save_aspm_state(struct pci_dev *pdev);
+void pci_restore_aspm_state(struct pci_dev *pdev);
 #else
 static inline void pcie_aspm_init_link_state(struct pci_dev *pdev) { }
 static inline void pcie_aspm_exit_link_state(struct pci_dev *pdev) { }
 static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { }
+static inline void pci_save_aspm_state(struct pci_dev *pdev) { }
+static inline void pci_restore_aspm_state(struct pci_dev *pdev) { }
 #endif
 
 #ifdef CONFIG_PCIE_ECRC
drivers/pci/pcie/aspm.c +109 -0

121 lines not on this page · raw message

Signature · 1 line
2.39.2
#12 lore raw
Quoted 10 lines · Mika Westerberg

@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):

- Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
- Program L1 SS before L1 enables
- Program L1 SS enables after rest of the fields in the capability

Sadly, same as before. With s2idle, power consumption stays up, but suspend/resume works, with deep it does not correctly suspend the PCI devices.

#13 lore raw

Hi Thomas,

Quoted 7 lines · Thomas Witt
Quoted 7 lines · Mika Westerberg
Quoted 6 lines
Quoted 6 lines

@Thomas, below is an updated patch. I wonder if you could try it out? This one restores L1 substates first and then L0s/L1 as the spec suggests. If this does not work, them I'm not sure what to do because now we should be doing exactly what the spec is saying (unless I misinterpret something):

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

   - Write L1 enables on the upstream component first then downstream (this is taken care by the parent child order of the Linux PM).
   - Program L1 SS before L1 enables
   - Program L1 SS enables after rest of the fields in the capability

Sadly, same as before. With s2idle, power consumption stays up, but suspend/resume works, with deep it does not correctly suspend the PCI devices.

Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.

The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.

https://bugzilla.kernel.org/attachment.cgi?id=304553

Please let me know if it works. Thanks.

David

#14 lore raw
Quoted 6 lines · David E. Box

Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.

Quoted 8 lines

The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.

https://bugzilla.kernel.org/attachment.cgi?id=304553

Please let me know if it works. Thanks.

David

Hi David,

I tried your patch, and I see no difference from Mika's. Still not coming back from suspend.

Thomas

#15 lore raw

Hi Thomas,

Quoted 12 lines · Thomas Witt
Quoted 11 lines · David E. Box

Mika is now out on extended vacation. We still need a solution that will enable the L1 substate save/restore without breaking your system. I'd like to try to get the power consumption lowered on your system while suspended with s2idle. The s0ix self test script will really help to tell us where to start. You can provide the results in the bugzilla.

The other thing we can do is find out why it only breaks under S3. It could be timing related, so I've attached another patch to the bugzilla to test this.

https://bugzilla.kernel.org/attachment.cgi?id=304553

Quoted 8 lines
Quoted 3 lines

Please let me know if it works. Thanks.

David

Hi David,

I tried your patch, and I see no difference from Mika's. Still not coming back from suspend.

Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.

#16 lore raw
Quoted 5 lines · Mika Westerberg

Hi Thomas,

Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.

Hi Mika,

I did, with -s it gives

Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.

with -r on

Your system did not achieve the runtime PC10 state during screen ON

additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")

with -r off, it tries xset which fails due to a lack of xserver.

Thomas

#17 lore raw

Hi Thomas,

Quoted 11 lines · Thomas Witt
Quoted 5 lines · Mika Westerberg

Hi Thomas,

Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.

Hi Mika,

I did, with -s it gives

Quoted 7 lines

Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.

with -r on

Your system did not achieve the runtime PC10 state during screen ON

Thanks for trying. Did you change the "mem_sleep" back to "s2idle" before you run the script?

Quoted 3 lines

additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")

@David, do you know what might be the issue?

Quoted 1 line

with -r off, it tries xset which fails due to a lack of xserver.

You do have graphics running right? I mean i915 driver is enabled and all the firmwares are in place (should come with the distro). I'm asking because s2idle typically requires that graphics and pretty much all the devices on the SoC have a driver and the accompanying firmwares, and that they enter D3 properly.

#18 lore raw
Quoted 18 lines · Mika Westerberg

Hi Thomas,

Quoted 15 lines · Thomas Witt
Quoted 5 lines · Mika Westerberg

Hi Thomas,

Thanks for trying that. Did you manage to try out the S0ix script David suggested? That should show us hopefully what is draining the battery in s2idle.

Hi Mika,

I did, with -s it gives

Your system does not support low power S0 idle capability. Isolation suggestion:
Please check BIOS low power S0 idle capability setting.

Quoted 6 lines
Quoted 3 lines

with -r on

Your system did not achieve the runtime PC10 state during screen ON

Thanks for trying. Did you change the "mem_sleep" back to "s2idle" before you run the script?

The script checks the FADT to determine if the system supports S0ix and it found that it didn't which is weird since Thomas is setting "mem_sleep" to "deep" from the default "s2idle" which is based on this bit.

Here are the commands to check it.

sudo acpidump -n FADT -b
iasl -d facp.dat
grep "Low Power S0 Idle" facp.dsl

Thomas, can you confirm what the value of mem_sleep is when you boot and run the above to confirm what your hardware supports?

Quoted 6 lines
Quoted 4 lines

additionally, it encounters a syntax error:
./s0ix-selftest-tool.sh: line 1182: wc:: syntax error in expression (error token is ":")

@David, do you know what might be the issue?

Yes. The latest kernel changes the output of the ltr_show command by adding a PMC number prefix (since Meteor Lake has more than one PMC now). The script is erring on the unexpected colon. We'll get this fixed.

David

Quoted 8 lines
Quoted 2 lines

with -r off, it tries xset which fails due to a lack of xserver.

You do have graphics running right? I mean i915 driver is enabled and all the firmwares are in place (should come with the distro). I'm asking because s2idle typically requires that graphics and pretty much all the devices on the SoC have a driver and the accompanying firmwares, and that they enter D3 properly.

#19 lore raw
In reply to Mika Westerberg
Quoted 33 lines · Mika Westerberg

Hi Bjorn,

Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Hello Mika,

Quoted 8 lines
Quoted 4 lines

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.

Quoted 8 lines

To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.

If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."

Quoted 8 lines

Fair enough.

I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to /sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).

JFYI actually it is a mainline kernel parameter.

Reference the documentation here:
https://www.kernel.org/doc/html/v6.4/admin-guide/kernel-parameters.html?highlight=mem_sleep_default

Quoted 12 lines

So when forcing S3 this means the system suspend and resume is done through the BIOS who is supposed to enable wakes and program the devices accordingly during and after S3 before the OS is given back the control, it might very well be that it already did something for the L1 configuration here before Linux (with this patch) reconfigured them and this is causing the problem.

@Thomas, is there any particular reason you have this option in the command line? There is possibility that S3 is not even fully validated if the system advertises S0 low power sleep instead.

#20 lore raw
Quoted 43 lines · Mario Limonciello
Quoted 33 lines · Mika Westerberg

Hi Bjorn,

Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Mika Westerberg
Quoted 10 lines · Thomas Witt
Quoted 6 lines · Mika Westerberg

Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted saving and restoring of ASPM L1 Substates due to a regression that caused resume from suspend to fail on certain systems. However, we never added this capability back and this is now causing systems fail to enter low power CPU states, drawing more power from the battery.

Hello Mika,

Quoted 8 lines
Quoted 4 lines

I am sorry, but your patch (applied on top of master) triggers the exact same behaviour I described in
<https://bugzilla.kernel.org/show_bug.cgi?id=216877> (nvme and wifi become unavailable during suspend/resume)

Thanks for testing! Can you provide the output of dmidecode from that system? We can add it to the denylist as well to avoid the issue on your system.

Quoted 8 lines

To me this says we don't completely understand the mechanism of the failure. If BIOS made L1SS work initially, Linux should be able to make it work again after suspend/resume.

If we can identify an actual hardware or firmware defect, it makes good sense to add a quirk or denylist. But I'll push back a little if it's just "there's some problem we don't understand on this system, so avoid it."

Quoted 8 lines

Fair enough.

I've looked at the Thomas' system dmesg that he attached to the bugzilla and he has mem_sleep_default=deep in the kernel command line. There is no such option in the mainline kernel but I suppose this makes systemd (or some initscript) to write "deep" to /sys/power/mem_sleep, thus forcing S3 instead of the default the ACPI tables suggest, which probably is S0ix (Intel low power state which does not involve BIOS).

Quoted 4 lines

JFYI actually it is a mainline kernel parameter.

Reference the documentation here:
https://www.kernel.org/doc/html/v6.4/admin-guide/kernel-parameters.html?highlight=mem_sleep_default

Indeed it is. Thanks for correcting me. I grepped this from my source tree but somehow screwed it up and it did not show up anything. Now that I checked again it is there.