Skip to contentSkip to conversation

cpufreq/thermal regression in 6.10

DISCUSSION Johan Hovold 4 participants 9 messages Jun 9, 2024 – Jun 22, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

	cpu4/cpufreq/scaling_max_freq:940800
	cpu5/cpufreq/scaling_max_freq:940800
	cpu6/cpufreq/scaling_max_freq:940800
	cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

This obviously leads to a massive performance drop and could possibly also be related to reports like this one:

https://lore.kernel.org/all/CAHk-=wjwFGQZcDinK=BkEaA8FSyVg5NaUe0BobxowxeZ5PvetA@mail.gmail.com/

I assume the regression may have been introduced by all the thermal work that went into 6.10-rc1, but I don't have time to try to track this down myself right now (and will be away from keyboard most of next week).

I've confirmed that 6.9 works as expected.

Johan

#regzbot introduced: v6.9..v6.10-rc2

#1 lore raw

Hi,

Thanks for the report.

Quoted 11 lines · Johan Hovold

Hi,

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

Quoted 6 lines
        cpu4/cpufreq/scaling_max_freq:940800
        cpu5/cpufreq/scaling_max_freq:940800
        cpu6/cpufreq/scaling_max_freq:940800
        cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

So apparently something fails to update its frequency QoS request.

Would it be possible to provoke this with thermal debug enabled (CONFIG_THERMAL_DEBUGFS set) and see what's there in /sys/kernel/debug/thermal/?

Quoted 10 lines

This obviously leads to a massive performance drop and could possibly also be related to reports like this one:

https://lore.kernel.org/all/CAHk-=wjwFGQZcDinK=BkEaA8FSyVg5NaUe0BobxowxeZ5PvetA@mail.gmail.com/

I assume the regression may have been introduced by all the thermal work that went into 6.10-rc1, but I don't have time to try to track this down myself right now (and will be away from keyboard most of next week).

I've confirmed that 6.9 works as expected.

Well, I'd need to ask someone else affected by this, then.

#2 lore raw
Quoted 35 lines · Rafael J. Wysocki

Hi,

Thanks for the report.

Quoted 11 lines · Johan Hovold

Hi,

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

Quoted 6 lines
        cpu4/cpufreq/scaling_max_freq:940800
        cpu5/cpufreq/scaling_max_freq:940800
        cpu6/cpufreq/scaling_max_freq:940800
        cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

So apparently something fails to update its frequency QoS request.

Would it be possible to provoke this with thermal debug enabled (CONFIG_THERMAL_DEBUGFS set) and see what's there in /sys/kernel/debug/thermal/?

Quoted 5 lines

This obviously leads to a massive performance drop and could possibly also be related to reports like this one:

https://lore.kernel.org/all/CAHk-=wjwFGQZcDinK=BkEaA8FSyVg5NaUe0BobxowxeZ5PvetA@mail.gmail.com/

Quoted 7 lines
Quoted 5 lines

I assume the regression may have been introduced by all the thermal work that went into 6.10-rc1, but I don't have time to try to track this down myself right now (and will be away from keyboard most of next week).

I've confirmed that 6.9 works as expected.

Well, I'd need to ask someone else affected by this, then.

If this is the step-wise governor, the problem might have been introduced by commit

042a3d80f118 thermal: core: Move passive polling management to the core

which removed passive polling count updates from that governor, so if the thermal zone in question has passive polling only and no regular polling, temperature updates may stop coming before the governor drops the cooling device states to the "no target" level.

So please test the attached partial revert of the above commit when you can.

---
 drivers/thermal/gov_step_wise.c |   10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/thermal/gov_step_wise.c +10 -0
Index: linux-pm/drivers/thermal/gov_step_wise.c
===================================================================
@@ -93,6 +93,16 @@ static void thermal_zone_trip_update(str
 		if (instance->initialized && old_target == instance->target)
 			continue;
 
+		if (trip->type == THERMAL_TRIP_PASSIVE) {
+			/* If needed, update the status of passive polling. */
+			if (old_target == THERMAL_NO_TARGET &&
+			    instance->target != THERMAL_NO_TARGET)
+				tz->passive++;
+			else if (old_target != THERMAL_NO_TARGET &&
+				 instance->target == THERMAL_NO_TARGET)
+				tz->passive--;
+		}
+
 		instance->initialized = true;
 
 		mutex_lock(&instance->cdev->lock);
author Tested-by
#3 lore raw
Quoted 21 lines · Rafael J. Wysocki
Quoted 10 lines · Rafael J. Wysocki
Quoted 8 lines · Johan Hovold

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

Quoted 6 lines
Quoted 6 lines
        cpu4/cpufreq/scaling_max_freq:940800
        cpu5/cpufreq/scaling_max_freq:940800
        cpu6/cpufreq/scaling_max_freq:940800
        cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

If this is the step-wise governor, the problem might have been introduced by commit

Quoted 8 lines

042a3d80f118 thermal: core: Move passive polling management to the core

which removed passive polling count updates from that governor, so if the thermal zone in question has passive polling only and no regular polling, temperature updates may stop coming before the governor drops the cooling device states to the "no target" level.

So please test the attached partial revert of the above commit when you can.

Thanks for the quick fix. The partial revert seems to do the trick:

Johan

#4 lore raw
Quoted 27 lines · Johan Hovold
Quoted 26 lines · Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Johan Hovold

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

Quoted 6 lines
Quoted 6 lines
         cpu4/cpufreq/scaling_max_freq:940800
         cpu5/cpufreq/scaling_max_freq:940800
         cpu6/cpufreq/scaling_max_freq:940800
         cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

If this is the step-wise governor, the problem might have been introduced by commit

042a3d80f118 thermal: core: Move passive polling management to the core

which removed passive polling count updates from that governor, so if the thermal zone in question has passive polling only and no regular polling, temperature updates may stop coming before the governor drops the cooling device states to the "no target" level.

Quoted 6 lines
Quoted 1 line

So please test the attached partial revert of the above commit when you can.

Thanks for the quick fix. The partial revert seems to do the trick:

Johan

I can also confirm that it's working here!

#5 lore raw
In reply to Rafael J. Wysocki

Hi there,

unfortunately I experienced the issue with the fix applied. I had to revert this and  the original commit to get back to normal behaviour. My system (also Lenovo Thinkpad X13s) uses the schedutil governor, the behaviour is as described from Steev and Johan. The full throttling happened during a package build and left the performance cores at 940800.

Cheers

Jens

Quoted 50 lines · Rafael J. Wysocki
Quoted 30 lines · Rafael J. Wysocki

Hi,

Thanks for the report.

Quoted 10 lines · Johan Hovold

Hi,

Steev reported to me off-list that the CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2.

I just confirmed that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP:

Quoted 6 lines
         cpu4/cpufreq/scaling_max_freq:940800
         cpu5/cpufreq/scaling_max_freq:940800
         cpu6/cpufreq/scaling_max_freq:940800
         cpu7/cpufreq/scaling_max_freq:940800

when the temperature drops again.

So apparently something fails to update its frequency QoS request.

Would it be possible to provoke this with thermal debug enabled (CONFIG_THERMAL_DEBUGFS set) and see what's there in /sys/kernel/debug/thermal/?

Quoted 3 lines

This obviously leads to a massive performance drop and could possibly also be related to reports like this one:

Quoted 8 lines
Quoted 7 lines

https://lore.kernel.org/all/CAHk-=wjwFGQZcDinK=BkEaA8FSyVg5NaUe0BobxowxeZ5PvetA@mail.gmail.com/

I assume the regression may have been introduced by all the thermal work that went into 6.10-rc1, but I don't have time to try to track this down myself right now (and will be away from keyboard most of next week).

I've confirmed that 6.9 works as expected.

Well, I'd need to ask someone else affected by this, then.

If this is the step-wise governor, the problem might have been introduced by commit

042a3d80f118 thermal: core: Move passive polling management to the core

which removed passive polling count updates from that governor, so if the thermal zone in question has passive polling only and no regular polling, temperature updates may stop coming before the governor drops the cooling device states to the "no target" level.

So please test the attached partial revert of the above commit when you can.

#6 lore raw

Hi,

Quoted 8 lines · Jens Glathe

Hi there,

unfortunately I experienced the issue with the fix applied. I had to revert this and the original commit to get back to normal behaviour. My system (also Lenovo Thinkpad X13s) uses the schedutil governor, the behaviour is as described from Steev and Johan. The full throttling happened during a package build and left the performance cores at 940800.

So can you please test the attached patch, on top of the fix?

---
 drivers/thermal/gov_step_wise.c |   19 +------------------
 1 file changed, 1 insertion(+), 18 deletions(-)
drivers/thermal/gov_step_wise.c +1 -18
Index: linux-pm/drivers/thermal/gov_step_wise.c
===================================================================
@@ -55,7 +55,7 @@ static unsigned long get_target_state(st
 		if (cur_state <= instance->lower)
 			return THERMAL_NO_TARGET;
 
-		return clamp(cur_state - 1, instance->lower, instance->upper);
+		return instance->lower;
 	}
 
 	return instance->target;
@@ -93,23 +93,6 @@ static void thermal_zone_trip_update(str
 		if (instance->initialized && old_target == instance->target)
 			continue;
 
-		if (trip->type == THERMAL_TRIP_PASSIVE) {
-			/*
-			 * If the target state for this thermal instance
-			 * changes from THERMAL_NO_TARGET to something else,
-			 * ensure that the zone temperature will be updated
-			 * (assuming enabled passive cooling) until it becomes
-			 * THERMAL_NO_TARGET again, or the cooling device may
-			 * not be reset to its initial state.
-			 */
-			if (old_target == THERMAL_NO_TARGET &&
-			    instance->target != THERMAL_NO_TARGET)
-				tz->passive++;
-			else if (old_target != THERMAL_NO_TARGET &&
-				 instance->target == THERMAL_NO_TARGET)
-				tz->passive--;
-		}
-
 		instance->initialized = true;
 
 		mutex_lock(&instance->cdev->lock);
Tested-by
#7 lore raw

Hi there,

thank you for the fast fix. Applied, built, installed. Test is successful, performance core scaling up to 2995200 comes back when skin temp drops below 55°C.

Cheers

Jens

Quoted 12 lines · Rafael J. Wysocki

Hi,

Quoted 7 lines · Jens Glathe

Hi there,

unfortunately I experienced the issue with the fix applied. I had to revert this and the original commit to get back to normal behaviour. My system (also Lenovo Thinkpad X13s) uses the schedutil governor, the behaviour is as described from Steev and Johan. The full throttling happened during a package build and left the performance cores at 940800.

So can you please test the attached patch, on top of the fix?

#8 lore raw
Quoted 8 lines · Jens Glathe

Hi there,

thank you for the fast fix. Applied, built, installed. Test is successful, performance core scaling up to 2995200 comes back when skin temp drops below 55°C.

Great, thanks for testing!

I'll submit the patch shortly and it would be good if the others affected by this issue could try it to confirm that it doesn't regress anything.

Quoted 14 lines · Rafael J. Wysocki
Quoted 12 lines

Hi,

Quoted 7 lines · Jens Glathe

Hi there,

unfortunately I experienced the issue with the fix applied. I had to revert this and the original commit to get back to normal behaviour. My system (also Lenovo Thinkpad X13s) uses the schedutil governor, the behaviour is as described from Steev and Johan. The full throttling happened during a package build and left the performance cores at 940800.

So can you please test the attached patch, on top of the fix?