: thermal and an scaling governor issues
Before anything, please CC: me, as I'm not subscribed to the list.
Right now I'm running 6.12.38+deb13-amd64 from Debian
testing/unstable, which are the same due to the freeze until next week :)
It all seems to have started at the beginning of the year. The
symptom was hiccups which could be seen as high load (f.i. 40 on an 8 core machine) with not much CPU usage. I tracked it down to prometheus-node-exporter stalling on disk accesses. Looking with strace I find there were multisecond reads on files related to sensonrs in general, which I still experience:
23:30:13.855780 (+ 0.004490) openat(AT_FDCWD, "/sys/class/hwmon/hwmon1/in0_input", O_RDONLY) = 3 <0.000152>
23:30:13.856286 (+ 0.000494) fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 <0.000052>
23:30:13.856574 (+ 0.000286) read(3, "12686\n", 4096) = 6 <3.196855>
23:30:17.053632 (+ 3.197072) close(3) = 0 <0.000061>23:33:39.689505 (+ 0.000167) openat(AT_FDCWD, "/sys/class/hwmon/hwmon5/fan1_input", O_RDONLY) = 3 <0.000081>
23:33:39.689708 (+ 0.000201) fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 <0.000047>
23:33:39.689870 (+ 0.000164) read(3, "3225\n", 4096) = 5 <0.018190>
23:33:39.708237 (+ 0.018376) close(3) = 0 <0.000060>23:33:39.710771 (+ 0.000151) openat(AT_FDCWD, "/sys/class/hwmon/hwmon5/fan2_input", O_RDONLY) = 3 <0.000073>
23:33:39.710947 (+ 0.000176) fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 <0.000042>
23:33:39.711089 (+ 0.000141) read(3, "3215\n", 4096) = 5 <0.128783>
23:33:39.840088 (+ 0.129024) close(3) = 0 <0.000067> Accumulating all those reads led to the node exporter to gradually
take more and more time to finish the scrape, going beyond the 15s between scrapes. I'm not sure, but I think at some point the node exporter just launches goroutines to do these sweeps and they just pile up one on top of the other. I had to disable the collectors.
That's problem #1.
Today I was (ab)using the 8 cores compiling stuff, while watching
videos, so a load of above 10. I'm using (KDE's?) upowerd, but I'm not sure this has any impact. KDE's power plasmoid shows it's using the p[erformance governor, but checking the files I get another story:
$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor powersave powersave powersave powersave powersave powersave powersave powersave
So I manually set them to performance:
$ echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor performance
$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor performance performance performance performance performance performance performance performance
But CPU frequency stays below the 1GHz for a 3.8GHz CPU and the
temps stay below 47C.
Now, I'm not sure if these things are connected, but I wager they
do. Even if they aren't, they are annoying independently. The problem
is: I have no idea how to debug this. I see no particular info in dmesg. I tried other Debian kernel versions all down to the latest 5.x I could get. I'm open to a slow tracking down of this thing because it has been working for some 6.5y before these behaviors. My other alternative is to buy new HW.
Cheers, -- Marcos.
Quoted 1 line · Marcos Dione
Before anything, please CC: me, as I'm not subscribed to the list.
This is no longer the case, I'm subscribed now.
Another data point: CPU freq goes back to normal, when my
compilation stops (last night before 2AM), but goes back down when I launch it again this morning (at around 10:30):
https://i.imgur.com/l0yZYCQ.png
Cheers, -- Marcos.
Quoted 6 lines · Marcos Dione
On Sun, Aug 03, 2025 at 10:55:45AM +0200, Marcos Dione wrote:
Another data point: CPU freq goes back to normal, when my
compilation stops (last night before 2AM), but goes back down when I launch it again this morning (at around 10:30):
Another one: I started compiling last night at ~21:00, but later at
~21:50, after a few back and forth between high and low frequencies, it dropped to low and staed there, even when the compilation lasted for another 4h.
https://i.imgur.com/qb41dZ0.png
Cheers, -- Marcos.
Quoted 1 line · Marcos Dione
I gave it a go, reading 3 pages, but got not much better (but
further). You can read it all here:
https://en.osm.town/@mdione/114964442166494174
In short, I only confirmed that I'm using intel_pstate, but that all
settings point to the performace setting, yet I sill get between 0.8 and 1.1MHz.
I'm going to reboot this machine now, and frankly I hope I don't hit
this bug again, but:
a) I'm sure I'm going to; and
b) The file reading times still are an issue.
Cheers, -- Marcos.
Quoted 11 lines · Marcos Dione
Quoted 1 line · Marcos Dione
Before anything, please CC: me, as I'm not subscribed to the list.
This is no longer the case, I'm subscribed now.
Another data point: CPU freq goes back to normal, when my
compilation stops (last night before 2AM), but goes back down when I launch it again this morning (at around 10:30):
IIUC, it is a laptop.
There can be a couple of things. The difficult part is the firmware can do actions under the hood, the userspace may change the governors depending on the temperature and the kernel can do something else.
I suggest to investigate first the temperature sensor for the skin (or case) and the battery. What are their temperature when you compile ?
Given the policy on laptops is to protect the user first if the skin/case temp is above 43°C, then an action is done and that can result on limiting the processors.
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog
Quoted 1 line · Daniel Lezcano
IIUC, it is a laptop.
yes.
Quoted 3 lines
There can be a couple of things. The difficult part is the firmware can do actions under the hood, the userspace may change the governors depending on the temperature and the kernel can do something else.
Do we get any events about this or is 100% transparent?
Quoted 2 lines
I suggest to investigate first the temperature sensor for the skin (or case) and the battery. What are their temperature when you compile ?
As I mention, I had to disable monitoring, but I could check
manually. All I can say is that that was particularly not hot (much less heatwave-y) week, so it _should_ have been cooler than usual (for this period). Also, a reboot fixed it, so it was not that.
Cheers, -- Marcos.
Quoted 10 lines · Marcos Dione
Quoted 1 line · Daniel Lezcano
IIUC, it is a laptop.
yes.
Quoted 3 lines
There can be a couple of things. The difficult part is the firmware can do actions under the hood, the userspace may change the governors depending on the temperature and the kernel can do something else.
Do we get any events about this or is 100% transparent?
It depends on the firmware and the hardware. Well all the thermal stack is arch specific so it is hard to give a simple answer. But if the system is configured with thermal zones and trip points, then it should be possible to monitor the temperature events when the temperature thresholds are crossed.
Quoted 7 lines
Quoted 2 lines
I suggest to investigate first the temperature sensor for the skin (or case) and the battery. What are their temperature when you compile ?
As I mention, I had to disable monitoring, but I could check
manually. All I can say is that that was particularly not hot (much less heatwave-y) week, so it _should_ have been cooler than usual (for this period). Also, a reboot fixed it, so it was not that.
If the governors are changed, that means the userspace does an action and it is usually the thermal daemon. And this one IIUC is supposed to monitor the skin temperature.
If it happens again, you should double check the skin temperature sensor (or case sensor). This one is supposed to be monitored by the thermal daemon and should stay below 43°C. The thermald can change the governor if it detects the skin temperature sensor is getting hot.
One common scenario is fast charging while computing. And with the years, the thermal compound can lose its thermal conduction properties leading to the heat not being evacuated efficiently.
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog