T20 Cpuidle Freeze
Hi Rafael, dear community
One of our customers reported seeing freezes when running the LTS Linux kernel 4.9.x on our Toradex Colibri T20 modules [1]. I was able to reproduce a complete SoC lock-up after a few minutes also running the latest 4.14-rc7 while LTS 4.4.x seemed to run stable.
Having attempted a multi-level bisection points towards the following first bad commit:
9c4b2867ed7c8c8784dd417ffd16e705e81eb145
cpuidle: menu: Fix menu_select() for CPUIDLE_DRIVER_STATE_START == 0
Unfortunately as drivers/cpuidle/governors/menu.c has gotten further edits since it seems not trivial to just revert it.
However I found out that it indeed has to do with CPU idle as when I did disable the CONFIG_CPU_IDLE Linux kernel configuration option also LTS 4.9.59 as well as latest 4.14-rc7 run now stable overnight.
Does anybody have any clue what exactly may be happening and/or why cpuidle may not run stable on T20? Or is everybody always just disabling cpuidle on T20 anyway?
Thanks!
[1] https://www.toradex.com/community/questions/16838/actual-lts-kernel
-49-on-colibri-t20.html
Best regards - Mit freundlichen Grüssen - Meilleures salutations
Marcel Ziswiler
Platform Manager Embedded Linux
Toradex AG
Altsagenstrasse 5 | 6048 Horw/Luzern | Switzerland | T: +41 41 500 48 00 (main line) | Direct: +41 41 500 48 10
Quoted 21 lines · Marcel Ziswiler
Hi Rafael, dear community
One of our customers reported seeing freezes when running the LTS Linux kernel 4.9.x on our Toradex Colibri T20 modules [1]. I was able to reproduce a complete SoC lock-up after a few minutes also running the latest 4.14-rc7 while LTS 4.4.x seemed to run stable.
Having attempted a multi-level bisection points towards the following first bad commit:
9c4b2867ed7c8c8784dd417ffd16e705e81eb145cpuidle: menu: Fix
menu_select()forCPUIDLE_DRIVER_STATE_START== 0Unfortunately as
drivers/cpuidle/governors/menu.chas gotten further edits since it seems not trivial to just revert it.However I found out that it indeed has to do with CPU idle as when I did disable the
CONFIG_CPU_IDLELinux kernel configuration option also LTS 4.9.59 as well as latest 4.14-rc7 run now stable overnight.
Quoted 8 lines
Does anybody have any clue what exactly may be happening and/or why cpuidle may not run stable on T20? Or is everybody always just disabling cpuidle on T20 anyway?
Thanks!
[1] https://www.toradex.com/community/questions/16838/actual-lts-kernel
-49-on-colibri-t20.html
I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage 162283 32905 254669 32905
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time 436981763 2110484666 458260707 2121781516
# uptime 18:50:24 up 44 min, 1 user, load average: 0.15, 0.08, 0.07
It could be that cpuidle unmasks some other issue on the Colibri.
Quoted 9 lines · Dmitry Osipenko
I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage 162283 32905 254669 32905
Quoted 8 lines
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time 436981763 2110484666 458260707 2121781516# uptime 18:50:24 up 44 min, 1 user, load average: 0.15, 0.08, 0.07
OK, thanks. Good to know.
Quoted 1 line
It could be that cpuidle unmasks some other issue on the Colibri.
Yes, that's also my thinking.
What hardware did you run this on?
Does your kernel configuration differ from the stock tegra_defconfig I used?
What exact device tree source are you using (or just stock)?
Maybe the compiler version you are using could also have some influence?
If you have any additional suggestions on what else could be relevant please let me know.
Quoted 19 lines · Marcel Ziswiler
Quoted 9 lines · Dmitry Osipenko
I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage 162283 32905 254669 32905Quoted 8 lines
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time 436981763 2110484666 458260707 2121781516# uptime 18:50:24 up 44 min, 1 user, load average: 0.15, 0.08, 0.07
Quoted 8 lines
OK, thanks. Good to know.
Quoted 1 line
It could be that cpuidle unmasks some other issue on the Colibri.
Yes, that's also my thinking.
What hardware did you run this on?
I ran it on Acer A500 tablet, which is tegra20.
Quoted 2 lines
Does your kernel configuration differ from the stock
tegra_defconfigI used?
Doesn't differ, used stock tegra_defconfig as well.
Quoted 2 lines
What exact device tree source are you using (or just stock)?
https://gist.github.com/digetx/2f624a0df4caff657ef28863b5354d5b
Quoted 2 lines
Maybe the compiler version you are using could also have some influence?
Compiler bugs aren't rareness, I'm using armv7a-hardfloat-linux-gnueabi-gcc (Gentoo 6.4.0 p1.0) 6.4.0
Quoted 2 lines
If you have any additional suggestions on what else could be relevant please let me know.
1) I suppose you could attach JTAG and see where hang happens.
2) Enable all kernel debug Kconfig options.
3) Disable all non-critical device drivers in Kconfig, so that you could boot. See if it makes difference.
4) Probably just adding some debug printk's would be good enough to localize the offending place in the code.