Skip to contentSkip to conversation

T20 Cpuidle Freeze

DISCUSSION Marcel Ziswiler 2 participants 4 messages Nov 3, 2017 – Nov 5, 2017 Tegra architecture support

lore
Share & tools
lore
#0 lore raw

Hi Rafael, dear community

One of our customers reported seeing freezes when running the LTS Linux kernel 4.9.x on our Toradex Colibri T20 modules [1]. I was able to reproduce a complete SoC lock-up after a few minutes also running the latest 4.14-rc7 while LTS 4.4.x seemed to run stable.

Having attempted a multi-level bisection points towards the following first bad commit:

9c4b2867ed7c8c8784dd417ffd16e705e81eb145

cpuidle: menu: Fix menu_select() for CPUIDLE_DRIVER_STATE_START == 0

Unfortunately as drivers/cpuidle/governors/menu.c has gotten further edits since it seems not trivial to just revert it.

However I found out that it indeed has to do with CPU idle as when I did disable the CONFIG_CPU_IDLE Linux kernel configuration option also LTS 4.9.59 as well as latest 4.14-rc7 run now stable overnight.

Does anybody have any clue what exactly may be happening and/or why cpuidle may not run stable on T20? Or is everybody always just disabling cpuidle on T20 anyway?

Thanks!

[1] https://www.toradex.com/community/questions/16838/actual-lts-kernel
-49-on-colibri-t20.html

Best regards - Mit freundlichen Grüssen - Meilleures salutations

Marcel Ziswiler
Platform Manager Embedded Linux

Toradex AG
Altsagenstrasse 5 | 6048 Horw/Luzern | Switzerland | T: +41 41 500 48 00 (main line) | Direct: +41 41 500 48 10

#1 lore raw
Quoted 21 lines · Marcel Ziswiler

Hi Rafael, dear community

One of our customers reported seeing freezes when running the LTS Linux kernel 4.9.x on our Toradex Colibri T20 modules [1]. I was able to reproduce a complete SoC lock-up after a few minutes also running the latest 4.14-rc7 while LTS 4.4.x seemed to run stable.

Having attempted a multi-level bisection points towards the following first bad commit:

9c4b2867ed7c8c8784dd417ffd16e705e81eb145

cpuidle: menu: Fix menu_select() for CPUIDLE_DRIVER_STATE_START == 0

Unfortunately as drivers/cpuidle/governors/menu.c has gotten further edits since it seems not trivial to just revert it.

However I found out that it indeed has to do with CPU idle as when I did disable the CONFIG_CPU_IDLE Linux kernel configuration option also LTS 4.9.59 as well as latest 4.14-rc7 run now stable overnight.

Quoted 8 lines

Does anybody have any clue what exactly may be happening and/or why cpuidle may not run stable on T20? Or is everybody always just disabling cpuidle on T20 anyway?

Thanks!

[1] https://www.toradex.com/community/questions/16838/actual-lts-kernel
-49-on-colibri-t20.html

I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.

# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage
162283
32905
254669
32905
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time
436981763
2110484666
458260707
2121781516
# uptime
 18:50:24 up 44 min,  1 user,  load average: 0.15, 0.08, 0.07

It could be that cpuidle unmasks some other issue on the Colibri.

#2 lore raw
Quoted 9 lines · Dmitry Osipenko

I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.

# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage
162283
32905
254669
32905
Quoted 8 lines
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time
436981763
2110484666
458260707
2121781516
# uptime
 18:50:24 up 44 min,  1 user,  load average: 0.15, 0.08, 0.07

OK, thanks. Good to know.

Quoted 1 line

It could be that cpuidle unmasks some other issue on the Colibri.

Yes, that's also my thinking.

What hardware did you run this on?

Does your kernel configuration differ from the stock tegra_defconfig I used?

What exact device tree source are you using (or just stock)?

Maybe the compiler version you are using could also have some influence?

If you have any additional suggestions on what else could be relevant please let me know.

#3 lore raw
Quoted 19 lines · Marcel Ziswiler
Quoted 9 lines · Dmitry Osipenko

I haven't seen any problems with the cpuidle on next and 4.14-rc7 works fine.

# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/usage
162283
32905
254669
32905
Quoted 8 lines
# cat /sys/devices/system/cpu/cpu[0-1]/cpuidle/state[0-1]/time
436981763
2110484666
458260707
2121781516
# uptime
 18:50:24 up 44 min,  1 user,  load average: 0.15, 0.08, 0.07
Quoted 8 lines

OK, thanks. Good to know.

Quoted 1 line

It could be that cpuidle unmasks some other issue on the Colibri.

Yes, that's also my thinking.

What hardware did you run this on?

I ran it on Acer A500 tablet, which is tegra20.

Quoted 2 lines

Does your kernel configuration differ from the stock tegra_defconfig I used?

Doesn't differ, used stock tegra_defconfig as well.

Quoted 2 lines

What exact device tree source are you using (or just stock)?

https://gist.github.com/digetx/2f624a0df4caff657ef28863b5354d5b

Quoted 2 lines

Maybe the compiler version you are using could also have some influence?

Compiler bugs aren't rareness, I'm using armv7a-hardfloat-linux-gnueabi-gcc (Gentoo 6.4.0 p1.0) 6.4.0

Quoted 2 lines

If you have any additional suggestions on what else could be relevant please let me know.

1) I suppose you could attach JTAG and see where hang happens.

2) Enable all kernel debug Kconfig options.

3) Disable all non-critical device drivers in Kconfig, so that you could boot. See if it makes difference.

4) Probably just adding some debug printk's would be good enough to localize the offending place in the code.