CPU_IDLE causes random reboots on custom 4430
Hi,
We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly bean. The board works quite well, but we experience random hangs and the watchdog kicks the board to reboot.
The problem is that kmsg_last doesn't not show any useful information and the fiq debugger refuses to work when the device is locked up. We also enabled various kernel lock debugging configs to get more info ... in vain.
Experimenting with it for a while (removing non-standard drivers like accel, charger, battery, etc) we found that removing CONFIG_CPU_IDLE or setting disallow_smp_idle in cpuidle44xx.c seems to cure the random lockups and make the board work fine (apart from the maybe obvious consequences that it doesn't go to deep sleep anymore).
Further below a few sample infos from the board.
We'd very much appreciate any further help on how to proceed to find the root cause and happy to provide any more info from sys, debug or omapconf tool.
Rgds,
Chris
A dmesg from a normal boot:
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/dmesg.log
Kernel source:
https://github.com/Kuzma30/kernel3NookTablet/tree/jellybean
omapconf --cpuinfo
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/omapconf.cpuinfo
cat /sys/kernel/debug/clock/summary
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/clock.summary
... what else can we provide to track this down?
Quoted 7 lines · Chris Hoffmann
Hi,
We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly bean. The board works quite well, but we experience random hangs and the watchdog kicks the board to reboot.
On the same kernel, you should have support for the persistent log. You might want to check the output. That should give you pointers on what CPU was doing before the freeze which resulted in reboot.
Regards
Santosh
Quoted 11 lines · Santosh
Quoted 7 lines · Chris Hoffmann
Hi,
We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly bean. The board works quite well, but we experience random hangs and the watchdog kicks the board to reboot.
On the same kernel, you should have support for the persistent log. You might want to check the output. That should give you pointers on what CPU was doing before the freeze which resulted in reboot.
Hi,
I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o the problem doesn't seem to occur. It could be that printk-ing alleviates the issue.
Also the watchdog seems to shutdown the device rather than rebooting it (or it hangs?) and then I can't provide /proc/last_kmsg.
How could I provide more info?
Rgds,
Chris
Quoted 20 lines · Chris Hoffmann
Quoted 11 lines · Santosh
Quoted 8 lines · Chris Hoffmann
Hi,
We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly bean. The board works quite well, but we experience random hangs and the watchdog kicks the board to reboot.
Quoted 4 lines
On the same kernel, you should have support for the persistent log. You might
want to check the output. That should give you pointers on what CPU was doing before the freeze which resulted in reboot.Hi,
Quoted 9 lines
I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o the problem doesn't seem to occur. It could be that printk-ing alleviates the issue.
Also the watchdog seems to shutdown the device rather than rebooting it (or it hangs?) and then I can't provide
/proc/last_kmsg.How could I provide more info?
Check if you have "/sys/kernel/debug/persistent_trace" available on your kernel. This generally helps whenever there are hangs, the last call stack is stored on memory and on the reboot it can be cat'ed to see if some useful information about hang is available.
Regards
Santosh
Quoted 28 lines · Santosh
Quoted 27 lines · Chris Hoffmann
Quoted 11 lines · Santosh
Quoted 8 lines · Chris Hoffmann
Hi,
We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly bean. The board works quite well, but we experience random hangs and the watchdog kicks the board to reboot.
Quoted 4 lines
On the same kernel, you should have support for the persistent log. You might
want to check the output. That should give you pointers on what CPU was doing before the freeze which resulted in reboot.Hi,
I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o the problem doesn't seem to occur. It could be that printk-ing alleviates the issue.
Also the watchdog seems to shutdown the device rather than rebooting it (or it hangs?) and then I can't provide
/proc/last_kmsg.
Quoted 6 lines
Quoted 2 lines
How could I provide more info?
Check if you have "
/sys/kernel/debug/persistent_trace" available on your kernel. This generally helps whenever there are hangs, the last call stack is stored on memory and on the reboot it can be cat'ed to see if some useful information about hang is available.
Hi Santosh, all,
the p-android-omap-3.0 doesn't have the persistent_trace but I was able to backport it from 3.4 without major issues (only tricky part is that in p-android-3.4 there's no apparent user of that device in omapzoom kernel).
The problem is now that the omap-watchdog doesn't kick the device to reboot but rather to shutdown, so I still have no trace. Soft-rebooting shows that I can get persistent_trace.
Switching the persistent tracer to ecc=true does not help as it seems to overload the device completely when activating the tracing.
Rgds,
Chris