Kernel hang on OMAP3 based Beagle board, RTC issue?
At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.
Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.
Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set
May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.
Any idea?
Thanks and please correct if something is wrong in the summary
Dirk
[1] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T07:19:47
[2] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T16:57:47
Quoted 19 lines
At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.
Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.
Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in
cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set
Quoted 8 lines
May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.
Any idea?
Thanks and please correct if something is wrong in the summary
Dirk
Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
- Anand
Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:
Quoted 24 lines
Quoted 24 lines
At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.
Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.
Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in
cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch setMay be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.
Any idea?
Quoted 7 lines
Quoted 3 lines
Thanks and please correct if something is wrong in the summary
Dirk
Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
Tried that and I get the same hang, so I don't think the RTC is to blame.
Quoted 33 lines · Koen Kooi
Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:
Quoted 30 lines
Quoted 28 lines
At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.
Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.
Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in
cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch setMay be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.
Any idea?
Thanks and please correct if something is wrong in the summary
Dirk
Quoted 6 lines
Quoted 3 lines
Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
Tried that and I get the same hang, so I don't think the RTC is to blame.
Short update just in case anybody has an idea:
Gadiyar spent some time with git-bisect (thanks!):
It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP: Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND
This is a 9-patch series by Jouni Hogander related to suspend and power-management.
Dirk
Quoted 63 lines
-----Original Message-----
From: Dirk Behme [mailto:[email protected]]
Sent: Saturday, July 05, 2008 12:56 PM
To: [email protected]
Cc: Gadiyar, Anand
Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?Quoted 34 lines · Koen Kooi
Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:
Quoted 31 lines
Quoted 29 lines
At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.
Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.
Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in
cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch setMay be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.
Any idea?
Thanks and please correct if something is wrong in the summary
Dirk
Quoted 6 lines
Quoted 3 lines
Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
Tried that and I get the same hang, so I don't think the RTC is to blame.
Short update just in case anybody has an idea:
Gadiyar spent some time with git-bisect (thanks!):
It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Movepm.ctopm24xx.c
andsleep.Stosleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config
optionsCONFIG_OMAP_RESET_CLOCKSandCONFIG_SUSPEND
Quoted 4 lines
This is a 9-patch series by Jouni Hogander related to suspend and power-management.
Dirk
I'm still working on the git-bisect. I'm currently waiting for ten minutes after boot before marking a commit good or bad. If the hang happens more than ten minutes after boot, I might miss that and mark the commit good - so I will re-check the good commits one more time and confirm.
- Anand
Hi All,
<snip>
Quoted 25 lines
Quoted 12 lines
Quoted 1 line
Tried that and I get the same hang, so I don't think the RTC is to blame.
Short update just in case anybody has an idea:
Gadiyar spent some time with git-bisect (thanks!):
It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Movepm.ctopm24xx.candsleep.Stosleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config optionsCONFIG_OMAP_RESET_CLOCKSandCONFIG_SUSPENDQuoted 4 lines
This is a 9-patch series by Jouni Hogander related to suspend and power-management.
Dirk
I'm still working on the git-bisect. I'm currently waiting for ten minutes
after boot before marking a commit good or bad. If the hang happens more
than ten minutes after boot, I might miss that and mark the commit good - so
I will re-check the good commits one more time and confirm.
Quoted 1 line
- Anand
As far as I can tell, the beagle hang issue aries after this commit:
397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit commit 397f49cde4e75cf662a23814097db35f3e6c6b62
Author: Jouni Hogander <[email protected]> Date: Fri May 16 13:58:25 2008 +0300
34XX: PM: Initial version of suspend and dynamic retention
This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and pm_idle.
Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.
These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx
Regards,
Anand
Hi,
Quoted 47 lines · Anand
Hi All,
<snip>
Quoted 25 lines
Quoted 12 lines
Quoted 1 line
Tried that and I get the same hang, so I don't think the RTC is to blame.
Short update just in case anybody has an idea:
Gadiyar spent some time with git-bisect (thanks!):
It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Movepm.ctopm24xx.candsleep.Stosleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config optionsCONFIG_OMAP_RESET_CLOCKSandCONFIG_SUSPENDQuoted 4 lines
This is a 9-patch series by Jouni Hogander related to suspend and power-management.
Dirk
I'm still working on the git-bisect. I'm currently waiting for ten minutes
after boot before marking a commit good or bad. If the hang happens more
than ten minutes after boot, I might miss that and mark the commit good - so
I will re-check the good commits one more time and confirm.Quoted 1 line
- Anand
As far as I can tell, the beagle hang issue aries after this commit:
397f49cde4e75cf662a23814097db35f3e6c6b62is first bad commit commit397f49cde4e75cf662a23814097db35f3e6c6b62Author: Jouni Hogander <[email protected]> Date: Fri May 16 13:58:25 2008 +030034XX: PM: Initial version of suspend and dynamic retention
This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and
pm_idle.Signed-off-by Jouni Hogander <[email protected]>Signed-off-by Tony Lindgren <[email protected]>
Quoted 7 lines
Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.
These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx
Unfortunately I have only sdp board here and I'm not able to reproduce this with it. Are you using omap3_beagle_defconfig without modifications there?
You mentioned prcm interrupts in this thread. Are you normally seeing them? I'm wondering this because you shouldn't see them with current l-o tree and default beagle config. Are you enabling sleep_while_idle (echo 1 > /sys/power/sleep_while_idle)?
You have end up to conclusion that "34XX: PM: Initial version of suspend and dynamic retention" is causing this? Could you try to apply this on top of mentioned commit and see if you still can see that hang:
arch/arm/mach-omap2/pm34xx.c +5 -5
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" ); - goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {
Signature · 6 lines
Jouni Högander -- To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,
Quoted 51 lines
Hi,
Quoted 47 lines · Anand
Hi All,
<snip>
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Tried that and I get the same hang, so I don't think the RTC is to blame.
Short update just in case anybody has an idea:
Gadiyar spent some time with git-bisect (thanks!):
It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Movepm.ctopm24xx.candsleep.Stosleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config optionsCONFIG_OMAP_RESET_CLOCKSandCONFIG_SUSPENDThis is a 9-patch series by Jouni Hogander related to suspend and power-management.
Quoted 8 lines
Quoted 1 line
Dirk
I'm still working on the git-bisect. I'm currently waiting for ten minutes after boot before marking a commit good or bad. If the hang happens more than ten minutes after boot, I might miss that and mark the commit good - so I will re-check the good commits one more time and confirm.
- Anand
As far as I can tell, the beagle hang issue aries after this commit:
397f49cde4e75cf662a23814097db35f3e6c6b62is first bad commit commit397f49cde4e75cf662a23814097db35f3e6c6b62Author: Jouni Hogander <[email protected]> Date: Fri May 16 13:58:25 2008 +030034XX: PM: Initial version of suspend and dynamic retention
This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and
pm_idle.Signed-off-by Jouni Hogander <[email protected]>Signed-off-by Tony Lindgren <[email protected]>Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.
Quoted 9 lines
Quoted 4 lines
These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xxUnfortunately I have only sdp board here and I'm not able to reproduce this with it. Are you using
omap3_beagle_defconfigwithout modifications there?
Yes.
Quoted 5 lines
You mentioned prcm interrupts in this thread. Are you normally seeing them? I'm wondering this because you shouldn't see them with current l-o tree and default beagle config. Are you enabling
sleep_while_idle(echo 1 >/sys/power/sleep_while_idle)?
Haven't tried this. All we did was boot the board from MMC and leave it running for around ten minutes.
Quoted 17 lines
You have end up to conclusion that "34XX: PM: Initial version of suspend and dynamic retention" is causing this? Could you try to apply this on top of mentioned commit and see if you still can see that hang:
arch/arm/mach-omap2/pm34xx.c +0
-3@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" );
arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5
-2- goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {
With this change, the hang does not happene. Applied this patch on the current linux-omap tree.
Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?
Regards,
Anand
Op 14 jul 2008, om 15:27 heeft Gadiyar, Anand het volgende geschreven:
Quoted 26 lines
Quoted 12 lines
arch/arm/mach-omap2/pm34xx.c +0
-3@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" );arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5
-2- goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {With this change, the hang does not happene. Applied this patch on the current linux-omap tree.
Quoted 3 lines
Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?
With that patch I still get the same behaviour:
serial console freezes, but responds to sysrq
I don't know if its related, but after the serial console freezes animated backgrounds in X stop animating, but will refresh if you press a key on the keyboard. The same effect can be seen with running 'top' on the screen. I'm using the CPUidle patches posted to the list a few weeks back + the timer suppression patch.
regards,
Koen
Quoted 24 lines · Koen Kooi
Quoted 24 lines
Quoted 9 lines
arch/arm/mach-omap2/pm34xx.c +0
-3@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" );arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5
-2- goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {With this change, the hang does not happene. Applied this patch on the current
linux-omap tree.
Quoted 7 lines
Quoted 3 lines
Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?
With that patch I still get the same behaviour:
serial console freezes, but responds to sysrq
Likewise on Beagle rev B4 here. Patching the PM code as above does not seem to make a difference; nor leaving out the RTC.
The failure mode that seems to recur here is that a userspace binary will exit, but no shell prompt is displayed afterwards. Once wedged, Sysrq-T provided some interesting output, included below. uart_wait_until_sent() ? That doesn't seem right.
N.B., on the Beagle here, TWL4030 IRQ handling has to be commented out to accomplish anything useful. Otherwise the console drowns in "TWL4030 module irq 368 is disabled but can't be masked!" messages on at least 90% of reboots. This is in drivers/i2c/chips/twl4030-core.c, commenting out the lines underneath:
/* install an irq handler to demultiplex the TWL4030 interrupt */
Have not yet had a chance to study the problem - could be a TWL4030 GPIO interrupt issue?
- Paul
<6>sh Ssh S c024cf70 c024cf70 0 362 1
0 362 1
[<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0) from [<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8) (schedule_timeout+0x90/0xb8)
[<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8) (schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>] (schedule_timeout_interruptible+0x28/0x2c) (schedule_timeout_interruptible+0x28/0x2c)
r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e r5:ffff888e
r4:c7d9e000 r4:c7d9e000
[<c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c) (schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from [<c0053fe4>] (msleep_interruptible+0x28/0x58) (msleep_interruptible+0x28/0x58)
[<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58) (msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>] (uart_wait_until_sent+0x94/0xf8) (uart_wait_until_sent+0x94/0xf8)
r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000
[<c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8) (uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>] (tty_wait_until_sent+0xe4/0xf4) (tty_wait_until_sent+0xe4/0xf4)
r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640 r5:c7d6b640
r4:c7d9fd30 r4:c7d9fd30
[<c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4) (tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>] (set_termios+0x1d8/0x4cc)
(set_termios+0x1d8/0x4cc)
[<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios+0x0/0x4cc) from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8) (tty_mode_ioctl+0x24c/0x3d8)
[<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8) (tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>] (n_tty_ioctl+0x258/0x268)
(n_tty_ioctl+0x258/0x268)
r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc r5:beaac3fc
r4:c7d80400 r4:c7d80400
[<c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl+0x0/0x268) from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04) (tty_ioctl+0xda8/0xe04)
r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc r5:beaac3fc
r4:00005403 r4:00005403
[<c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl+0x0/0xe04) from [<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78) (vfs_ioctl+0x34/0x78)
[<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78) from [<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280) (do_vfs_ioctl+0x270/0x280)
r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740
[<c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280) (do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>] (sys_ioctl+0x40/0x64)
(sys_ioctl+0x40/0x64)
r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc r5:beaac3fc
r4:00000000 r4:00000000
[<c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64) from [<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c) (ret_fast_syscall+0x0/0x2c)
r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000 r5:00000000
r4:00000a31 r4:00000a31
Op 15 jul 2008, om 09:49 heeft Paul Walmsley het volgende geschreven:
Quoted 43 lines · Koen Kooi
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
arch/arm/mach-omap2/pm34xx.c +3
-5@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" ); - goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains \n"); */
arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 5 lines
arch/arm/mach-omap2/pm34xx.c +2
-0+/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {With this change, the hang does not happene. Applied this patch on the current
linux-omap tree.Quoted 7 lines
Quoted 3 lines
Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?
With that patch I still get the same behaviour:
serial console freezes, but responds to sysrq
Likewise on Beagle rev B4 here. Patching the PM code as above does not seem to make a difference; nor leaving out the RTC.
The failure mode that seems to recur here is that a userspace binary will exit, but no shell prompt is displayed afterwards. Once wedged, Sysrq-T provided some interesting output, included below.
uart_wait_until_sent()? That doesn't seem right.
Quoted 4 lines
N.B., on the Beagle here, TWL4030 IRQ handling has to be commented out to accomplish anything useful. Otherwise the console drowns in "TWL4030 module irq 368 is disabled but can't be masked!" messages on at least 90% of reboots.
Cold boots seem to 'fix' that.
Quoted 7 lines
This is in
drivers/i2c/chips/twl4030-core.c, commenting out the lines underneath:/* install an irq handler to demultiplex the TWL4030 interrupt */
Have not yet had a chance to study the problem - could be a TWL4030 GPIO interrupt issue?
As discussed on #beagle yesterday: we suspect i2c bus issues that upset the TWL4030, dropping the bus speed to 400kHz (from 2.6MHz) seems to fix most issues. There was some disagreement if 400kHz is a sane speed when looking at the pull-up resistor values.
regards,
Koen
Quoted 80 lines
- Paul
<6>sh Ssh S
c024cf70c024cf700 362 1
0 362 1
[<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0) from [<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8) (schedule_timeout+0x90/0xb8)
[<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8) (schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>] (schedule_timeout_interruptible+0x28/0x2c) (schedule_timeout_interruptible+0x28/0x2c)
r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000[<
c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c) (schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from [<c0053fe4>] (msleep_interruptible+0x28/0x58) (msleep_interruptible+0x28/0x58)
[<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58) (msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>] (uart_wait_until_sent+0x94/0xf8) (uart_wait_until_sent+0x94/0xf8)
r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000[<
c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8) (uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>] (tty_wait_until_sent+0xe4/0xf4) (tty_wait_until_sent+0xe4/0xf4)
r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640 r5:c7d6b640 r4:c7d9fd30 r4:c7d9fd30[<
c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4) (tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>] (set_termios+0x1d8/0x4cc)
(set_termios+0x1d8/0x4cc)
[<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios+0x0/0x4cc)
from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8) (tty_mode_ioctl+0x24c/0x3d8)
[<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8) (tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>] (n_tty_ioctl+0x258/0x268)
(n_tty_ioctl+0x258/0x268)
r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc r5:beaac3fc r4:c7d80400 r4:c7d80400[<
c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl+0x0/0x268)
from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04) (tty_ioctl+0xda8/0xe04)
r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc r5:beaac3fc r4:00005403 r4:00005403[<
c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl+0x0/0xe04) from
[<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78) (vfs_ioctl+0x34/0x78)
[<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78) from [<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280) (do_vfs_ioctl+0x270/0x280)
r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740[<
c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280) (do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>] (sys_ioctl+0x40/0x64)
(sys_ioctl+0x40/0x64)
r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc r5:beaac3fc r4:00000000 r4:00000000[<
c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64) from [<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c) (ret_fast_syscall+0x0/0x2c)
r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000 r5:00000000 r4:00000a31 r4:00000a31
Quoted 41 lines
Quoted 26 lines
Quoted 12 lines
arch/arm/mach-omap2/pm34xx.c +0
-3@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" );arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5
-2- goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */ mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {With this change, the hang does not happene. Applied this patch on the current linux-omap tree.
Quoted 3 lines
Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?
With that patch I still get the same behaviour:
serial console freezes, but responds to sysrq
I don't know if its related, but after the serial console freezes animated backgrounds in X stop animating, but will refresh if you press a key on the keyboard. The same effect can be seen with running 'top' on the screen.
I'm using the CPUidle patches posted to the list a few weeks back + the timer suppression patch.
Quoted 3 lines
regards,
Koen
Okay. Guess I should have waited longer after applying the patch.
- Anand
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
arch/arm/mach-omap2/pm34xx.c +5
-5@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" ); - goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains\n"); */ +/* goto err2; */ +/* } */
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390
Quoted 2 lines
arch/arm/mach-omap2/pm34xx.c +0
-0mpu_pwrdm =pwrdm_lookup ("mpu_pwrdm" );if (mpu_pwrdm ==NULL ) {
Hi Jouni,
Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?
Thanks.
Regards,
Khasim
Op 31 jul 2008, om 15:08 heeft Syed Mohammed, Khasim het volgende geschreven:
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
arch/arm/mach-omap2/pm34xx.c +3
-5@@ -380,11 +380,11 @@ int __init omap3_pm_init(void) goto err1; } - ret =pwrdm_for_each (pwrdms_setup); -if (ret) { -printk (KERN_ERR "Failed to setup powerdomains\n" ); - goto err2; - } +/* ret = pwrdm_for_each(pwrdms_setup); */ +/* if (ret) { */ +/* printk(KERN_ERR "Failed to setup powerdomains \n"); */
arch/arm/mach-omap2/pm34xx.c +2
-0+/* goto err2; */ +/* } */
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
mpu_pwrdm = pwrdm_lookup("mpu_pwrdm"); if (mpu_pwrdm == NULL) {Hi Jouni,
Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?
I've had these lines commented out for some weeks now and still get the hang.
Quoted 12 lines
Thanks.
Regards,
Khasim--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Koen:
Quoted 7 lines
-----Original Message-----
From: Koen Kooi [mailto:[email protected]]
Sent: Thursday, July 31, 2008 7:03 PM
To: Syed Mohammed, Khasim
Cc: Gadiyar, Anand; H?gander Jouni; Dirk Behme; [email protected]
Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?
<snip>
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Hi Jouni,
Quoted 8 lines
Quoted 4 lines
Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?
I've had these lines commented out for some weeks now and still get the hang.
I didn't apply any other patches to OMAP GIT, I just cloned a new version of OMAP GIT and built it for Beagle using default config, also commented the lines mentioned by Jouni.
If I don't comment these lines then UART hangs immediately after 30min approx, with lines commented I am running for 10 hrs now with out a hang.
Any way the issue could still exists, I would like to know what exactly this code does to help me in focusing on a specific area.
May be we miss porting some lines of code to Beagle.
Regards,
Khasim