Skip to contentSkip to conversation

Kernel hang on OMAP3 based Beagle board, RTC issue?

DISCUSSION Dirk Behme 6 participants 15 messages Jul 2, 2008 – Jul 31, 2008

Subsystems (4)
lore
Share & tools
lore

Dirk Behme

author
#0 lore raw

At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.

Any idea?

Thanks and please correct if something is wrong in the summary

Dirk

[1] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T07:19:47

[2] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T16:57:47

#1 lore raw
Quoted 19 lines

At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

Quoted 8 lines

May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.

Any idea?

Thanks and please correct if something is wrong in the summary

Dirk

Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

- Anand

#2 lore raw

Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:

Quoted 24 lines
Quoted 24 lines

At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.

Any idea?

Quoted 7 lines
Quoted 3 lines

Thanks and please correct if something is wrong in the summary

Dirk

Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

Tried that and I get the same hang, so I don't think the RTC is to blame.

author
#3 lore raw
Quoted 33 lines · Koen Kooi

Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:

Quoted 30 lines
Quoted 28 lines

At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.

Any idea?

Thanks and please correct if something is wrong in the summary

Dirk

Quoted 6 lines
Quoted 3 lines

Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

Tried that and I get the same hang, so I don't think the RTC is to blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP: Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

This is a 9-patch series by Jouni Hogander related to suspend and power-management.

Dirk

#4 lore raw
Quoted 63 lines

-----Original Message-----
From: Dirk Behme [mailto:[email protected]]
Sent: Saturday, July 05, 2008 12:56 PM
To: [email protected]
Cc: Gadiyar, Anand
Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

Quoted 34 lines · Koen Kooi

Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:

Quoted 31 lines
Quoted 29 lines

At Beagle IRC there was some discussion [1] [2] today about kernel hang on OMAP3 based Beagle board. I'll try to summarize discussion and Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. You don't see any crash, but after a while typing, anything doesn't work. SD stops as well, so you can only do stuff that's in ram, any IO will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When booting the git kernel after some time there are no timer or PRCM interrupts and the control resides in cpu_idle. We don't know where exactly the problem in RTC is, but while browsing the history of files related to RTC Khasim found that some changes were done to the files to handle T2 interrupts in a centralized way for USB, Battery and RTC. It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

May be this is the change that is affecting RTC. But it's unclear why it is affecting GP timer and PRCM interrupts.

Any idea?

Thanks and please correct if something is wrong in the summary

Dirk

Quoted 6 lines
Quoted 3 lines

Khasim also found that T2 RTC is not enabled by default in 3430sdp defconfig, but it appears to be enabled in the beagleboard defconfig. Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

Tried that and I get the same hang, so I don't think the RTC is to blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c
and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config
options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

Quoted 4 lines

This is a 9-patch series by Jouni Hogander related to suspend and power-management.

Dirk

I'm still working on the git-bisect. I'm currently waiting for ten minutes after boot before marking a commit good or bad. If the hang happens more than ten minutes after boot, I might miss that and mark the commit good - so I will re-check the good commits one more time and confirm.

- Anand

#5 lore raw

Hi All,

<snip>

Quoted 25 lines
Quoted 12 lines
Quoted 1 line

Tried that and I get the same hang, so I don't think the RTC is to blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

Quoted 4 lines

This is a 9-patch series by Jouni Hogander related to suspend and power-management.

Dirk

I'm still working on the git-bisect. I'm currently waiting for ten minutes
after boot before marking a commit good or bad. If the hang happens more
than ten minutes after boot, I might miss that and mark the commit good - so
I will re-check the good commits one more time and confirm.

Quoted 1 line

- Anand

As far as I can tell, the beagle hang issue aries after this commit:

397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit commit 397f49cde4e75cf662a23814097db35f3e6c6b62

Author: Jouni Hogander <[email protected]>
Date:   Fri May 16 13:58:25 2008 +0300

34XX: PM: Initial version of suspend and dynamic retention

This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and pm_idle.

Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.

These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx

Regards,
Anand

#6 lore raw

Hi,

Quoted 47 lines · Anand

Hi All,

<snip>

Quoted 25 lines
Quoted 12 lines
Quoted 1 line

Tried that and I get the same hang, so I don't think the RTC is to blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

Quoted 4 lines

This is a 9-patch series by Jouni Hogander related to suspend and power-management.

Dirk

I'm still working on the git-bisect. I'm currently waiting for ten minutes
after boot before marking a commit good or bad. If the hang happens more
than ten minutes after boot, I might miss that and mark the commit good - so
I will re-check the good commits one more time and confirm.

Quoted 1 line

- Anand

As far as I can tell, the beagle hang issue aries after this commit:

397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit commit 397f49cde4e75cf662a23814097db35f3e6c6b62

Author: Jouni Hogander <[email protected]>
Date:   Fri May 16 13:58:25 2008 +0300

34XX: PM: Initial version of suspend and dynamic retention

This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and pm_idle.

Quoted 7 lines

Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.

These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx

Unfortunately I have only sdp board here and I'm not able to reproduce this with it. Are you using omap3_beagle_defconfig without modifications there?

You mentioned prcm interrupts in this thread. Are you normally seeing them? I'm wondering this because you shouldn't see them with current l-o tree and default beagle config. Are you enabling sleep_while_idle (echo 1 > /sys/power/sleep_while_idle)?

You have end up to conclusion that "34XX: PM: Initial version of suspend and dynamic retention" is causing this? Could you try to apply this on top of mentioned commit and see if you still can see that hang:

arch/arm/mach-omap2/pm34xx.c +5 -5
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
                goto err1;
        }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
        if (mpu_pwrdm == NULL) {
Signature · 6 lines
Jouni Högander

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw

Hi,

Quoted 51 lines

Hi,

Quoted 47 lines · Anand

Hi All,

<snip>

Quoted 15 lines
Quoted 15 lines
Quoted 1 line

Tried that and I get the same hang, so I don't think the RTC is to blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

This is a 9-patch series by Jouni Hogander related to suspend and power-management.

Quoted 8 lines
Quoted 1 line

Dirk

I'm still working on the git-bisect. I'm currently waiting for ten minutes after boot before marking a commit good or bad. If the hang happens more than ten minutes after boot, I might miss that and mark the commit good - so I will re-check the good commits one more time and confirm.

- Anand

As far as I can tell, the beagle hang issue aries after this commit:

397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit commit 397f49cde4e75cf662a23814097db35f3e6c6b62

Author: Jouni Hogander <[email protected]>
Date:   Fri May 16 13:58:25 2008 +0300

34XX: PM: Initial version of suspend and dynamic retention

This is initial version of suspend and dynamic retention for 34xx. Omap is tried to put to full retention on suspend and pm_idle.

Could someone please test and confirm if the previous commit works without a hang? I waited for around 20 minutes and did not observe a hang.

Quoted 9 lines
Quoted 4 lines

These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx

Unfortunately I have only sdp board here and I'm not able to reproduce this with it. Are you using omap3_beagle_defconfig without modifications there?

Yes.

Quoted 5 lines

You mentioned prcm interrupts in this thread. Are you normally seeing them? I'm wondering this because you shouldn't see them with current l-o tree and default beagle config. Are you enabling sleep_while_idle (echo 1 > /sys/power/sleep_while_idle)?

Haven't tried this. All we did was boot the board from MMC and leave it running for around ten minutes.

Quoted 17 lines

You have end up to conclusion that "34XX: PM: Initial version of suspend and dynamic retention" is causing this? Could you try to apply this on top of mentioned commit and see if you still can see that hang:

arch/arm/mach-omap2/pm34xx.c +0 -3
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
                goto err1;
        }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5 -2
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
        if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current linux-omap tree.

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?

Regards,
Anand

#8 lore raw

Op 14 jul 2008, om 15:27 heeft Gadiyar, Anand het volgende geschreven:

Quoted 26 lines
Quoted 12 lines
arch/arm/mach-omap2/pm34xx.c +0 -3
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
               goto err1;
       }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5 -2
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
       mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
       if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current linux-omap tree.

Quoted 3 lines

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?

With that patch I still get the same behaviour:

serial console freezes, but responds to sysrq

I don't know if its related, but after the serial console freezes animated backgrounds in X stop animating, but will refresh if you press a key on the keyboard. The same effect can be seen with running 'top' on the screen. I'm using the CPUidle patches posted to the list a few weeks back + the timer suppression patch.

regards,

Koen

#9 lore raw
Quoted 24 lines · Koen Kooi
Quoted 24 lines
Quoted 9 lines
arch/arm/mach-omap2/pm34xx.c +0 -3
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
              goto err1;
      }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5 -2
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
      mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
      if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current
linux-omap tree.

Quoted 7 lines
Quoted 3 lines

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?

With that patch I still get the same behaviour:

serial console freezes, but responds to sysrq

Likewise on Beagle rev B4 here. Patching the PM code as above does not seem to make a difference; nor leaving out the RTC.

The failure mode that seems to recur here is that a userspace binary will exit, but no shell prompt is displayed afterwards. Once wedged, Sysrq-T provided some interesting output, included below. uart_wait_until_sent() ? That doesn't seem right.

N.B., on the Beagle here, TWL4030 IRQ handling has to be commented out to accomplish anything useful. Otherwise the console drowns in "TWL4030 module irq 368 is disabled but can't be masked!" messages on at least 90% of reboots. This is in drivers/i2c/chips/twl4030-core.c, commenting out the lines underneath:

/* install an irq handler to demultiplex the TWL4030 interrupt */

Have not yet had a chance to study the problem - could be a TWL4030 GPIO interrupt issue?

- Paul

<6>sh Ssh S c024cf70 c024cf70 0 362 1

    0   362      1

[<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0) from [<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8) (schedule_timeout+0x90/0xb8)
[<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8) (schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>] (schedule_timeout_interruptible+0x28/0x2c) (schedule_timeout_interruptible+0x28/0x2c)
r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e r5:ffff888e
r4:c7d9e000 r4:c7d9e000

[<c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c) (schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from [<c0053fe4>] (msleep_interruptible+0x28/0x58) (msleep_interruptible+0x28/0x58)
[<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58) (msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>] (uart_wait_until_sent+0x94/0xf8) (uart_wait_until_sent+0x94/0xf8)
r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000

[<c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8) (uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>] (tty_wait_until_sent+0xe4/0xf4) (tty_wait_until_sent+0xe4/0xf4)
r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640 r5:c7d6b640
r4:c7d9fd30 r4:c7d9fd30

[<c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4) (tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>] (set_termios+0x1d8/0x4cc)
(set_termios+0x1d8/0x4cc)
[<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios+0x0/0x4cc) from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8) (tty_mode_ioctl+0x24c/0x3d8)
[<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8) (tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>] (n_tty_ioctl+0x258/0x268)
(n_tty_ioctl+0x258/0x268)
r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc r5:beaac3fc
r4:c7d80400 r4:c7d80400

[<c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl+0x0/0x268) from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04) (tty_ioctl+0xda8/0xe04)
r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc r5:beaac3fc
r4:00005403 r4:00005403

[<c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl+0x0/0xe04) from [<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78) (vfs_ioctl+0x34/0x78)
[<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78) from [<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280) (do_vfs_ioctl+0x270/0x280)
r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740

[<c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280) (do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>] (sys_ioctl+0x40/0x64)
(sys_ioctl+0x40/0x64)
r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc r5:beaac3fc
r4:00000000 r4:00000000

[<c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64) from [<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c) (ret_fast_syscall+0x0/0x2c)
r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000 r5:00000000
r4:00000a31 r4:00000a31

#10 lore raw

Op 15 jul 2008, om 09:49 heeft Paul Walmsley het volgende geschreven:

Quoted 43 lines · Koen Kooi
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
arch/arm/mach-omap2/pm34xx.c +3 -5
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
             goto err1;
     }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains 

\n"); */

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 5 lines
arch/arm/mach-omap2/pm34xx.c +2 -0
+/*             goto err2; */
+/*     } */
 
     mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
     if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current
linux-omap tree.

Quoted 7 lines
Quoted 3 lines

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?

With that patch I still get the same behaviour:

serial console freezes, but responds to sysrq

Likewise on Beagle rev B4 here. Patching the PM code as above does not seem to make a difference; nor leaving out the RTC.

The failure mode that seems to recur here is that a userspace binary will exit, but no shell prompt is displayed afterwards. Once wedged, Sysrq-T provided some interesting output, included below. uart_wait_until_sent() ? That doesn't seem right.

Quoted 4 lines

N.B., on the Beagle here, TWL4030 IRQ handling has to be commented out to accomplish anything useful. Otherwise the console drowns in "TWL4030 module irq 368 is disabled but can't be masked!" messages on at least 90% of reboots.

Cold boots seem to 'fix' that.

Quoted 7 lines

This is in drivers/i2c/chips/twl4030-core.c, commenting out the lines underneath:

/* install an irq handler to demultiplex the TWL4030 interrupt */

Have not yet had a chance to study the problem - could be a TWL4030 GPIO interrupt issue?

As discussed on #beagle yesterday: we suspect i2c bus issues that upset the TWL4030, dropping the bus speed to 400kHz (from 2.6MHz) seems to fix most issues. There was some disagreement if 400kHz is a sane speed when looking at the pull-up resistor values.

regards,

Koen

Quoted 80 lines

- Paul

<6>sh Ssh S c024cf70 c024cf70 0 362 1
0 362 1
[<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0) from [<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8) (schedule_timeout+0x90/0xb8)
[<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8) (schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>] (schedule_timeout_interruptible+0x28/0x2c) (schedule_timeout_interruptible+0x28/0x2c)
r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000

[<c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c) (schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from [<c0053fe4>] (msleep_interruptible+0x28/0x58) (msleep_interruptible+0x28/0x58)
[<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58) (msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>] (uart_wait_until_sent+0x94/0xf8) (uart_wait_until_sent+0x94/0xf8)
r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000

[<c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8) (uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>] (tty_wait_until_sent+0xe4/0xf4) (tty_wait_until_sent+0xe4/0xf4)
r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640 r5:c7d6b640 r4:c7d9fd30 r4:c7d9fd30

[<c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4) (tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>] (set_termios+0x1d8/0x4cc)
(set_termios+0x1d8/0x4cc)
[<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios +0x0/0x4cc)
from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8) (tty_mode_ioctl+0x24c/0x3d8)
[<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8) (tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>] (n_tty_ioctl+0x258/0x268)
(n_tty_ioctl+0x258/0x268)
r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc r5:beaac3fc r4:c7d80400 r4:c7d80400

[<c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl +0x0/0x268)
from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04) (tty_ioctl+0xda8/0xe04)
r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc r5:beaac3fc r4:00005403 r4:00005403

[<c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl +0x0/0xe04) from
[<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78) (vfs_ioctl+0x34/0x78)
[<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78) from [<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280) (do_vfs_ioctl+0x270/0x280)
r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740

[<c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280) (do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>] (sys_ioctl+0x40/0x64)
(sys_ioctl+0x40/0x64)
r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc r5:beaac3fc r4:00000000 r4:00000000

[<c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64) from [<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c) (ret_fast_syscall+0x0/0x2c)
r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000 r5:00000000 r4:00000a31 r4:00000a31

#11 lore raw
In reply to Koen Kooi
Quoted 41 lines
Quoted 26 lines
Quoted 12 lines
arch/arm/mach-omap2/pm34xx.c +0 -3
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
               goto err1;
       }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 10 lines
arch/arm/mach-omap2/pm34xx.c +5 -2
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
       mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
       if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current linux-omap tree.

Quoted 3 lines

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang happens, but could it be because the SDP uses ttyS0 by default as the console while the Beagle is on ttyS2?

With that patch I still get the same behaviour:

serial console freezes, but responds to sysrq

I don't know if its related, but after the serial console freezes animated backgrounds in X stop animating, but will refresh if you press a key on the keyboard. The same effect can be seen with running 'top' on the screen.
I'm using the CPUidle patches posted to the list a few weeks back + the timer suppression patch.

Quoted 3 lines

regards,

Koen

Okay. Guess I should have waited longer after applying the patch.

- Anand

#12 lore raw
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
arch/arm/mach-omap2/pm34xx.c +5 -5
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
               goto err1;
       }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

arch/arm/mach-omap2/pm34xx.c int __init omap3_pm_init(void) · line 390

Quoted 2 lines
arch/arm/mach-omap2/pm34xx.c +0 -0
       mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
       if (mpu_pwrdm == NULL) {

Hi Jouni,

Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?

Thanks.

Regards,
Khasim

#13 lore raw

Op 31 jul 2008, om 15:08 heeft Syed Mohammed, Khasim het volgende geschreven:

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
arch/arm/mach-omap2/pm34xx.c +3 -5
index 202c269..308835b 100644
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
              goto err1;
      }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains 

\n"); */

arch/arm/mach-omap2/pm34xx.c +2 -0
+/*             goto err2; */
+/*     } */
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
      mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
      if (mpu_pwrdm == NULL) {

Hi Jouni,

Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?

I've had these lines commented out for some weeks now and still get the hang.

Quoted 12 lines

Thanks.

Regards,
Khasim

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw

Hi Koen:

Quoted 7 lines

-----Original Message-----
From: Koen Kooi [mailto:[email protected]]
Sent: Thursday, July 31, 2008 7:03 PM
To: Syed Mohammed, Khasim
Cc: Gadiyar, Anand; H?gander Jouni; Dirk Behme; [email protected]
Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

<snip>

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Hi Jouni,

Quoted 8 lines
Quoted 4 lines

Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?

I've had these lines commented out for some weeks now and still get the hang.

I didn't apply any other patches to OMAP GIT, I just cloned a new version of OMAP GIT and built it for Beagle using default config, also commented the lines mentioned by Jouni.

If I don't comment these lines then UART hangs immediately after 30min approx, with lines commented I am running for 10 hrs now with out a hang.

Any way the issue could still exists, I would like to know what exactly this code does to help me in focusing on a specific area.

May be we miss porting some lines of code to Beagle.

Regards,
Khasim