2.6.25-rc2-mm1 - boot hangs on ia64
25-rc2-mm1 is hanging early in boot on my HP ia64 numa platform. I saw the "Strange hang on ia64 with CONFIG_PRINTK_TIME=y" thread on lkml:
http://marc.info/?t0288396800001&r=1&w=4
However, my config does not include PRINTK_TIME=y. In fact, hang occurs with ia64 defconfig as well--right after the "Loading...initrd...done" message. 2.6.25-rc2 boots OK.
Bisecting the broken-out series appears to indict 'git-sched.patch'. I went ahead and added Ingo's patch, discussed in the "strange hang" thread, even tho' I hadn't enabled printk timestamps. No effect.
Anyone else seeing this?
Regards,
Lee
Hi
Fujitsu machine can't boot too.
my bisect indicate git-sched.patch cause regression too.
Thanks.
Quoted 14 lines
25-rc2-mm1 is hanging early in boot on my HP ia64 numa platform. I saw the "Strange hang on ia64 with
CONFIG_PRINTK_TIME=y" thread on lkml:http://marc.info/?t0288396800001&r=1&w=4However, my config does not include PRINTK_TIME=y. In fact, hang occurs with ia64 defconfig as well--right after the "Loading...initrd...done" message. 2.6.25-rc2 boots OK.
Bisecting the broken-out series appears to indict 'git-sched.patch'. I went ahead and added Ingo's patch, discussed in the "strange hang" thread, even tho' I hadn't enabled printk timestamps. No effect.
Anyone else seeing this?
Quoted 2 lines · KOSAKI Motohiro
Fujitsu machine can't boot too. my bisect indicate git-sched.patch cause regression too.
hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:
http://people.redhat.com/mingo/sched-devel.git/README
it's just a handful of commits so it should be relatively quick to figure out. My only guess would be:
Subject: sched: make early bootup sched_clock() use safer
but i think this has been ruled out before ...
Ingo
Hi Ingo
Quoted 7 lines
Quoted 2 lines
Fujitsu machine can't boot too. my bisect indicate git-sched.patch cause regression too.
hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:
How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?
Quoted 6 lines
it's just a handful of commits so it should be relatively quick to figure out. My only guess would be:
Subject: sched: make early bootup
sched_clock()use saferbut i think this has been ruled out before ...
rc2-mm1 + that patch doesn't boot too.
stop at the same point ;)
Quoted 7 lines · KOSAKI Motohiro
Quoted 4 lines
hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:
How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?
yeah, please. If it's caused by sched-devel.git then you should see the hang there too.
Ingo
Hi Ingo,
CC'ed Steven Rostedt
I found the following patch cause regression by bisect.
2.6.25-rc2-mm1: doesn't boot
2.6.25-rc2-mm1 + revert following patch: works well
but I think it is very strange. runqueue_is_locked() seems simple and have not bug. ;)
What do you think this problem?
-------------------------------------------------------------------
commit 033394c2c097215ac556a446154af24fbf18b064
Author: Steven Rostedt <[email protected]> Date: Mon Feb 25 21:15:44 2008 +0100
printk: dont wake up klogd with the rq locked
It is not wise to place a printk where the runqueue lock is held.
Quoted 13 lines · KOSAKI Motohiro
Quoted 8 lines
Quoted 5 lines
hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:
How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?
yeah, please. If it's caused by sched-devel.git then you should see the hang there too.
Quoted 5 lines · KOSAKI Motohiro
Hi Ingo,
CC'ed Steven RostedtI found the following patch cause regression by bisect.
Quoted 7 lines
2.6.25-rc2-mm1: doesn't boot
2.6.25-rc2-mm1 + revert following patch: works wellbut I think it is very strange.
runqueue_is_locked()seems simple and have not bug. ;)What do you think this problem?
thanks for bisecting it down! Could ia64 have trouble accessing the percpu data structures of the scheduler?
does the patch below resolve the hang?
Ingo
------------------------->
Subject: sched: fix wake_up_klogd()
From: Ingo Molnar <[email protected]>
Date: Thu Feb 28 12:42:45 CET 2008
on some platforms if we printk too early it might not be safe to call into the scheduler data structures.
--- kernel/printk.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Index: linux/kernel/printk.c =================================--- linux.orig/kernel/printk.c +++ linux/kernel/printk.c
@@ -948,7 +948,8 @@ int is_console_locked(void)
void wake_up_klogd(void)
{
- if (!oops_in_progress && waitqueue_active(&log_wait))
+ if (!oops_in_progress && waitqueue_active(&log_wait) &&
+ !runqueue_is_locked())
wake_up_interruptible(&log_wait);
}
@@ -1000,7 +1001,7 @@ void release_console_sem(void)
* If we try to wake up klogd while printing with the runqueue lock
* held, this will deadlock.
*/
- if (wake_klogd && !runqueue_is_locked())
+ if (wake_klogd)
wake_up_klogd();
}
EXPORT_SYMBOL(release_console_sem);
KOSAKI Motohiro
Tested-byHi Ingo,
Quoted 4 lines
thanks for bisecting it down! Could ia64 have trouble accessing the percpu data structures of the scheduler?
does the patch below resolve the hang?
Thanks!
that patch works well on my test environment.
BTW: Your work is ultimate fast. it's wonderful :)
Review void release_console_sem(void) · line 1006
Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1
-1@@ -1000,7 +1001,7 @@ void release_console_sem(void) * If we try to wake up klogd while printing with the runqueue lock * held, this will deadlock. */ - if (wake_klogd && !runqueue_is_locked()) + if (wake_klogd) wake_up_klogd(); }
I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.
Quoted 14 lines · Andrew Morton
Review void release_console_sem(void) · line 1006
Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1
-1@@ -1000,7 +1001,7 @@ void release_console_sem(void) * If we try to wake up klogd while printing with the runqueue lock * held, this will deadlock. */ - if (wake_klogd && !runqueue_is_locked()) + if (wake_klogd) wake_up_klogd(); }I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.
well, the problem was ia64, not a problem on x86 or other platforms. The problem here is ia64 not setting up percpu data structures soon enough. It has blown up in the past in other areas, and it will likely blow up in the future in other areas as well. It's just not robust to have init dependencies on such basic data structures like percpu areas like that.
Ingo
Quoted 18 lines · On Thu, 28 Feb 2008 20:12:14 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 14 lines · Andrew Morton
Review void release_console_sem(void) · line 1006
Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1
-1@@ -1000,7 +1001,7 @@ void release_console_sem(void) * If we try to wake up klogd while printing with the runqueue lock * held, this will deadlock. */ - if (wake_klogd && !runqueue_is_locked()) + if (wake_klogd) wake_up_klogd(); }I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.
Quoted 1 line
well, the problem was ia64, not a problem on x86 or other platforms.
I am referring to the original change which made klogd wakeups unreliable.
Quoted 9 lines · Andrew Morton
Quoted 6 lines
Quoted 4 lines
I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.
well, the problem was ia64, not a problem on x86 or other platforms.
I am referring to the original change which made klogd wakeups unreliable.
oh, indeed - agreed - i missed the fact that is_locked check is sporadic and can cause other CPUs to prevent the wakeup of klogd. I've zapped the change.
Ingo