Skip to contentSkip to conversation

2.6.25-rc2-mm1 - boot hangs on ia64

DISCUSSION Lee Schermerhorn 4 participants 12 messages Feb 25, 2008 – Feb 28, 2008

lore
Share & tools
lore
#0 lore raw

25-rc2-mm1 is hanging early in boot on my HP ia64 numa platform. I saw the "Strange hang on ia64 with CONFIG_PRINTK_TIME=y" thread on lkml:

	http://marc.info/?t0288396800001&r=1&w=4

However, my config does not include PRINTK_TIME=y. In fact, hang occurs with ia64 defconfig as well--right after the "Loading...initrd...done" message. 2.6.25-rc2 boots OK.

Bisecting the broken-out series appears to indict 'git-sched.patch'. I went ahead and added Ingo's patch, discussed in the "strange hang" thread, even tho' I hadn't enabled printk timestamps. No effect.

Anyone else seeing this?

Regards,
Lee

#1 lore raw

Hi

Fujitsu machine can't boot too.
my bisect indicate git-sched.patch cause regression too.

Thanks.

Quoted 14 lines

25-rc2-mm1 is hanging early in boot on my HP ia64 numa platform. I saw the "Strange hang on ia64 with CONFIG_PRINTK_TIME=y" thread on lkml:

	http://marc.info/?t0288396800001&r=1&w=4

However, my config does not include PRINTK_TIME=y. In fact, hang occurs with ia64 defconfig as well--right after the "Loading...initrd...done" message. 2.6.25-rc2 boots OK.

Bisecting the broken-out series appears to indict 'git-sched.patch'. I went ahead and added Ingo's patch, discussed in the "strange hang" thread, even tho' I hadn't enabled printk timestamps. No effect.

Anyone else seeing this?

#2 lore raw
Quoted 2 lines · KOSAKI Motohiro

Fujitsu machine can't boot too. my bisect indicate git-sched.patch cause regression too.

hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:

http://people.redhat.com/mingo/sched-devel.git/README

it's just a handful of commits so it should be relatively quick to figure out. My only guess would be:

Subject: sched: make early bootup sched_clock() use safer

but i think this has been ruled out before ...

Ingo

#3 lore raw

Hi Ingo

Quoted 7 lines
Quoted 2 lines

Fujitsu machine can't boot too. my bisect indicate git-sched.patch cause regression too.

hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:

http://people.redhat.com/mingo/sched-devel.git/README

How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?

Quoted 6 lines

it's just a handful of commits so it should be relatively quick to figure out. My only guess would be:

Subject: sched: make early bootup sched_clock() use safer

but i think this has been ruled out before ...

rc2-mm1 + that patch doesn't boot too.
stop at the same point ;)

#4 lore raw
Quoted 7 lines · KOSAKI Motohiro
Quoted 4 lines

hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:

http://people.redhat.com/mingo/sched-devel.git/README

How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?

yeah, please. If it's caused by sched-devel.git then you should see the hang there too.

Ingo

#5 lore raw

Hi Ingo,
CC'ed Steven Rostedt

I found the following patch cause regression by bisect.

2.6.25-rc2-mm1: doesn't boot
2.6.25-rc2-mm1 + revert following patch: works well

but I think it is very strange. runqueue_is_locked() seems simple and have not bug. ;)

What do you think this problem?

-------------------------------------------------------------------
commit 033394c2c097215ac556a446154af24fbf18b064

Author: Steven Rostedt <[email protected]>
Date:   Mon Feb 25 21:15:44 2008 +0100

printk: dont wake up klogd with the rq locked

It is not wise to place a printk where the runqueue lock is held.

Quoted 13 lines · KOSAKI Motohiro
Quoted 8 lines
Quoted 5 lines

hm, that's a bit weird - nothing really should have broken it. Could you try to do a specific bisection of sched-devel.git:

http://people.redhat.com/mingo/sched-devel.git/README

How do I know revision of git-sched.patch of 2.6.25-rc2-mm1? Should I do bisect from HEAD of sched-devel.git?

yeah, please. If it's caused by sched-devel.git then you should see the hang there too.

#6 lore raw
Quoted 5 lines · KOSAKI Motohiro

Hi Ingo,
CC'ed Steven Rostedt

I found the following patch cause regression by bisect.

Quoted 7 lines

2.6.25-rc2-mm1: doesn't boot
2.6.25-rc2-mm1 + revert following patch: works well

but I think it is very strange. runqueue_is_locked() seems simple and have not bug. ;)

What do you think this problem?

thanks for bisecting it down! Could ia64 have trouble accessing the percpu data structures of the scheduler?

does the patch below resolve the hang?

Ingo

------------------------->
Subject: sched: fix wake_up_klogd()
From: Ingo Molnar <[email protected]>
Date: Thu Feb 28 12:42:45 CET 2008

on some platforms if we printk too early it might not be safe to call into the scheduler data structures.

---
 kernel/printk.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Index: linux/kernel/printk.c =================================--- linux.orig/kernel/printk.c +++ linux/kernel/printk.c
@@ -948,7 +948,8 @@ int is_console_locked(void)

void wake_up_klogd(void)
{
- if (!oops_in_progress && waitqueue_active(&log_wait))
+ if (!oops_in_progress && waitqueue_active(&log_wait) &&
+ !runqueue_is_locked())
wake_up_interruptible(&log_wait);
}

@@ -1000,7 +1001,7 @@ void release_console_sem(void)
* If we try to wake up klogd while printing with the runqueue lock
* held, this will deadlock.
*/
- if (wake_klogd && !runqueue_is_locked())
+ if (wake_klogd)
wake_up_klogd();
}
EXPORT_SYMBOL(release_console_sem);

Tested-by
#7 lore raw

Hi Ingo,

Quoted 4 lines

thanks for bisecting it down! Could ia64 have trouble accessing the percpu data structures of the scheduler?

does the patch below resolve the hang?

Thanks!
that patch works well on my test environment.

BTW: Your work is ultimate fast. it's wonderful :)

#8 lore raw
In reply to Ingo Molnar

Review void release_console_sem(void) · line 1006

Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1 -1
@@ -1000,7 +1001,7 @@ void release_console_sem(void)
 	 * If we try to wake up klogd while printing with the runqueue lock
 	 * held, this will deadlock.
 	 */
-	if (wake_klogd && !runqueue_is_locked())
+	if (wake_klogd)
 		wake_up_klogd();
 }

I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.

#9 lore raw
Quoted 14 lines · Andrew Morton

Review void release_console_sem(void) · line 1006

Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1 -1
@@ -1000,7 +1001,7 @@ void release_console_sem(void)
 	 * If we try to wake up klogd while printing with the runqueue lock
 	 * held, this will deadlock.
 	 */
-	if (wake_klogd && !runqueue_is_locked())
+	if (wake_klogd)
 		wake_up_klogd();
 }

I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.

well, the problem was ia64, not a problem on x86 or other platforms. The problem here is ia64 not setting up percpu data structures soon enough. It has blown up in the past in other areas, and it will likely blow up in the future in other areas as well. It's just not robust to have init dependencies on such basic data structures like percpu areas like that.

Ingo

#10 lore raw
Quoted 18 lines · On Thu, 28 Feb 2008 20:12:14 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 14 lines · Andrew Morton

Review void release_console_sem(void) · line 1006

Quoted 8 lines · On Thu, 28 Feb 2008 12:50:41 +0100 Ingo Molnar <[email protected]> wrote:
(quoted diff) +1 -1
@@ -1000,7 +1001,7 @@ void release_console_sem(void)
 	 * If we try to wake up klogd while printing with the runqueue lock
 	 * held, this will deadlock.
 	 */
-	if (wake_klogd && !runqueue_is_locked())
+	if (wake_klogd)
 		wake_up_klogd();
 }

I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.

Quoted 1 line

well, the problem was ia64, not a problem on x86 or other platforms.

I am referring to the original change which made klogd wakeups unreliable.

#11 lore raw
Quoted 9 lines · Andrew Morton
Quoted 6 lines
Quoted 4 lines

I don't think we shoudl have added that hack in the first place. It solves a problem which about three developers hit four times in five years but it has made kernel logging less reliable for everyone.

well, the problem was ia64, not a problem on x86 or other platforms.

I am referring to the original change which made klogd wakeups unreliable.

oh, indeed - agreed - i missed the fact that is_locked check is sporadic and can cause other CPUs to prevent the wakeup of klogd. I've zapped the change.

Ingo