Skip to contentSkip to conversation

Total machine lockup w/ current kernels while installing from CD

DISCUSSION Bernhard Rosenkraenzer 2 participants 6 messages May 11, 2006 – May 15, 2006

lore
Share & tools
lore
#0 lore raw

Hi,
I've built a CD that installs a customized system; basically what it does is boot from CD (iso9660) and run

mke2fs -m0 -j -O dir_index,filetype,has_journal,sparse_super /dev/hda1 mount -text3 /dev/hda1 /mnt/dest
rpm -r /mnt/dest -ivh /RPMS/*

This worked perfectly up until some recent kernel updates - with current kernels (both Linus and -mm branch), the system locks up totally at a random point during rpm installation (everything goes down, including the NumLock LED etc).

I'm currently building some old kernels to see when this problem was introduced and sort out what patch did it.
With 2.6.16-rc6, the problem occurs, but unlike later revisions it gives an error message before freezing:

BUG: soft lockup detected on CPU#0!

Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is at journal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX: c2d34788 ECX: 00000001 EDX: c785e000
ESI: b3ff8d04 EDI: 000000f0 EBP: b683b840 DS: 007b ES: 007b
CR0: 8005003b CR2: 0841f7fc CR3: 17217000 CR4: 000006d0

 [<b02bd52e>] schedule+0x2ee/0x5b6
 [<b01a6a88>] kjournald+0x201/0x213
 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49
 [<b01a6937>] kjournald+0xb0/0x213
 [<b01a5ffa>] commit_timeout+0x0/0x9
 [<b012a789>] autoremove_wake_function+0x0/0x4b
 [<b01a6887>] kjournald+0x0/0x213
 [<b0101005>] kernel_thread_helper+0x5/0xb

I'll keep trying to isolate which patch introduced this; at the moment the last known working kernel is 2.6.15 (and 2.6.15-mm4), the first known broken one is 2.6.16-rc6; the ones in between are still waiting for the compile box.

#1 lore raw
Quoted 27 lines · Bernhard Rosenkraenzer

Hi,
I've built a CD that installs a customized system; basically what it does is boot from CD (iso9660) and run

mke2fs -m0 -j -O dir_index,filetype,has_journal,sparse_super /dev/hda1 mount -text3 /dev/hda1 /mnt/dest
rpm -r /mnt/dest -ivh /RPMS/*

This worked perfectly up until some recent kernel updates - with current kernels (both Linus and -mm branch), the system locks up totally at a random point during rpm installation (everything goes down, including the NumLock LED etc).

I'm currently building some old kernels to see when this problem was introduced and sort out what patch did it.
With 2.6.16-rc6, the problem occurs, but unlike later revisions it gives an error message before freezing:

BUG: soft lockup detected on CPU#0!

Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is at journal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX: c2d34788 ECX: 00000001 EDX: c785e000
ESI: b3ff8d04 EDI: 000000f0 EBP: b683b840 DS: 007b ES: 007b
CR0: 8005003b CR2: 0841f7fc CR3: 17217000 CR4: 000006d0

Quoted 8 lines
 [<b02bd52e>] schedule+0x2ee/0x5b6
 [<b01a6a88>] kjournald+0x201/0x213
 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49
 [<b01a6937>] kjournald+0xb0/0x213
 [<b01a5ffa>] commit_timeout+0x0/0x9
 [<b012a789>] autoremove_wake_function+0x0/0x4b
 [<b01a6887>] kjournald+0x0/0x213
 [<b0101005>] kernel_thread_helper+0x5/0xb

I'm finally through compiling all kernels between the last good and first bad one (slow hardware sucks ;) ) -- the problem was introduced in the patch from 2.6.16-rc5 to 2.6.16-rc6, and is apparently not present in any -mm releases before 2.6.16-rc6.

I'll try to isolate which change between 2.6.16-rc5 and -rc6 causes it... But it would be nice if someone who knows a bit more about the code involved could look at it.
l

#2 lore raw
In reply to Bernhard Rosenkraenzer
Quoted 12 lines · Bernhard Rosenkraenzer

Hi,
I've built a CD that installs a customized system
[... crash at a random point ...]

BUG: soft lockup detected on CPU#0!

Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is at journal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX: c2d34788 ECX: 00000001 EDX: c785e000
ESI: b3ff8d04 EDI: 000000f0 EBP: b683b840 DS: 007b ES: 007b
CR0: 8005003b CR2: 0841f7fc CR3: 17217000 CR4: 000006d0

Quoted 8 lines
 [<b02bd52e>] schedule+0x2ee/0x5b6
 [<b01a6a88>] kjournald+0x201/0x213
 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49
 [<b01a6937>] kjournald+0xb0/0x213
 [<b01a5ffa>] commit_timeout+0x0/0x9
 [<b012a789>] autoremove_wake_function+0x0/0x4b
 [<b01a6887>] kjournald+0x0/0x213
 [<b0101005>] kernel_thread_helper+0x5/0xb

After backing out lots of changes, I've figured out the problem is caused by this bit of 2.6.16-rc6:

kernel/sched.c +2 -0
diff -urN linux-2.6.16-rc5/kernel/sched.c linux-2.6.16-rc6/kernel/sched.c
@@ -4028,6 +4021,8 @@
 	 */
 	if (unlikely(preempt_count()))
 		return;
+	if (unlikely(system_state != SYSTEM_RUNNING))
+		return;
 	do {
 		add_preempt_count(PREEMPT_ACTIVE);
 		schedule();

The problem is that (to save a couple of bits of space), my simple installer was running inside an initrd -- and system_state isn't set to SYSTEM_RUNNING before linuxrc is executed --> scheduler breakage causes the oops.

Since initrds are deprecated anyway, I've used this opportunity to update the installer to use initramfs instead, and as expected, the problem went away -- nevertheless [unless initrds are being removed in 2.6.18 anyway] we should probably fix this or at least document it somewhe.

#3 lore raw
Quoted 30 lines · Bernhard Rosenkraenzer
Quoted 12 lines · Bernhard Rosenkraenzer

Hi,
I've built a CD that installs a customized system
[... crash at a random point ...]

BUG: soft lockup detected on CPU#0!

Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is at journal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX: c2d34788 ECX: 00000001 EDX: c785e000
ESI: b3ff8d04 EDI: 000000f0 EBP: b683b840 DS: 007b ES: 007b
CR0: 8005003b CR2: 0841f7fc CR3: 17217000 CR4: 000006d0

Quoted 8 lines
 [<b02bd52e>] schedule+0x2ee/0x5b6
 [<b01a6a88>] kjournald+0x201/0x213
 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49
 [<b01a6937>] kjournald+0xb0/0x213
 [<b01a5ffa>] commit_timeout+0x0/0x9
 [<b012a789>] autoremove_wake_function+0x0/0x4b
 [<b01a6887>] kjournald+0x0/0x213
 [<b0101005>] kernel_thread_helper+0x5/0xb

After backing out lots of changes, I've figured out the problem is caused by this bit of 2.6.16-rc6:

diff -urN linux-2.6.16-rc5/kernel/sched.c linux-2.6.16-rc6/kernel/sched.c

kernel/sched.c line 4028

Quoted 8 lines
kernel/sched.c +2 -0
@@ -4028,6 +4021,8 @@
 	 */
 	if (unlikely(preempt_count()))
 		return;
+	if (unlikely(system_state != SYSTEM_RUNNING))
+		return;
 	do {
 		add_preempt_count(PREEMPT_ACTIVE);
 		schedule();

(That's cond_resched())

Quoted 4 lines

The problem is that (to save a couple of bits of space), my simple installer was running inside an initrd -- and system_state isn't set to SYSTEM_RUNNING before linuxrc is executed --> scheduler breakage causes the oops.

ah-hah.

It's odd that we'll run initrds in a !SYSTEM_RUNNING state.

It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?

If so, I'd assume that the ext3 filesystem was mounted on a very slow device - perhaps an IDE disk in PIO mode?

Perhaps we should poke the softlockup detector if someone called cond_resched() when in a reschedulable state.

#4 lore raw
Quoted 1 line · Andrew Morton

It's odd that we'll run initrds in a !SYSTEM_RUNNING state.

True, especially because we run initramfs in SYSTEM_RUNNING state.

Quoted 2 lines

It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?

No, it was a lockup and the system just hung at the point forever, so the lockup detection was right.

Quoted 2 lines

If so, I'd assume that the ext3 filesystem was mounted on a very slow device - perhaps an IDE disk in PIO mode?

That too - happened with a pretty stupid 5-liner installation script that just formats the disk and installs a set of customized rpms from CD.

The hw we installed this on is Asus Pundit-R boxes, which means, basically, weird IDE setup (no secondary IDE --> harddisk on hda, CD drive on hdb) on an ATI IGP chipset, not exactly the fastest out there.

#5 lore raw
Quoted 11 lines · Bernhard Rosenkraenzer
Quoted 1 line · Andrew Morton

It's odd that we'll run initrds in a !SYSTEM_RUNNING state.

True, especially because we run initramfs in SYSTEM_RUNNING state.

Quoted 2 lines

It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?

No, it was a lockup and the system just hung at the point forever, so the lockup detection was right.

This is odd. Your machine hangs if cond_resched() is a no-op. This should not happen.

What does `grep PREEMPT .config' say?