Total machine lockup w/ current kernels while installing from CD
Hi,
I've built a CD that installs a customized system; basically what it does is boot from CD (iso9660) and run
mke2fs -m0 -j -O dir_index,filetype,has_journal,sparse_super /dev/hda1 mount -text3 /dev/hda1 /mnt/dest
rpm -r /mnt/dest -ivh /RPMS/*
This worked perfectly up until some recent kernel updates - with current kernels (both Linus and -mm branch), the system locks up totally at a random point during rpm installation (everything goes down, including the NumLock LED etc).
I'm currently building some old kernels to see when this problem was introduced and sort out what patch did it.
With 2.6.16-rc6, the problem occurs, but unlike later revisions it gives an error message before freezing:
BUG: soft lockup detected on CPU#0!
Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is at journal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX: c2d34788 ECX: 00000001 EDX: c785e000
ESI: b3ff8d04 EDI: 000000f0 EBP: b683b840 DS: 007b ES: 007b
CR0: 8005003b CR2: 0841f7fc CR3: 17217000 CR4: 000006d0
[<b02bd52e>] schedule+0x2ee/0x5b6 [<b01a6a88>] kjournald+0x201/0x213 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49 [<b01a6937>] kjournald+0xb0/0x213 [<b01a5ffa>] commit_timeout+0x0/0x9 [<b012a789>] autoremove_wake_function+0x0/0x4b [<b01a6887>] kjournald+0x0/0x213 [<b0101005>] kernel_thread_helper+0x5/0xb
I'll keep trying to isolate which patch introduced this; at the moment the last known working kernel is 2.6.15 (and 2.6.15-mm4), the first known broken one is 2.6.16-rc6; the ones in between are still waiting for the compile box.
Quoted 27 lines · Bernhard Rosenkraenzer
Hi,
I've built a CD that installs a customized system; basically what it does is boot from CD (iso9660) and runmke2fs -m0 -j -O dir_index,filetype,has_journal,sparse_super
/dev/hda1mount -text3/dev/hda1/mnt/dest
rpm -r/mnt/dest-ivh /RPMS/*This worked perfectly up until some recent kernel updates - with current kernels (both Linus and -mm branch), the system locks up totally at a random point during rpm installation (everything goes down, including the NumLock LED etc).
I'm currently building some old kernels to see when this problem was introduced and sort out what patch did it.
With 2.6.16-rc6, the problem occurs, but unlike later revisions it gives an error message before freezing:BUG: soft lockup detected on CPU#0!Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is atjournal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX:c2d34788ECX: 00000001 EDX:c785e000
ESI:b3ff8d04EDI:000000f0EBP:b683b840DS: 007b ES: 007b
CR0:8005003bCR2:0841f7fcCR3: 17217000 CR4:000006d0
Quoted 8 lines
[<b02bd52e>] schedule+0x2ee/0x5b6 [<b01a6a88>] kjournald+0x201/0x213 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49 [<b01a6937>] kjournald+0xb0/0x213 [<b01a5ffa>] commit_timeout+0x0/0x9 [<b012a789>] autoremove_wake_function+0x0/0x4b [<b01a6887>] kjournald+0x0/0x213 [<b0101005>] kernel_thread_helper+0x5/0xb
I'm finally through compiling all kernels between the last good and first bad one (slow hardware sucks ;) ) -- the problem was introduced in the patch from 2.6.16-rc5 to 2.6.16-rc6, and is apparently not present in any -mm releases before 2.6.16-rc6.
I'll try to isolate which change between 2.6.16-rc5 and -rc6 causes it... But it would be nice if someone who knows a bit more about the code involved could look at it.
l
Quoted 12 lines · Bernhard Rosenkraenzer
Hi,
I've built a CD that installs a customized system
[... crash at a random point ...]BUG: soft lockup detected on CPU#0!Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is atjournal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX:c2d34788ECX: 00000001 EDX:c785e000
ESI:b3ff8d04EDI:000000f0EBP:b683b840DS: 007b ES: 007b
CR0:8005003bCR2:0841f7fcCR3: 17217000 CR4:000006d0
Quoted 8 lines
[<b02bd52e>] schedule+0x2ee/0x5b6 [<b01a6a88>] kjournald+0x201/0x213 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49 [<b01a6937>] kjournald+0xb0/0x213 [<b01a5ffa>] commit_timeout+0x0/0x9 [<b012a789>] autoremove_wake_function+0x0/0x4b [<b01a6887>] kjournald+0x0/0x213 [<b0101005>] kernel_thread_helper+0x5/0xb
After backing out lots of changes, I've figured out the problem is caused by this bit of 2.6.16-rc6:
kernel/sched.c +2 -0
@@ -4028,6 +4021,8 @@*/ if (unlikely (preempt_count ()))return ; +if (unlikely (system_state !=SYSTEM_RUNNING )) +return ; do { add_preempt_count(PREEMPT_ACTIVE); schedule();
The problem is that (to save a couple of bits of space), my simple installer was running inside an initrd -- and system_state isn't set to SYSTEM_RUNNING before linuxrc is executed --> scheduler breakage causes the oops.
Since initrds are deprecated anyway, I've used this opportunity to update the installer to use initramfs instead, and as expected, the problem went away -- nevertheless [unless initrds are being removed in 2.6.18 anyway] we should probably fix this or at least document it somewhe.
Quoted 30 lines · Bernhard Rosenkraenzer
Quoted 12 lines · Bernhard Rosenkraenzer
Hi,
I've built a CD that installs a customized system
[... crash at a random point ...]BUG: soft lockup detected on CPU#0!Pid: 421, comm: kjournald
EIP: 0060:[<b01a2f52>] CPU: 0
EIP is atjournal_commit_transaction+0x92e/0xfcc
EFLAGS: 00000297 Not tainted (2.6.16-rc6 #1)
EAX: 00000001 EBX:c2d34788ECX: 00000001 EDX:c785e000
ESI:b3ff8d04EDI:000000f0EBP:b683b840DS: 007b ES: 007b
CR0:8005003bCR2:0841f7fcCR3: 17217000 CR4:000006d0Quoted 8 lines
[<b02bd52e>] schedule+0x2ee/0x5b6 [<b01a6a88>] kjournald+0x201/0x213 [<b0111089>] smp_apic_timer_interrupt+0x32/0x49 [<b01a6937>] kjournald+0xb0/0x213 [<b01a5ffa>] commit_timeout+0x0/0x9 [<b012a789>] autoremove_wake_function+0x0/0x4b [<b01a6887>] kjournald+0x0/0x213 [<b0101005>] kernel_thread_helper+0x5/0xbAfter backing out lots of changes, I've figured out the problem is caused by this bit of 2.6.16-rc6:
diff -urN
linux-2.6.16-rc5/kernel/sched.clinux-2.6.16-rc6/kernel/sched.c
kernel/sched.c line 4028
Quoted 8 lines
kernel/sched.c +2
-0@@ -4028,6 +4021,8 @@*/ if (unlikely (preempt_count ()))return ; +if (unlikely (system_state !=SYSTEM_RUNNING )) +return ; do { add_preempt_count(PREEMPT_ACTIVE); schedule();
(That's cond_resched())
Quoted 4 lines
The problem is that (to save a couple of bits of space), my simple installer was running inside an initrd -- and
system_stateisn't set toSYSTEM_RUNNINGbefore linuxrc is executed --> scheduler breakage causes the oops.
ah-hah.
It's odd that we'll run initrds in a !SYSTEM_RUNNING state.
It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?
If so, I'd assume that the ext3 filesystem was mounted on a very slow device - perhaps an IDE disk in PIO mode?
Perhaps we should poke the softlockup detector if someone called cond_resched() when in a reschedulable state.
Quoted 1 line · Andrew Morton
It's odd that we'll run initrds in a !SYSTEM_RUNNING state.
True, especially because we run initramfs in SYSTEM_RUNNING state.
Quoted 2 lines
It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?
No, it was a lockup and the system just hung at the point forever, so the lockup detection was right.
Quoted 2 lines
If so, I'd assume that the ext3 filesystem was mounted on a very slow device - perhaps an IDE disk in PIO mode?
That too - happened with a pretty stupid 5-liner installation script that just formats the disk and installs a set of customized rpms from CD.
The hw we installed this on is Asus Pundit-R boxes, which means, basically, weird IDE setup (no secondary IDE --> harddisk on hda, CD drive on hdb) on an ATI IGP chipset, not exactly the fastest out there.
Quoted 11 lines · Bernhard Rosenkraenzer
Quoted 1 line · Andrew Morton
It's odd that we'll run initrds in a !SYSTEM_RUNNING state.
True, especially because we run initramfs in
SYSTEM_RUNNINGstate.Quoted 2 lines
It's not an oops - it's sort-of a warning. Did the system actually continue to run and boot up OK?
No, it was a lockup and the system just hung at the point forever, so the lockup detection was right.
This is odd. Your machine hangs if cond_resched() is a no-op. This should not happen.
What does `grep PREEMPT .config' say?