[2.6.27] early exception - lockdep related?
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down. The strange thing is that it comes and goes with different kernel versions, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached), at least in one case the difference between a non-working kernel and a working one is CONFIG_DEBUG enabled in the latter.
The address printed is inside the function __lock_acqurie:
in __lock_acquire (/home/kronos/src/linux-2.6.git/kernel/lockdep.c:727). 722
723 /*
724 * We can walk the hash lockfree, because the hash only 725 * grows, and we are careful when adding entries to the end: 726 */
727 list_for_each_entry(class, hash_head, hash_entry) {
728 if (class->key == key) {
729 WARN_ON_ONCE(class->name != lock->name);
730 return class;
731 }
And the disassembly (faulting address is 0xffffffff80253b66)
0xffffffff80253b00 <__lock_acquire+299>: shr $0x34,%rax
0xffffffff80253b04 <__lock_acquire+303>: shl $0x4,%rax
0xffffffff80253b08 <__lock_acquire+307>: lea -0x7f57e9b0(%rax),%rdx
0xffffffff80253b0f <__lock_acquire+314>: mov -0x7f57e9b0(%rax),%r13
0xffffffff80253b16 <__lock_acquire+321>: jmp 0xffffffff80253b66 <__lock_acquire+401>
0xffffffff80253b18 <__lock_acquire+323>: cmp %r8,0x20(%r13)
0xffffffff80253b1c <__lock_acquire+327>: jne 0xffffffff80253b63 <__lock_acquire+398>
0xffffffff80253b1e <__lock_acquire+329>: mov -0x90(%rbp),%r10
0xffffffff80253b25 <__lock_acquire+336>: mov 0x10(%r10),%r10
0xffffffff80253b29 <__lock_acquire+340>: cmp %r10,0x140(%r13)
0xffffffff80253b30 <__lock_acquire+347>: je 0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b36 <__lock_acquire+353>: cmpl $0x0,0x51dc4f(%rip)
# 0xffffffff8077178c <__warned.16970>
0xffffffff80253b3d <__lock_acquire+360>: jne 0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b43 <__lock_acquire+366>: mov $0x2d9,%esi
0xffffffff80253b48 <__lock_acquire+371>: mov $0xffffffff805691c7,%rdi
0xffffffff80253b4f <__lock_acquire+378>: callq 0xffffffff80232bae <warn_on_slowpath>
0xffffffff80253b54 <__lock_acquire+383>: movl $0x1,0x51dc2e(%rip)
# 0xffffffff8077178c <__warned.16970>
0xffffffff80253b5e <__lock_acquire+393>: jmpq 0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b63 <__lock_acquire+398>: mov %rax,%r13
0xffffffff80253b66 <__lock_acquire+401>: mov 0x0(%r13),%rax <-- HERE
0xffffffff80253b6a <__lock_acquire+405>: cmp %rdx,%r13
0xffffffff80253b6d <__lock_acquire+408>: prefetcht0 (%rax)
0xffffffff80253b70 <__lock_acquire+411>: jne 0xffffffff80253b18 <__lock_acquire+323>
0xffffffff80253b72 <__lock_acquire+413>: callq 0xffffffff802513ba <static_obj>
0xffffffff80253b77 <__lock_acquire+418>: test %eax,%eax
0xffffffff80253b79 <__lock_acquire+420>: jne 0xffffffff80253ba8 <__lock_acquire+467>
0xffffffff80253b7b <__lock_acquire+422>: callq 0xffffffff8031db50 <debug_locks_off>
0xffffffff80253b80 <__lock_acquire+427>: mov $0xffffffff80569212,%rdi
0xffffffff80253b87 <__lock_acquire+434>: xor %eax,%eax
0xffffffff80253b89 <__lock_acquire+436>: callq 0xffffffff80487bd3 <printk>
0xffffffff80253b8e <__lock_acquire+441>: mov $0xffffffff8056923c,%rdi
0xffffffff80253b95 <__lock_acquire+448>: xor %eax,%eax
0xffffffff80253b97 <__lock_acquire+450>: callq 0xffffffff80487bd3 <printk>
0xffffffff80253b9c <__lock_acquire+455>: mov $0xffffffff8056926c,%rdi
0xffffffff80253ba3 <__lock_acquire+462>: jmpq 0xffffffff8025477c <__lock_acquire+3495>
0xffffffff80253ba8 <__lock_acquire+467>: mov -0x90(%rbp),%rdx
0xffffffff80253baf <__lock_acquire+474>: mov (%rdx),%rax
0xffffffff80253bb2 <__lock_acquire+477>: pushfq
0xffffffff80253bb3 <__lock_acquire+478>: popq -0x68(%rbp)
0xffffffff80253bb6 <__lock_acquire+481>: cli
0xffffffff80253bb7 <__lock_acquire+482>: mov $0x100,%ebx
0xffffffff80253bbc <__lock_acquire+487>: lock xadd %bx,0x51dbcb(%rip)
# 0xffffffff80771790 <lockdep_lock>
0xffffffff80253bc5 <__lock_acquire+496>: cmp %bh,%bl
0xffffffff80253bc7 <__lock_acquire+498>: je 0xffffffff80253bd3 <__lock_acquire+510>
0xffffffff80253bc9 <__lock_acquire+500>: pause
0xffffffff80253bcb <__lock_acquire+502>: mov 0x51dbbf(%rip),%bl
# 0xffffffff80771790 <lockdep_lock>
0xffffffff80253bd1 <__lock_acquire+508>: jmp 0xffffffff80253bc5 <__lock_acquire+496>
0xffffffff80253bd3 <__lock_acquire+510>: cmpl $0x0,0x397676(%rip)
# 0xffffffff805eb250 <debug_locks>
0xffffffff80253bda <__lock_acquire+517>: jne 0xffffffff80253beb <__lock_acquire+534>
0xffffffff80253bdc <__lock_acquire+519>: incb 0x51dbae(%rip) # 0xffffffff80771790 <lockdep_lock>
0xffffffff80253be2 <__lock_acquire+525>: pushq -0x68(%rbp)
0xffffffff80253be5 <__lock_acquire+528>: popfqI actually tried to bisect it using an early working kernel and this first (read: the first pull from Linus' tree that gave me a broken kernel). The result is inconclusive, the commit pointed out was:
e5f363e358cf16e4ad13a6826e15088c5495efe9 is first bad commit commit e5f363e358cf16e4ad13a6826e15088c5495efe9
Author: Ingo Molnar <[email protected]> Date: Mon Aug 11 12:37:27 2008 +0200
lockdep: increase MAX_LOCKDEP_KEYS
certain configs produce:
[ 70.076229] BUG: MAX_LOCKDEP_KEYS too low!
[ 70.080230] turning off the locking correctness validator.
tune them up.
but reverting it didn't make any difference.
Is there any other action that I can take to debug this issue? Btw, do you think that kgdb can be usefull? I can give it a try in the weekend...
Luca
Signature · 1 line
Carpe diem, quam minimum credula postero. (Q. Horatius Flaccus)
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.
Does it print one of these nice stack traces?
Quoted 2 lines
The strange thing is that it comes and goes with different kernel versions
What kernel version did you generate the below with - and what config.
Quoted 2 lines
, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)
you seem to have forgotten that attachment.
Quoted 9 lines
, at least in one
case the difference between a non-working kernel and a working one isCONFIG_DEBUGenabled in the latter.The address printed is inside the function
__lock_acqurie:in
__lock_acquire(/home/kronos/src/linux-2.6.git/kernel/lockdep.c:727). 722
723 /*
Quoted 8 lines
724 * We can walk the hash lockfree, because the hash only 725 * grows, and we are careful when adding entries to the end: 726 */727
list_for_each_entry(class, hash_head, hash_entry){
728 if (class->key== key) {
729WARN_ON_ONCE(class->name != lock->name);
730 return class;
731 }
Right - except this isn't in __lock_acquire, its from look_up_lock_class, which probably gets inlined.
Quoted 1 line
And the disassembly (faulting address is
0xffffffff80253b66)
<snip asm>
Quoted 16 lines
I actually tried to bisect it using an early working kernel and this first (read: the first pull from Linus' tree that gave me a broken kernel). The result is inconclusive, the commit pointed out was:
e5f363e358cf16e4ad13a6826e15088c5495efe9is first bad commit commite5f363e358cf16e4ad13a6826e15088c5495efe9Author: Ingo Molnar <[email protected]> Date: Mon Aug 11 12:37:27 2008 +0200lockdep: increase
MAX_LOCKDEP_KEYScertain configs produce:
[ 70.076229] BUG:
MAX_LOCKDEP_KEYStoo low!
[ 70.080230] turning off the locking correctness validator.
Quoted 8 lines
tune them up.
Signed-off-by Ingo Molnar <[email protected]>but reverting it didn't make any difference.
Is there any other action that I can take to debug this issue? Btw, do you think that kgdb can be usefull? I can give it a try in the weekend...
Might - I've never tried it..
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Quoted 4 lines
Quoted 2 lines
The strange thing is that it comes and goes with different kernel versions
What kernel version did you generate the below with - and what config.
-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.
Quoted 4 lines
Quoted 2 lines
, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)
you seem to have forgotten that attachment.
Ops, will send as soon as I get home.
Luca
Quoted 11 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Until it arrives, you could try poking at netconsole..
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The strange thing is that it comes and goes with different kernel versions
What kernel version did you generate the below with - and what config.
-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.
OK, once you send you .config, I'll try and reproduce on one of my machine. Perhaps you can also provide the git-describe output of a known bad one.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)
you seem to have forgotten that attachment.
Ops, will send as soon as I get home.
Sure thing..
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Until it arrives, you could try poking at netconsole..
Hum, I forgot to mention that it dies very early, just after:
Kernel is alive
Kernel is really alive
<exception>
<dead>
Is netconsole already up at that point?
Luca
Quoted 16 lines · Luca Tettamanti
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Until it arrives, you could try poking at netconsole..
Quoted 8 lines
Hum, I forgot to mention that it dies very early, just after:
Kernel is alive
Kernel is really alive
<exception>
<dead>Is netconsole already up at that point?
Good question, I suppose not, that only happens after the device probing has found the eth card.
Quoted 16 lines · Luca Tettamanti
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Until it arrives, you could try poking at netconsole..
Quoted 8 lines
Hum, I forgot to mention that it dies very early, just after:
Kernel is alive
Kernel is really alive
<exception>
<dead>Is netconsole already up at that point?
Another thing you could try is booting this kernel in qemu and try attaching to its gdb stub.
I've had varying levels of success doing that, you need a very recent (might still be svn snapshot) of qemu to get x86_64 gdb to work iirc.
Quoted 20 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Quoted 4 lines
Quoted 2 lines
The strange thing is that it comes and goes with different kernel versions
What kernel version did you generate the below with - and what config.
-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)
you seem to have forgotten that attachment.
Ops, will send as soon as I get home.
Config is attached.
I tried netconsole, but got not output. No luck with qemu either :S
QEMU from SVN reports that:
qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000000a0000
KVM goes a bit further but dies due to an exception while the kernel is still executing in real mode.
Luca
Quoted 30 lines · Luca Tettamanti
Quoted 20 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti
Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.Does it print one of these nice stack traces?
Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).
Quoted 4 lines
Quoted 2 lines
The strange thing is that it comes and goes with different kernel versions
What kernel version did you generate the below with - and what config.
-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)
you seem to have forgotten that attachment.
Ops, will send as soon as I get home.
Config is attached.
Quoted 7 lines
I tried netconsole, but got not output. No luck with qemu either :S
QEMU from SVN reports that:
qemu: fatal: Trying to execute code outside RAM or ROM at0x00000000000a0000KVM goes a bit further but dies due to an exception while the kernel is still executing in real mode.
Sadly your config just boots, albeit not to userspace due to missing drivers.
root@twins:/mnt/build/linux-2.6# git-describe v2.6.27-rc5-6-gbef69ea
root@twins:/mnt/build/linux-2.6# /mnt/md0/cross/bin/x86_64-linux-gcc --version x86_64-linux-gcc (GCC) 4.3.1 20080510 (prerelease)
Quoted 2 lines · Peter Zijlstra
Sadly your config just boots, albeit not to userspace due to missing drivers.
Yes, I managed to boot it with qemu... I tried kgdb - without luck - kernel dies too early.
I also managed to get a stack trace :D
http://img151.imageshack.us/my.php?image=tracedm1.jpg
It seems that lockdep is an innocent bystander... the kernel died with panic() in __reserve_early, and then took another exception while printing the panic (I guess).
Will add further debug stuff to see wtf is going on.
Luca