Skip to contentSkip to conversation

[2.6.27] early exception - lockdep related?

DISCUSSION Luca Tettamanti 2 participants 10 messages Sep 2, 2008 – Sep 4, 2008

lore
Share & tools
lore
#0 lore raw

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down. The strange thing is that it comes and goes with different kernel versions, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached), at least in one case the difference between a non-working kernel and a working one is CONFIG_DEBUG enabled in the latter.

The address printed is inside the function __lock_acqurie:

in __lock_acquire (/home/kronos/src/linux-2.6.git/kernel/lockdep.c:727). 722
723 /*

724              * We can walk the hash lockfree, because the hash only
725              * grows, and we are careful when adding entries to the end:
726              */

727 list_for_each_entry(class, hash_head, hash_entry) {
728 if (class->key == key) {
729 WARN_ON_ONCE(class->name != lock->name);
730 return class;
731 }

And the disassembly (faulting address is 0xffffffff80253b66)

0xffffffff80253b00 <__lock_acquire+299>:        shr    $0x34,%rax
0xffffffff80253b04 <__lock_acquire+303>:        shl    $0x4,%rax
0xffffffff80253b08 <__lock_acquire+307>:        lea    -0x7f57e9b0(%rax),%rdx
0xffffffff80253b0f <__lock_acquire+314>:        mov    -0x7f57e9b0(%rax),%r13
0xffffffff80253b16 <__lock_acquire+321>:        jmp    0xffffffff80253b66 <__lock_acquire+401>
0xffffffff80253b18 <__lock_acquire+323>:        cmp    %r8,0x20(%r13)
0xffffffff80253b1c <__lock_acquire+327>:        jne    0xffffffff80253b63 <__lock_acquire+398>
0xffffffff80253b1e <__lock_acquire+329>:        mov    -0x90(%rbp),%r10
0xffffffff80253b25 <__lock_acquire+336>:        mov    0x10(%r10),%r10
0xffffffff80253b29 <__lock_acquire+340>:        cmp    %r10,0x140(%r13)
0xffffffff80253b30 <__lock_acquire+347>:        je     0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b36 <__lock_acquire+353>:        cmpl   $0x0,0x51dc4f(%rip)
  # 0xffffffff8077178c <__warned.16970>
0xffffffff80253b3d <__lock_acquire+360>:        jne    0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b43 <__lock_acquire+366>:        mov    $0x2d9,%esi
0xffffffff80253b48 <__lock_acquire+371>:        mov    $0xffffffff805691c7,%rdi
0xffffffff80253b4f <__lock_acquire+378>:        callq  0xffffffff80232bae <warn_on_slowpath>
0xffffffff80253b54 <__lock_acquire+383>:        movl   $0x1,0x51dc2e(%rip)
  # 0xffffffff8077178c <__warned.16970>
0xffffffff80253b5e <__lock_acquire+393>:        jmpq   0xffffffff80253ea0 <__lock_acquire+1227>
0xffffffff80253b63 <__lock_acquire+398>:        mov    %rax,%r13 
0xffffffff80253b66 <__lock_acquire+401>:        mov    0x0(%r13),%rax <-- HERE
0xffffffff80253b6a <__lock_acquire+405>:        cmp    %rdx,%r13
0xffffffff80253b6d <__lock_acquire+408>:        prefetcht0 (%rax)
0xffffffff80253b70 <__lock_acquire+411>:        jne    0xffffffff80253b18 <__lock_acquire+323>
0xffffffff80253b72 <__lock_acquire+413>:        callq  0xffffffff802513ba <static_obj>
0xffffffff80253b77 <__lock_acquire+418>:        test   %eax,%eax
0xffffffff80253b79 <__lock_acquire+420>:        jne    0xffffffff80253ba8 <__lock_acquire+467>
0xffffffff80253b7b <__lock_acquire+422>:        callq  0xffffffff8031db50 <debug_locks_off>
0xffffffff80253b80 <__lock_acquire+427>:        mov    $0xffffffff80569212,%rdi
0xffffffff80253b87 <__lock_acquire+434>:        xor    %eax,%eax
0xffffffff80253b89 <__lock_acquire+436>:        callq  0xffffffff80487bd3 <printk>
0xffffffff80253b8e <__lock_acquire+441>:        mov    $0xffffffff8056923c,%rdi
0xffffffff80253b95 <__lock_acquire+448>:        xor    %eax,%eax
0xffffffff80253b97 <__lock_acquire+450>:        callq  0xffffffff80487bd3 <printk>
0xffffffff80253b9c <__lock_acquire+455>:        mov    $0xffffffff8056926c,%rdi
0xffffffff80253ba3 <__lock_acquire+462>:        jmpq   0xffffffff8025477c <__lock_acquire+3495>
0xffffffff80253ba8 <__lock_acquire+467>:        mov    -0x90(%rbp),%rdx
0xffffffff80253baf <__lock_acquire+474>:        mov    (%rdx),%rax
0xffffffff80253bb2 <__lock_acquire+477>:        pushfq
0xffffffff80253bb3 <__lock_acquire+478>:        popq   -0x68(%rbp)
0xffffffff80253bb6 <__lock_acquire+481>:        cli
0xffffffff80253bb7 <__lock_acquire+482>:        mov    $0x100,%ebx
0xffffffff80253bbc <__lock_acquire+487>:        lock xadd %bx,0x51dbcb(%rip)
    # 0xffffffff80771790 <lockdep_lock>
0xffffffff80253bc5 <__lock_acquire+496>:        cmp    %bh,%bl
0xffffffff80253bc7 <__lock_acquire+498>:        je     0xffffffff80253bd3 <__lock_acquire+510>
0xffffffff80253bc9 <__lock_acquire+500>:        pause
0xffffffff80253bcb <__lock_acquire+502>:        mov    0x51dbbf(%rip),%bl
 # 0xffffffff80771790 <lockdep_lock>
0xffffffff80253bd1 <__lock_acquire+508>:        jmp    0xffffffff80253bc5 <__lock_acquire+496>
0xffffffff80253bd3 <__lock_acquire+510>:        cmpl   $0x0,0x397676(%rip)
  # 0xffffffff805eb250 <debug_locks>
0xffffffff80253bda <__lock_acquire+517>:        jne    0xffffffff80253beb <__lock_acquire+534>
0xffffffff80253bdc <__lock_acquire+519>:        incb   0x51dbae(%rip)        # 0xffffffff80771790 <lockdep_lock>
0xffffffff80253be2 <__lock_acquire+525>:        pushq  -0x68(%rbp)
0xffffffff80253be5 <__lock_acquire+528>:        popfq

I actually tried to bisect it using an early working kernel and this first (read: the first pull from Linus' tree that gave me a broken kernel). The result is inconclusive, the commit pointed out was:

e5f363e358cf16e4ad13a6826e15088c5495efe9 is first bad commit commit e5f363e358cf16e4ad13a6826e15088c5495efe9

Author: Ingo Molnar <[email protected]>
Date:   Mon Aug 11 12:37:27 2008 +0200

lockdep: increase MAX_LOCKDEP_KEYS

certain configs produce:

[ 70.076229] BUG: MAX_LOCKDEP_KEYS too low!
[ 70.080230] turning off the locking correctness validator.

tune them up.

but reverting it didn't make any difference.

Is there any other action that I can take to debug this issue? Btw, do you think that kgdb can be usefull? I can give it a try in the weekend...

Luca

Signature · 1 line
Carpe diem, quam minimum credula postero. (Q. Horatius Flaccus)
#1 lore raw
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Does it print one of these nice stack traces?

Quoted 2 lines

The strange thing is that it comes and goes with different kernel versions

What kernel version did you generate the below with - and what config.

Quoted 2 lines

, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)

you seem to have forgotten that attachment.

Quoted 9 lines

, at least in one
case the difference between a non-working kernel and a working one is CONFIG_DEBUG enabled in the latter.

The address printed is inside the function __lock_acqurie:

in __lock_acquire (/home/kronos/src/linux-2.6.git/kernel/lockdep.c:727). 722
723 /*

Quoted 8 lines
724              * We can walk the hash lockfree, because the hash only
725              * grows, and we are careful when adding entries to the end:
726              */

727 list_for_each_entry(class, hash_head, hash_entry) {
728 if (class->key == key) {
729 WARN_ON_ONCE(class->name != lock->name);
730 return class;
731 }

Right - except this isn't in __lock_acquire, its from look_up_lock_class, which probably gets inlined.

Quoted 1 line

And the disassembly (faulting address is 0xffffffff80253b66)

<snip asm>

Quoted 16 lines

I actually tried to bisect it using an early working kernel and this first (read: the first pull from Linus' tree that gave me a broken kernel). The result is inconclusive, the commit pointed out was:

e5f363e358cf16e4ad13a6826e15088c5495efe9 is first bad commit commit e5f363e358cf16e4ad13a6826e15088c5495efe9

Author: Ingo Molnar <[email protected]>
Date:   Mon Aug 11 12:37:27 2008 +0200

lockdep: increase MAX_LOCKDEP_KEYS

certain configs produce:

[ 70.076229] BUG: MAX_LOCKDEP_KEYS too low!
[ 70.080230] turning off the locking correctness validator.

Quoted 8 lines

tune them up.

but reverting it didn't make any difference.

Is there any other action that I can take to debug this issue? Btw, do you think that kgdb can be usefull? I can give it a try in the weekend...

Might - I've never tried it..

#2 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Quoted 4 lines
Quoted 2 lines

The strange thing is that it comes and goes with different kernel versions

What kernel version did you generate the below with - and what config.

-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.

Quoted 4 lines
Quoted 2 lines

, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)

you seem to have forgotten that attachment.

Ops, will send as soon as I get home.

Luca

#3 lore raw
Quoted 11 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Until it arrives, you could try poking at netconsole..

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The strange thing is that it comes and goes with different kernel versions

What kernel version did you generate the below with - and what config.

-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.

OK, once you send you .config, I'll try and reproduce on one of my machine. Perhaps you can also provide the git-describe output of a known bad one.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)

you seem to have forgotten that attachment.

Ops, will send as soon as I get home.

Sure thing..

#4 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Until it arrives, you could try poking at netconsole..

Hum, I forgot to mention that it dies very early, just after:

Kernel is alive
Kernel is really alive
<exception>
<dead>

Is netconsole already up at that point?

Luca

#5 lore raw
Quoted 16 lines · Luca Tettamanti
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Until it arrives, you could try poking at netconsole..

Quoted 8 lines

Hum, I forgot to mention that it dies very early, just after:

Kernel is alive
Kernel is really alive
<exception>
<dead>

Is netconsole already up at that point?

Good question, I suppose not, that only happens after the device probing has found the eth card.

#6 lore raw
In reply to Luca Tettamanti
Quoted 16 lines · Luca Tettamanti
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Luca Tettamanti
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Until it arrives, you could try poking at netconsole..

Quoted 8 lines

Hum, I forgot to mention that it dies very early, just after:

Kernel is alive
Kernel is really alive
<exception>
<dead>

Is netconsole already up at that point?

Another thing you could try is booting this kernel in qemu and try attaching to its gdb stub.

I've had varying levels of success doing that, you need a very recent (might still be svn snapshot) of qemu to get x86_64 gdb to work iirc.

#7 lore raw
In reply to Luca Tettamanti
Quoted 20 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Quoted 4 lines
Quoted 2 lines

The strange thing is that it comes and goes with different kernel versions

What kernel version did you generate the below with - and what config.

-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)

you seem to have forgotten that attachment.

Ops, will send as soon as I get home.

Config is attached.

I tried netconsole, but got not output. No luck with qemu either :S

QEMU from SVN reports that:
qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000000a0000

KVM goes a bit further but dies due to an exception while the kernel is still executing in real mode.

Luca

#8 lore raw
Quoted 30 lines · Luca Tettamanti
Quoted 20 lines · Luca Tettamanti
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Luca Tettamanti

Hello,
I'm seeing an early exception (0e) - which seems related to lockdep - at boot with many 2.6.27 kernels and I'm having troubles to track it down.

Does it print one of these nice stack traces?

Of course not ;-) It dies before dumping the stack trace (or at least it doesn't make to the console - that machine has a serial port, but my notebook does not... I've ordered a usb-to-serial adapter).

Quoted 4 lines
Quoted 2 lines

The strange thing is that it comes and goes with different kernel versions

What kernel version did you generate the below with - and what config.

-rc2 is working, after that it's intermittent. The last kernel that I tried is -git from a couple of days ago.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

, but a "bad" kernel consistently fails across reboots. It also seems to be sensitive to the configuration (attached)

you seem to have forgotten that attachment.

Ops, will send as soon as I get home.

Config is attached.

Quoted 7 lines

I tried netconsole, but got not output. No luck with qemu either :S

QEMU from SVN reports that:
qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000000a0000

KVM goes a bit further but dies due to an exception while the kernel is still executing in real mode.

Sadly your config just boots, albeit not to userspace due to missing drivers.

root@twins:/mnt/build/linux-2.6# git-describe
v2.6.27-rc5-6-gbef69ea
root@twins:/mnt/build/linux-2.6# /mnt/md0/cross/bin/x86_64-linux-gcc --version
x86_64-linux-gcc (GCC) 4.3.1 20080510 (prerelease)
#9 lore raw
Quoted 2 lines · Peter Zijlstra

Sadly your config just boots, albeit not to userspace due to missing drivers.

Yes, I managed to boot it with qemu... I tried kgdb - without luck - kernel dies too early.
I also managed to get a stack trace :D

http://img151.imageshack.us/my.php?image=tracedm1.jpg

It seems that lockdep is an innocent bystander... the kernel died with panic() in __reserve_early, and then took another exception while printing the panic (I guess).
Will add further debug stuff to see wtf is going on.

Luca