Skip to contentSkip to conversation

kernel crash debugging

DISCUSSION Lee 1 participant 2 messages Nov 9, 2005

lore
Share & tools
lore

Lee

author
#0 lore raw

Hello,

I am running kernel version '2.6.13-gentoo-r3'

My hardware is as follows:
- motherboard: Tyan Tiger 230T
- processors: 2 Pentium II 1.13Ghz
- memory: 1.5GB

I have been having intermittent lockups for a while now.

At first, I thought it had something to do with vmware, but this is no occuring with a non-tainted kernel.

I recently read a thread on the mailing list about how having preempting enabled can hide bugs, so I have disabled all forms of preempting, but I still am unable to determine the cause.

I have also attached a second system to this system via a null modem cable and am capturing kernel output by redirecting its console to the serial port.

However, the kernel is not outputing any information when it locks up.

Can anyone advise on the following:
- determining why the kernel is locking up
- locating where in the code is causing the lock up
- alternative methods of debugging
- increasing the console log level

I have attached my kernel config file to this email.

Please advise,
Lee

Signature · 4 lines
Lee
[email protected]

 08:06:58 up 2 days, 13:17,  0 users,  load average: 0.24, 0.12, 0.06
author
#1 lore raw
Quoted 5 lines

Can anyone advise on the following:
- determining why the kernel is locking up
- locating where in the code is causing the lock up
- alternative methods of debugging
- increasing the console log level

After booting my kernel with the options 'debug loglevel=9' I was able to get proper OOPS reports from the kernel on the serial console, which I have attached to this email.

Please advise on how I can help debug / fix this problem.

Thanks
Lee

Signature · 4 lines
Lee
[email protected]

 08:45:44 up 2 days, 13:56,  1 user,  load average: 0.00, 0.00, 0.00
Oops: 0000 [#1]
SMP DEBUG_PAGEALLOC
Modules linked in:
CPU:    1
EIP:    0060:[<c011816b>]    Not tainted VLI
EFLAGS: 00010082   (2.6.13-gentoo-r3-20051108-1407)

EIP is at do_page_fault+0xcb/0x6bf
eax: f7f08000 ebx: 0000000b ecx: 0000000d edx: 00000000
esi: 0000000e edi: c04689a2 ebp: f7f08110 esp: f7f08040
ds: 007b es: 007b ss: 0068
Process khubd (pid: 16, threadinfo=f7f07000 task=f7f06ac0)

Stack: c05b904c 0000000d f7f08064 00000000 00000000 0000008c 00000000 00000000
       f7f08118 f7f08118 c04689a2 00000000 0000000e 0000000b 00000000 00000000
       00000000 00000000 00000000 00030001 00000000 00000000 00000000 00000000
Call Trace:
 =======================
Unable to handle kernel paging request at virtual address fffff030
 printing eip:
c0103e83
*pde = 0065e067
Oops: 0000 [#2]
SMP DEBUG_PAGEALLOC
Modules linked in:
CPU:    1
EIP:    0060:[<c0103e83>]    Not tainted VLI
EFLAGS: 00010083   (2.6.13-gentoo-r3-20051108-1407)

EIP is at show_trace+0x83/0xc0
eax: 0000001c ebx: f7f07e94 ecx: fffffd32 edx: 00000001
esi: f7f07e94 edi: fffff000 ebp: f7f07e94 esp: f7f07e80
ds: 007b es: 007b ss: 0068
Process khubd (pid: 16, threadinfo=f7f07000 task=f7f06ac0)

Stack: c0466ce5 f7f07e94 f7f080a0 00000018 00000000 f7f07eb0 c0103f3f 00000000
       f7f08040 f7f07000 f7f0800c 00000000 f7f07ee8 c01040f0 00000000 f7f08040
       00000010 f7f07000 f7f06ac0 00010082 c04cf322 00000068 00000001 f7f07000
Call Trace:
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0118434>] do_page_fault+0x394/0x6bf
 [<c0103b63>] error_code+0x4f/0x54
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0118434>] do_page_fault+0x394/0x6bf
Code: 89 5c 24 04 e8 9f d1 01 00 89 5c 24 04 c7 04 24 7a fb 48 c0 e8 1f aa 03 00 c7 04 24 23 eb 46 c0 e8 83 d1 01 00 8b 36 eb bf 89 f3 <8b> 77 30 85 f6 74 0e c7 04 24 e5 6c 46 c0 e8 6a d1 01 00 eb 92
 eip: c012b43b
------------[ cut here ]------------
kernel BUG at include/asm/spinlock.h:136!
invalid operand: 0000 [#3]
SMP DEBUG_PAGEALLOC
Modules linked in:
CPU:    1
EIP:    0060:[<c044f3dc>]    Not tainted VLI
EFLAGS: 00010096   (2.6.13-gentoo-r3-20051108-1407)

EIP is at _spin_lock+0x3c/0x50
eax: 0000000e ebx: f7f08ff0 ecx: c04d1cec edx: 00000086
esi: f7f06ac0 edi: f7f08ff0 ebp: f7f07c80 esp: f7f07c74
ds: 007b es: 007b ss: 0068
Process khubd (pid: 16, threadinfo=f7f07000 task=f7f06ac0)

Stack: c046955d c012b43b f7f09ea0 f7f07c9c c012b43b f7f07c9c c019b3d2 f7f06ac0
       00000020 f7f06f94 f7f07cc0 c01224d9 f7f06ac0 00000001 f2b24f68 f7f07cd0
       f7f07ce8 00000020 c1da8ac0 f7f07cfc c01234a7 f7f06ac0 00000011 c01440cc
Call Trace:
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0104455>] do_trap+0xb5/0xc0
 [<c010470c>] do_invalid_op+0xbc/0xd0
 [<c0103b63>] error_code+0x4f/0x54
 [<c012b43b>] __exit_signal+0x3b/0x180
 [<c01224d9>] release_task+0x69/0x160
 [<c01234a7>] exit_notify+0x3c7/0x8d0
 [<c0123baf>] do_exit+0x1ff/0x3f0
 [<c01043a0>] do_trap+0x0/0xc0
 [<c0118434>] do_page_fault+0x394/0x6bf
 [<c0103b63>] error_code+0x4f/0x54
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0118434>] do_page_fault+0x394/0x6bf
Code: de 75 15 f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb f2 8b 5d fc 89 ec 5d c3 c7 04 24 5d 95 46 c0 8b 45 04 89 44 24 04 e8 24 1c cd ff <0f> 0b 88 00 b5 8d 46 c0 eb ce 8d 76 00 8d bc 27 00 00 00 00 81
 <1>Fixing recursive fault but reboot is needed!
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
c0119cb6
*pde = 00000000
Oops: 0002 [#4]
SMP DEBUG_PAGEALLOC
Modules linked in:
CPU:    1
EIP:    0060:[<c0119cb6>]    Not tainted VLI
EFLAGS: 00010086   (2.6.13-gentoo-r3-20051108-1407)

EIP is at dequeue_task+0x16/0x50
eax: 00000000 ebx: f7f06ae0 ecx: f7f06ac0 edx: f7f06ac0
esi: 00000000 edi: f7f06ac0 ebp: f7f07a58 esp: f7f07a50
ds: 007b es: 007b ss: 0068
Process khubd (pid: 16, threadinfo=f7f07000 task=f7f06ac0)

Stack: f7f06ac0 c1c13400 f7f07a6c c0119ff1 f7f06ac0 00000000 c1c13400 f7f07adc
       c044d5d7 f7f06ac0 c1c13400 00000286 f7f07a7c f7f06ac0 00000001 00000000
       f7f07ab0 c012a4e9 f7f06ac0 00000046 c05b28cc 00000001 c1c13400 00000246
Call Trace:
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0118434>] do_page_fault+0x394/0x6bf
 [<c0103b63>] error_code+0x4f/0x54
 [<c0119ff1>] deactivate_task+0x21/0x30
 [<c044d5d7>] schedule+0x157/0xd90
 [<c0123d21>] do_exit+0x371/0x3f0
 [<c01043a0>] do_trap+0x0/0xc0
 [<c0104455>] do_trap+0xb5/0xc0
 [<c010470c>] do_invalid_op+0xbc/0xd0
 [<c0103b63>] error_code+0x4f/0x54
 [<c012b43b>] __exit_signal+0x3b/0x180
 [<c01224d9>] release_task+0x69/0x160
 [<c01234a7>] exit_notify+0x3c7/0x8d0
 [<c0123baf>] do_exit+0x1ff/0x3f0
 [<c01043a0>] do_trap+0x0/0xc0
 [<c0118434>] do_page_fault+0x394/0x6bf
 [<c0103b63>] error_code+0x4f/0x54
 [<c0103f3f>] show_stack+0x7f/0xa0
 [<c01040f0>] show_registers+0x160/0x1d0
 [<c0104320>] die+0x100/0x180
 [<c0118434>] do_page_fault+0x394/0x6bf
Code: 8b 7d fc 89 ec 5d c3 89 1c 24 e8 56 6a 03 00 eb e7 8d 74 26 00 55 89 e5 83 ec 08 89 74 24 04 8b 55 08 8b 75 0c 89 1c 24 8d 5a 20 <ff> 0e 8b 42 20 8b 4b 04 89 01 89 48 04 c7 43 04 00 02 20 00 8b