Skip to contentSkip to conversation

2.6.9-rc4-bk2 bug report

DISCUSSION Nigel Kukard 2 participants 4 messages Oct 15, 2004 – Oct 17, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

While booting with 2.6.9rc4-bk2 I seem to get the below OOPS, I copied the modules & System.map file over to another box of mine where i used serial-console to grab the oops.

Anyone know what I can try to debug the below problem?

The box is a 2.8Ghz P4 with HT enabled (smp).

Kind Regards
Nigel Kukard

ksymoops 2.4.9 on i686 2.6.8-0.23. Options used

     -V (default)
     -K (specified)
     -L (specified)
     -o /tmp/2.6.9-0.8 (specified)
     -m /tmp/System.map-2.6.9-0.8 (specified)

No modules in ksyms, skipping objects kernel BUG at kernel/timer.c:414!
invalid operand: 0000 [#1]
EIP: 0060:[<c01279e0>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010007 (2.6.9-0.8)
eax: c1810a60 ebx: f7097d6c ecx: 0000000f edx: f7097d6c
esi: c18112e4 edi: c1810a60 ebp: 0000000f esp: c064afb0
ds: 007b es: 007b ss: 0068

Stack: 00000000 c05f9f88 c1810a60 c064afc8 c0127f86 c064afc8 c064afc8 c064afc8
       c01247c0 00000001 c05f9f88 c0643bc0 00000000 c01244a4 0000000a 
c05fbf90
       00000046 c0641484 006c8007 c01097f4
Call Trace:
 [<c0127f86>] run_timer_softirq+0xe6/0x160
 [<c01247c0>] tasklet_action+0x50/0xc0
 [<c01244a4>] __do_softirq+0x64/0xe0
 [<c01097f4>] do_softirq+0x44/0x4b
 [<c01161ed>] smp_apic_timer_interrupt+0xfd/0x100
 [<c0106fda>] apic_timer_interrupt+0x1a/0x20
 [<c010405a>] default_idle+0x2a/0x40
 [<c01040d4>] cpu_idle+0x24/0x60
 [<c05fc86d>] start_kernel+0x18d/0x1c0
Code: ca 53 89 c7 89 cd 8b 1e eb 11 90 89 da 39 7b 1c 75 19 89 f8 8b 1b e8 80 fb ff ff 39 f3 75 ec 89 36 89 76 04 89 e8 5b 5e 5f 5d c3 <0f> 0b 9e 01 ff 23 4d c0 eb dd 8d b6 00 00 00 00 0f bf 05 7a dc
Quoted 8 lines
Quoted 8 lines

EIP; c01279e0 <cascade+30/40> <=====

eax; c1810a60 <pg0+116aa60/3f958400>
ebx; f7097d6c <pg0+369f1d6c/3f958400>
edx; f7097d6c <pg0+369f1d6c/3f958400>
esi; c18112e4 <pg0+116b2e4/3f958400>
edi; c1810a60 <pg0+116aa60/3f958400>
esp; c064afb0 <softirq_stack+fb0/8000>

Trace; c0127f86 <run_timer_softirq+e6/160>
Trace; c01247c0 <tasklet_action+50/c0>
Trace; c01244a4 <__do_softirq+64/e0>
Trace; c01097f4 <do_softirq+44/4b>
Trace; c01161ed <smp_apic_timer_interrupt+fd/100>
Trace; c0106fda <apic_timer_interrupt+1a/20>
Trace; c010405a <default_idle+2a/40>
Trace; c01040d4 <cpu_idle+24/60>
Trace; c05fc86d <start_kernel+18d/1c0>

This architecture has variable length instructions, decoding before eip is unreliable, take these instructions with a pinch of salt.

Code; c01279b5 <cascade+5/40>
00000000 <_EIP>:
Code; c01279b5 <cascade+5/40>
0: ca 53 89 lret $0x8953
Code; c01279b8 <cascade+8/40>
3: c7 89 cd 8b 1e eb 11 movl $0xda899011,0xeb1e8bcd(%ecx)
Code; c01279bf <cascade+f/40>
a: 90 89 da
Code; c01279c2 <cascade+12/40>
d: 39 7b 1c cmp %edi,0x1c(%ebx)
Code; c01279c5 <cascade+15/40>
10: 75 19 jne 2b <_EIP+0x2b>
Code; c01279c7 <cascade+17/40>
12: 89 f8 mov %edi,%eax
Code; c01279c9 <cascade+19/40>
14: 8b 1b mov (%ebx),%ebx
Code; c01279cb <cascade+1b/40>
16: e8 80 fb ff ff call fffffb9b <_EIP+0xfffffb9b>
Code; c01279d0 <cascade+20/40>
1b: 39 f3 cmp %esi,%ebx
Code; c01279d2 <cascade+22/40>
1d: 75 ec jne b <_EIP+0xb>
Code; c01279d4 <cascade+24/40>
1f: 89 36 mov %esi,(%esi)
Code; c01279d6 <cascade+26/40>
21: 89 76 04 mov %esi,0x4(%esi)
Code; c01279d9 <cascade+29/40>
24: 89 e8 mov %ebp,%eax
Code; c01279db <cascade+2b/40>
26: 5b pop %ebx
Code; c01279dc <cascade+2c/40>
27: 5e pop %esi
Code; c01279dd <cascade+2d/40>
28: 5f pop %edi
Code; c01279de <cascade+2e/40>
29: 5d pop %ebp
Code; c01279df <cascade+2f/40>
2a: c3 ret

This decode from eip onwards should be reliable

Code; c01279e0 <cascade+30/40>
00000000 <_EIP>:
Code; c01279e0 <cascade+30/40> <=====
0: 0f 0b ud2a <=====
Code; c01279e2 <cascade+32/40>
2: 9e sahf
Code; c01279e3 <cascade+33/40>
3: 01 ff add %edi,%edi
Code; c01279e5 <cascade+35/40>
5: 23 4d c0 and 0xffffffc0(%ebp),%ecx
Code; c01279e8 <cascade+38/40>
8: eb dd jmp ffffffe7 <_EIP+0xffffffe7>
Code; c01279ea <cascade+3a/40>
a: 8d b6 00 00 00 00 lea 0x0(%esi),%esi
Code; c01279f0 <second_overflow+0/260>
10: 0f .byte 0xf
Code; c01279f1 <second_overflow+1/260>
11: bf .byte 0xbf
Code; c01279f2 <second_overflow+2/260>
12: 05 .byte 0x5
Code; c01279f3 <second_overflow+3/260>
13: 7a dc jp fffffff1 <_EIP+0xfffffff1>

#1 lore raw
Quoted 6 lines · Nigel Kukard

While booting with 2.6.9rc4-bk2 I seem to get the below OOPS, I copied
the modules & System.map file over to another box of mine where i used serial-console to grab the oops.

Anyone know what I can try to debug the below problem?

Something appears to have wrecked the system-wide timer queue. Possibilties are that some kernel module was unloaded but forgot to remove a pending timer, or some data structure was freed while holding a pending timer.

You could try enabling CONFIG_DEBUG_SLAB and CONFIG_DEBUG_PAGEALLOC, but I doubt if those would shed much light.

I'd suggest that you strip your .config down to the bare minimum which is needed to boot and see if the crash goes away. If it does, then it's just a matter of reintroducing .config options until you find which one caused the crash. Code inspection should then lead us to the bug.

#3 lore raw
In reply to a message not in the archive
Quoted 14 lines · Nigel Kukard
Quoted 11 lines · Andrew Morton
Quoted 7 lines · Nigel Kukard

While booting with 2.6.9rc4-bk2 I seem to get the below OOPS, I copied the modules & System.map file over to another box of mine where i used serial-console to grab the oops.

Anyone know what I can try to debug the below problem?

Quoted 9 lines
Quoted 6 lines

I'd suggest that you strip your .config down to the bare minimum which is needed to boot and see if the crash goes away. If it does, then it's just a matter of reintroducing .config options until you find which one caused the crash. Code inspection should then lead us to the bug.

Could it be caused by loading a module?

Yes. Or by unloading a module. The oops will happen some time _after_ the buggy module has done its buggy thing. Maybe you could change your modprobe command to be:

echo running modprobe $*
modprobe.old $*
sleep 5
echo finished modprobe $*

or something like that.

#4 lore raw
Quoted 10 lines · Andrew Morton
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

While booting with 2.6.9rc4-bk2 I seem to get the below OOPS, I copied the modules & System.map file over to another box of mine where i used serial-console to grab the oops.

Could it be caused by loading a module?

Yes. Or by unloading a module. The oops will happen some time _after_ the buggy module has done its buggy thing. Maybe you could change your modprobe command to be:

Quoted 8 lines

echo running modprobe $*
modprobe.old $*
sleep 5
echo finished modprobe $*

or something like that.

Ok, it seems to be related to specific hardware. I tried loading the exact same modules on another box in the exact same order, no crash. The box is actually running nicely.

I'm busy compiling the kernel again, this time with a few debug options ticked.

As far as I saw 2 days ago, its when I load uhci-hcd that the problem manifests itself... this is the order the modules are loaded in...

snd, snd_als4000, crc32, sis900, psmouse, evdev, tsdev, agpgart, sis_agp, pci_hotplug, usbcore, ohci_hcd, ub, ehci_hcd, ieee1394, ohci1394, uhci_hcd

The OOPS is triggered from loading uhci_hcd, so it could be ohci1394 that creates the corruption. I've had problems with ohci1394 in 26.8.1 aswell, I cannot load it, unload it, and reload it... generates a D state for the modprobe.

Any tips on how i can track down the offending line in source?

Regards
Nigel Kukard