New panic (io-apic / timer?) going from 2.5.44 to 2.5.44-mm1
Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2) panic consistently on boot for a 16 way NUMA-Q.
Normally this box will boot with TSC on or off. If anyone has any pointers as to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that might have touched this area (I can't see anything), please poke me in the eye. Otherwise I'll just keep digging into it ....
Diff of bootlogs with TSC on below. As per usual for SMP panics, it's garbage ;-(
BIOS bug, IO-APIC#7 ID 13 is already used!...
... fixing up to 9. (tell your hw vendor)
...changing IO-APIC physical APIC ID to 9 ... ok.
-..TIMER: vector=0x31 pin1=2 pin2=0
-testing the IO APIC.......................
-
-
-
-
-
-
-
-
+ge<4>al prot, 5-19 fault: 0000
+1 <4>C 5-22 3
+ EIP: , 6-060:[<0, 7-008>], 7-8ot ta, 7-18, 7-19S, 7-200, 2
+1e<4>, 7-000, 7-2eb, 8-0321c60 ecx: c0320, 9-8 edx: 0, 9-180, 9-19:<4>, 9-20
, 9-21 c3a38000 ebp: 00000001 esp: c3a39f14
+ector=0x31 pin1=2 pin2=0
+ 0068
+Process (pid: 49194, threadinf<3>..38000 task=f01c07f6)mer notStack: ted to IO
APIC
+00...t00000011 set up timer (IRQ0) through t e 8259A ... 0320c24 . found n 0)
...a c02dc960 c02c4800 00000000 00000000 <4>c3d.
+64..00000046 set uf40 mer ual Wire IRQ0000000 00128868 c02b1e3c 0004200
0 c01078cc k<4>00nu0003 <4>c02b2128 ourcCall160.
+:number [<c011d745>13 tasklet_hi_action+0x85/0xe0
+O-APIC #14 re] ido_ssft24.
+xnumber of [-AP108f40>] ido_IRQ+0x100/0x1mber [<c01078c #5 registn_int 24. ++numb/0x20
+O-APIC #8d72>] trelease.
+nnumberem+0xIO-APIC #7 [<c0118ccd 24.prinumber o5f IO-APIC
+8Code: ters: 24.
+anue.er of IO-A0IC #9 r panic: Aiee, killingngnterrup APIC...............rrupt handl
+ - not syncing
+. .
+.... register #00: 0D000000
If I patch it so that I can turn TSC off, and do so, I now get:
..TIMER: vector=0x31 pin1=2 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC ...trying to set up timer (IRQ0) through the 8259A ... ..... (found pin 0) ... failed.
...trying to set up timer as Virtual Wire IRQ... failed. ...trying to set up timer as ExtINT IRQ... failed :(.
Kernel panic: IO-APIC + timer doesn't work! pester [email protected]
which never used to happen before ... a good boot does this
... fixing up to 9. (tell your hw vendor)
...changing IO-APIC physical APIC ID to 9 ... ok.
..TIMER: vector=0x31 pin1=2 pin2=0
testing the IO APIC.......................
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 699.0999 MHz.
..... host bus clock speed is 99.0999 MHz.
Quoted 9 lines · Martin J. Bligh
Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2) panic consistently on boot for a 16 way NUMA-Q.
Normally this box will boot with TSC on or off. If anyone has any pointers as to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that might have touched this area (I can't see anything), please poke me in the eye. Otherwise I'll just keep digging into it ....
Is possibly the code which defers the allocation of the per-cpu memory until the secondary processors are being brought online.
I've decided to toss that. It's causing some grief for architectures, and only buys us 10k * (NR_CPUS - nr-cpus) worth of memory anyway.
Well. It would be useful for NUMA to be able to place the per-cpu storage into node-local memory. But the code doesn't do that. It just uses kmalloc on the boot cpu, and we don't have an alloc_pages_for_another_cpu() API..
Quoted 10 lines
Quoted 8 lines
Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2) panic consistently on boot for a 16 way NUMA-Q.
Normally this box will boot with TSC on or off. If anyone has any pointers as to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that might have touched this area (I can't see anything), please poke me in the eye. Otherwise I'll just keep digging into it ....
Quoted 2 lines
Is possibly the code which defers the allocation of the per-cpu memory until the secondary processors are being brought online.
Aha ... ;-) thanks for the pointer. Will poke at that.
Quoted 2 lines
I've decided to toss that. It's causing some grief for architectures, and only buys us 10k * (
NR_CPUS- nr-cpus) worth of memory anyway.
Hmmm ... I guess people with SMP normally have lots of RAM anyway, and those few that care could just config NR_CPUS down. Shame though, automagical is much nicer. Maybe we can work out exactly why it's broken instead.
Quoted 4 lines
Well. It would be useful for NUMA to be able to place the per-cpu storage into node-local memory. But the code doesn't do that. It just uses kmalloc on the boot cpu, and we don't have an
alloc_pages_for_another_cpu()API..
Well at a page granularity level you have alloc_pages_node(cpu_to_node). I suppose technically that's not guaranteed, but on boot it'll never fall back. If you need it in permanent KVA, you can remap it into the vmalloc reserve area.
Why do you need to do it that late, should be able to do it once you've parsed mptables? alloc_bootmem / alloc_bootmem_node ?
M.
Quoted 11 lines · Martin J. Bligh
Quoted 11 lines
Quoted 9 lines
Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2) panic consistently on boot for a 16 way NUMA-Q.
Normally this box will boot with TSC on or off. If anyone has any pointers as to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that might have touched this area (I can't see anything), please poke me in the eye. Otherwise I'll just keep digging into it ....
Quoted 4 lines
Quoted 2 lines
Is possibly the code which defers the allocation of the per-cpu memory until the secondary processors are being brought online.
Aha ... ;-) thanks for the pointer. Will poke at that.
The kgdb code plays around at that level too. A patch -R of kgdb.patch would be interesting.
Quoted 13 lines
Quoted 2 lines
I've decided to toss that. It's causing some grief for architectures, and only buys us 10k * (
NR_CPUS- nr-cpus) worth of memory anyway.Hmmm ... I guess people with SMP normally have lots of RAM anyway, and those few that care could just config
NR_CPUSdown. Shame though, automagical is much nicer. Maybe we can work out exactly why it's broken instead.Quoted 4 lines
Well. It would be useful for NUMA to be able to place the per-cpu storage into node-local memory. But the code doesn't do that. It just uses kmalloc on the boot cpu, and we don't have an
alloc_pages_for_another_cpu()API..
Quoted 4 lines
Well at a page granularity level you have
alloc_pages_node(cpu_to_node). I suppose technically that's not guaranteed, but on boot it'll never fall back. If you need it in permanent KVA, you can remap it into the vmalloc reserve area.
That would suit.
Quoted 2 lines
Why do you need to do it that late, should be able to do it once you've parsed mptables?
alloc_bootmem/alloc_bootmem_node?
Lack of architecture hooks, maybe? That'd be one for Rusty.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Is possibly the code which defers the allocation of the per-cpu memory until the secondary processors are being brought online.
Aha ... ;-) thanks for the pointer. Will poke at that.
Humpf. Well I tried -mm3 (after backing out Rusty's patch), and it still has the problem ;-( Must be something else.
Quoted 2 lines
The kgdb code plays around at that level too. A patch -R of kgdb.patch would be interesting.
OK ... will try that then brute force and ignorance (binary chop search) I guess.
M.
Quoted 5 lines
Quoted 2 lines
The kgdb code plays around at that level too. A patch -R of kgdb.patch would be interesting.
OK ... will try that then brute force and ignorance (binary chop search) I guess.
Your psychic powers seem to be on form. Was kgdb. Worked in 41-mm1 for me on the same machine though.
M.