Kernel panic - not syncing: Attempted to kill the idle task!
hi,
kernel panic on mainline(commit: 48d212a2eecaca) with a large system, which has 120Gb RAM & 8 numa nodes:
... [cut here] ...
[ 3.404017] Call Trace: [ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0 [ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50 [ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10 [ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80 [ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0 [ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0 [ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00 [ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70 [ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140 [ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264 [ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 00 00 48 8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0 48 c1 e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90 0f [ 3.420335] RIP [<ffffffff810af93b>] update_sd_lb_stats+0x27b/0x620 [ 3.421664] RSP <ffff88041922fb48> [ 3.422473] ---[ end trace 04b848dd1c06d585 ]--- [ 3.423472] Kernel panic - not syncing: Attempted to kill the idle task!
whole console log is here: http://www.sanweiying.org/download/debug_kernel/kernel_panic_48d212a2eecaca
and config file is http://www.sanweiying.org/download/debug_kernel/config_48d212a2eecaca
and I have tested different branches, more details is available in https://lkml.org/lkml/2012/6/6/704
here I do a simple summary:
a), mainline,
v3.4: no such panic
b), mainline, v3.5-rc1(commit: 48d212a2eecaca), yes
c), Andrea's tree, autonuma15 yes
d), tip/master(commit:b2f5ce55c4e68370) no such panic
let me know if you need any info.
Thanks,
Zhouping
Quoted 23 lines · ZhouPing Liu
kernel panic on mainline(commit:
48d212a2eecaca) with a large system, which has 120Gb RAM & 8 numa nodes:... [cut here] ...
[ 3.404017] Call Trace: [ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0 [ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50 [ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10 [ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80 [ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0 [ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0 [ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00 [ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70 [ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140 [ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264 [ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 00 00 48 8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0 48 c1 e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90 0f [ 3.420335] RIP [<ffffffff810af93b>] update_sd_lb_stats+0x27b/0x620 [ 3.421664] RSP <ffff88041922fb48> [ 3.422473] ---[ end trace 04b848dd1c06d585 ]--- [ 3.423472] Kernel panic - not syncing: Attempted to kill the idle task!
Quoted 6 lines
here I do a simple summary:
b), mainline, v3.5-rc1(commit:
48d212a2eecaca), yes
d), tip/master(commit:b2f5ce55c4e68370) no such paniclet me know if you need any info.
There's a number of patches in tip/sched/urgent that I think fix this (hence your D) and these should make their way to Linus shortly.
That said, can you provide me your node distance table so I can verify locally?
cat /sys/devices/system/node/node*/distance
----- Original Message -----
Quoted 46 lines
From: "Peter Zijlstra" <[email protected]>
To: "ZhouPing Liu" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>, "Linus Torvalds" <[email protected]>, "Hillf Danton" <[email protected]>, [email protected], "LKML" <[email protected]>
Sent: Saturday, June 9, 2012 2:46:57 AM
Subject: Re: Kernel panic - not syncing: Attempted to kill the idle task!Quoted 29 lines · ZhouPing Liu
kernel panic on mainline(commit:
48d212a2eecaca) with a large system, which
has 120Gb RAM & 8 numa nodes:... [cut here] ...
[ 3.404017] Call Trace: [ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0 [ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50 [ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10 [ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80 [ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0 [ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0 [ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00 [ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70 [ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140 [ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264 [ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 0000 48
8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0 48 c1
e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90 0f[ 3.420335] RIP [<ffffffff810af93b>] update_sd_lb_stats+0x27b/0x620 [ 3.421664] RSP <ffff88041922fb48> [ 3.422473] ---[ end trace 04b848dd1c06d585 ]--- [ 3.423472] Kernel panic - not syncing: Attempted to kill theidle task!
Quoted 7 lines
here I do a simple summary:
b), mainline, v3.5-rc1(commit:
48d212a2eecaca), yes
d), tip/master(commit:b2f5ce55c4e68370) no such
paniclet me know if you need any info.
Quoted 8 lines
There's a number of patches in tip/sched/urgent that I think fix this (hence your D) and these should make their way to Linus shortly.
That said, can you provide me your node distance table so I can verify
locally?cat
/sys/devices/system/node/node*/distance
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
Signature · 2 lines
Thanks, Zhouping
Quoted 9 lines · Zhouping Liu
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
You have to be kidding me right? That thing is a complete trainwreck, what idiot vendor did this?
If you boot that machine again, does it have the same stupid table or are we staring at white-noise?
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Zhouping Liu
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
Quoted 2 lines
You have to be kidding me right? That thing is a complete trainwreck, what idiot vendor did this?
HP ProLiant DL785 G6
http://www.sanweiying.org/download/debug_kernel/kernel_panic_48d212a2eecaca
contains dmesg and there's some serious fun with how the box boots:
node 0, cpus: 0, 8, 16...
node 1, cpus: 1, 9, 17...
Quoted 2 lines
If you boot that machine again, does it have the same stupid table or are we staring at white-noise?
I'll bet it does.
Signature · 2 lines
Regards/Gruss, Boris.
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Zhouping Liu
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
Quoted 2 lines
You have to be kidding me right? That thing is a complete trainwreck, what idiot vendor did this?
it's a HP's machine.
If I understand correctly, you meant the hardware has a bad configuration, just serious :) could you explain the reason? (you can ignore it if it's a stupid question)
Quoted 3 lines
If you boot that machine again, does it have the same stupid table or are we staring at white-noise?
indeed, it's always the same table.
And the issues is gone in the latest Linus tree(commit b84297197ce60), I'm not sure which patches fixed the issue.
Thanks,
Zhouping
Quoted 11 lines · Zhouping Liu
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Zhouping Liu
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
Quoted 6 lines
Quoted 2 lines
You have to be kidding me right? That thing is a complete trainwreck, what idiot vendor did this?
it's a HP's machine.
If I understand correctly, you meant the hardware has a bad configuration, just serious :) could you explain the reason? (you can ignore it if it's a stupid question)
^^^^it's a typo, sorry, I was willing to express my
*curious* (I recognized my bad English)
and I have one more machine which has the similar nodes table.
Thanks,
Zhouping
Quoted 11 lines · Zhouping Liu
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Zhouping Liu
# cat /sys/devices/system/node/node*/distance 10 17 17 24 24 24 30 30 18 10 30 18 18 24 24 24 18 24 10 24 24 17 30 30 24 18 23 10 24 17 17 30 24 17 24 24 10 18 30 18 31 24 17 18 18 10 24 24 30 24 30 17 24 24 10 18 30 24 30 24 17 24 17 10
Quoted 7 lines
Quoted 2 lines
You have to be kidding me right? That thing is a complete trainwreck, what idiot vendor did this?
it's a HP's machine.
If I understand correctly, you meant the hardware has a bad configuration, just serious :) could you explain the reason? (you can ignore it if it's a stupid question)
Not only a weird hardware setup (although if we go by this SLIT table, then that too).
The table itself has various problems:
1) the table isn't symmetric; T(i,j) != T(j,i), for instance, the
distance from 0->1 is different from 1->0 (17 vs 18).
2) 4 nodes have 2 connections, 4 nodes have 3 connections. Which have 2
connections seems completely without pattern, see 4).
3) if we read it like: 10 (self), {17,18} 1 hop, {23,24} 2 hops,
{30,31} 3 hops, its still obviously wrong, see 1->2 and 2->1 (this goes back to point 1 as well). 1->2 takes 3 hops while 2->1 takes 2 hops. Going by the 1 hop connections (which aside from the 17 vs 18 mess) are symmetric, both these should be 2 hops (1<->0<->2 in fact). There's multiple such 'mistakes'.
4) take a piece of paper and draw a cube, mark each corner as a node,
then high-light the single hop edges and be awestruck by the creative wiring.
This is by far the most 'creative' SLIT table I have ever seen and of course its HP again.. those guys have the most shitty BIOS record ever.
Now we could 'fix' up this table by doing a min-symmetry filter over it, but I'm tempted to just give up and do a single machine wide fall-back domain when we find crappy tables like this.
Quoted 10 lines · Linus Torvalds
Peter, you are very fundamentally wrong if you think the distance array has to be symmetric. That is fundamentally not true for many typologies.
The trivial example of a non symmetric case is just a simple unidirectional ring. The distance from n to n+1 is just one, but the distance from n+1 to n is n-1 hops.
So don't try to say that distances have to be symmetric. That's just garbage.
Sure, I realize this, but the 17,18 thing isn't a ring. It looks like something that should be symmetric but isn't.
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Linus Torvalds
Peter, you are very fundamentally wrong if you think the distance array has to be symmetric. That is fundamentally not true for many typologies.
The trivial example of a non symmetric case is just a simple unidirectional ring. The distance from n to n+1 is just one, but the distance from n+1 to n is n-1 hops.
Quoted 5 lines
Quoted 2 lines
So don't try to say that distances have to be symmetric. That's just garbage.
Sure, I realize this, but the 17,18 thing isn't a ring. It looks like something that should be symmetric but isn't.
Not a unidirectional one that is.. I played around a bit more and found a shape that isn't too odd:
2
/ \
0 5
/ / \
1 --- / --- 3
\ / /
4 6
\ /
7 I would've crossed 0<->6 instead of 1<->3 so the 2/3 connected nodes are spread better, but what do I know.
The only really odd thing is the numbering, which is what threw me.
But yeah, the possibility of uni-directional rings makes detecting obvious crack tables harder, which is why I haven't got it -- yet.
Quoted 2 lines · Linus Torvalds
you are very fundamentally wrong if you think the distance array has to be symmetric.
I just checked, both QPI and HT are bi-directional links so anything mid-sized x86 had better be symmetric. Big numa can of course stick in their own interconnects and might do uni-directional stuff, although I doubt anybody still does that.
The distance table I have for a 4096 cpu SGI machine is fully symmetric for instance.
This of course doesn't mean we should make hard assumptions about this in the scheduler -- and we don't. But in this case it does mean the HP table is very funny indeed.