2.6.12-rc4-mm2 boot failure
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Oops: Exception in kernel mode, sig: 5 [#1]^M SMP NR_CPUS=32 NUMA PSERIES LPAR ^M Modules linked in:^M NIP: C000000000099624 XER: 00000000 LR: C00000000009A014 CTR: C00000000028C0D4^M REGS: c00000000057ba10 TRAP: 0700 Not tainted (2.6.12-rc4-mm2-autokern1)^M MSR: 8000000000029032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 24004022^M DAR: 8000000000009032 DSISR: c0000000006c82bf^M TASK: c0000000005e2100[0] 'swapper' THREAD: c000000000578000 CPU: 0^M
GPR00: 0000000000000001 C00000000057BC90 C0000000006C0568 C00000077FFD2590 ^M
GPR04: 0000000000000000 FFFFFFFFFFFFFFFF C0000000006C83D0 C0000000005E3A24 ^M
GPR08: C0000000005E3A18 0000000000000000 C0000000006C83C8 C0000000006C82E8 ^M
GPR12: 000000000000000A C0000000005CD000 0000000000000000 0000000000000000 ^M
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ^M
GPR20: 0000000000230000 0000000003A10000 0000000000000060 0000000003F143C8 ^M
GPR24: C0000000005CD000 C0000000006BE208 C000000000577D68 0000000000008000 ^M
GPR28: 0000000000000000 00000000000080D0 C0000000005E2100 0000000000000001 ^M NIP [c000000000099624] .interleave_nodes+0x38/0xd0^M
LR [c00000000009a014] .alloc_pages_current+0x100/0x134^M
Call Trace:^M
[c00000000057bc90] [000000000000001d] 0x1d (unreliable)^M [c00000000057bd20] [c00000000009a014] .alloc_pages_current+0x100/0x134^M [c00000000057bdc0] [c00000000007abd4] .get_zeroed_page+0x28/0x90^M [c00000000057be40] [c0000000004e2e68] .pidmap_init+0x24/0xa0^M [c00000000057bed0] [c0000000004c7734] .start_kernel+0x21c/0x30c^M [c00000000057bf90] [c00000000000c010] .__setup_cpu_power3+0x0/0x4^M Instruction dump:^Mfba1ffe8 fbc1fff0 f8010010 f821ff71 60000000 ebcd0160 a93e0788 793f0020 ^M 7fe9fe70 7d20fa78 7c004850 54000ffe <0b000000> 3ba30010 38bf0001 38800001 ^M
<0>Kernel panic - not syncing: Attempted to kill the idle task!^M
Quoted 3 lines · Martin J. Bligh
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Probably. Christoph, this patch has crossed the grief threshold - I'll drop it.
Quoted 7 lines · --On Monday, May 16, 2005 14:25:04 -0700 Andrew Morton <[email protected]> wrote:
Quoted 3 lines · Martin J. Bligh
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Probably. Christoph, this patch has crossed the grief threshold - I'll drop it.
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
M.
Quoted 7 lines · Martin J. Bligh
Quoted 5 lines · --On Monday, May 16, 2005 14:25:04 -0700 Andrew Morton <[email protected]> wrote:
Quoted 3 lines · Martin J. Bligh
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Quoted 6 lines
Quoted 2 lines
Probably. Christoph, this patch has crossed the grief threshold - I'll drop it.
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
Some description of the failure would be helpful. A boot log? .config?
Does the box have CONFIG_NUMA off and CONFIG_DISCONTIG on?
Quoted 12 lines · --On Monday, May 16, 2005 14:40:57 -0700 Christoph Lameter…
Quoted 10 lines · Martin J. Bligh
Quoted 7 lines · --On Monday, May 16, 2005 14:25:04 -0700 Andrew Morton <[email protected]> wrote:
Quoted 3 lines · Martin J. Bligh
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Probably. Christoph, this patch has crossed the grief threshold - I'll drop it.
Quoted 7 lines
Quoted 3 lines
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
Some description of the failure would be helpful. A boot log? .config?
Does the box have
CONFIG_NUMAoff andCONFIG_DISCONTIGon?
attatched boot log. Config file is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/p570
M.
Connected to pSeries-101 console [channel connected] (~$quit to exit)
-- 0:conmux-control -- time-stamp -- May/16/05 9:51:14 --
-- 0:conmux-control -- time-stamp -- May/16/05 9:54:40 --
INIT: Switching to runlevel: 6
INIT: Sending processes the TERM signal
Boot logging started on /dev/hvc0(/dev/console) at Mon May 16 08:43:20 2005
Master Resource Control: previous runlevel: 3, switching to runlevel: [80C[10D[1m6[m
[m[?25hShutting down xinetd:7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down CRON daemon7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down mail service (Postfix)7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down Name Service Cache Daemon7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down DHCP server 7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down cupsd7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down slpd 7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down ipr dump daemon [80C[10D[1;32mdone[m
[m[?25hShutting down ipr update daemon [80C[10D[1;32mdone[m
Shutting down sound driver7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down SSH daemon7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down ipr initialization daemon [80C[10D[1;32mdone[m
[m[?25hRemove Net File System (NFS)[80C[10D[1munused[m
[m[?25hUmount SMB/ CIFS File Systems 7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down resource manager7[?25l[80C[10D[1;32mdone[m8[?25h
Shutting down RPC portmap daemon7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down syslog services7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hShutting down network interfaces:
eth0 device: Intel Corp. 82545EM Gigabit Ethernet Controller (Copper) (rev 01)
eth0 configuration: eth-id-00:02:55:53:7c:39
7[?25l[1A[80C[10D[1;32mdone[m8[?25hShutting down service network . . . . . . . . . . . . . . .7[?25l
[80C[10D[1;32mdone[m8[?25h [m[?25h7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hdisabling Hotplug 7[?25l[80C[10D[1;32mdone[m8[?25h
Saving random seed7[?25l[80C[10D[1;32mdone[m8[?25h
Unloading ISDN driver 7[?25l[80C[10D[1;32mdone[m8[?25h
[m[?25hRunning /etc/init.d/halt.local
7[?25l[1A[80C[10D[1;32mdone[m8[?25hSending all processes the TERM signal...
Received signal 15 (SIGTERM)
STAFProc ending normally
[1A[80C[10D[1;32mdone[m
Sending all processes the KILL signal...
[1A[80C[10D[1;32mdone[m
7[?25l[80C[10D[1;31mmissing[m8[?25h
Set Hardware Clock to the current System Time7[?25l[80C[10D[1;32mdone[m8[?25h
Turning off swap
7[?25l[1A[80C[10D[1;32mdone[m8[?25hRestore device permissions7[?25l
[80C[10D[1;32mdone[m8[?25h
Unmounting file systems/dev/cdrom umounted
devpts umounted
tmpfs umounted/dev/sda3 umounted
7[?25l[1A[80C[10D[1;32mdone[m8[?25h7[?25l
[80C[10D[1;32mdone[m8[?25h
[80C[10D[1;33mskipped[m
proc umounted
[10;440]Please stand by while rebooting the system...
md: stopping all md devices.
md: md0 switched to read-only mode.
Restarting system.
1 = SMS Menu 5 = Default Boot List
6 = Stored Boot List 8 = Open Firmware Prompt
memory keyboard network scsi speaker
STARTING SOFTWARE
PLEASE WAIT...
-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|
Elapsed time since release of system processors: 47237 mins 56 secs
Config file read, 1112 bytes
Welcome to yaboot version 1.3.11.SuSE
Enter "help" to get some basic usage information
boot: autobench
Please wait, loading kernel...
Elf32 kernel loaded...
zImage starting: loaded at 0x400000
Allocating 0x803000 bytes for kernel ...
gunzipping (0x3a00000 <- 0x407000:0x689d64)...done 0x6d3408 bytes0xdb38 bytes of heap consumed, max in use 0xa458
initrd head: 0x381300
OF stdout device is: /vdevice/vty@0
Hypertas detected, assuming LPAR !
command line: root=/dev/sda3 selinux=0 elevator=cfq autobench_args: profile=2 profile=2 memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000004117000
alloc_top : 0000000400000000
alloc_top_hi : 0000000780000000
rmo_top : 0000000400000000
ram_top : 0000000780000000
Looking for displays
instantiating rtas at 0x00000000ffd3a000... done
0000000000000000 : boot cpu 0000000000000000
0000000000000001 : starting cpu hw idx 0000000000000001... done 0000000000000002 : starting cpu hw idx 0000000000000002... done 0000000000000003 : starting cpu hw idx 0000000000000003... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000004218000 -> 0x000000000421938a
Device tree struct 0x000000000421a000 -> 0x0000000004224000
Calling quiesce ...
returning from prom_initfirmware_features = 0xd7f
Partition configured for 16 cpus.
Starting Linux PPC64 2.6.12-rc4-mm2-autokern1
-----------------------------------------------------
ppc64_pft_size = 0x1d ppc64_debug_switch = 0x0 ppc64_interrupt_controller = 0x2 systemcfg = 0xc00000000057c000 systemcfg->platform = 0x101 systemcfg->processorCount = 0x4
systemcfg->physicalMemorySize = 0x780000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80htab_address = 0x0000000000000000htab_hash_mask = 0x3fffff
-----------------------------------------------------
[boot]0100 MM Init
[boot]0100 MM Init Done
Linux version 2.6.12-rc4-mm2-autokern1 (root@el9-92-101) (gcc version 3.3.3-hammer) #1 SMP Mon May 16 08:42:00 PDT 2005 [boot]0012 Setup Arch
Top of RAM: 0x780000000, Total RAM: 0x780000000
Memory hole size: 0MB
Syscall map setup, 232 32 bits and 208 64 bits syscalls
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 20480 bytes
Using default idle loop
[boot]0015 Setup Done
Built 1 zonelists
Kernel command line: root=/dev/sda3 selinux=0 elevator=cfq autobench_args: profile=2 profile=2 kernel profiling enabled (shift: 2)
kernel profiling enabled (shift: 2)
[boot]0020 XICS Init
xics: no ISA interrupt controller
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 131072 bytes)time_init: decrementer frequency = 212.995842 MHztime_init: processor frequency = 1703.966736 MHzfirmware_features = 0xd7f
Partition configured for 16 cpus.
Starting Linux PPC64 2.6.12-rc4-mm2-autokern1
-----------------------------------------------------
ppc64_pft_size = 0x1d ppc64_debug_switch = 0x0 ppc64_interrupt_controller = 0x2 systemcfg = 0xc00000000057c000 systemcfg->platform = 0x101 systemcfg->processorCount = 0x4
systemcfg->physicalMemorySize = 0x780000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80htab_address = 0x0000000000000000htab_hash_mask = 0x3fffff
-----------------------------------------------------
[boot]0100 MM Init
[boot]0100 MM Init Done
Linux version 2.6.12-rc4-mm2-autokern1 (root@el9-92-101) (gcc version 3.3.3-hammer) #1 SMP Mon May 16 08:42:00 PDT 2005 [boot]0012 Setup Arch
Top of RAM: 0x780000000, Total RAM: 0x780000000
Memory hole size: 0MB
Syscall map setup, 232 32 bits and 208 64 bits syscalls
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 20480 bytes
Using default idle loop
[boot]0015 Setup Done
Built 1 zonelists
Kernel command line: root=/dev/sda3 selinux=0 elevator=cfq autobench_args: profile=2 profile=2 kernel profiling enabled (shift: 2)
kernel profiling enabled (shift: 2)
[boot]0020 XICS Init
xics: no ISA interrupt controller
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 131072 bytes)time_init: decrementer frequency = 212.995842 MHztime_init: processor frequency = 1703.966736 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
freeing bootmem node 0
Memory: 30961484k/31457280k available (4332k kernel code, 495104k reserved, 1304k data, 272k bss, 708k init) kernel BUG in interleave_nodes at mm/mempolicy.c:701!
Oops: Exception in kernel mode, sig: 5 [#1] SMP NR_CPUS=32 NUMA PSERIES LPAR Modules linked in: NIP: C000000000099624 XER: 00000000 LR: C00000000009A014 CTR: C00000000028C0D4 REGS: c00000000057ba10 TRAP: 0700 Not tainted (2.6.12-rc4-mm2-autokern1) MSR: 8000000000029032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 24004022 DAR: 8000000000009032 DSISR: c0000000006c82bf TASK: c0000000005e2100[0] 'swapper' THREAD: c000000000578000 CPU: 0
GPR00: 0000000000000001 C00000000057BC90 C0000000006C0568 C00000077FFD2590
GPR04: 0000000000000000 FFFFFFFFFFFFFFFF C0000000006C83D0 C0000000005E3A24
GPR08: C0000000005E3A18 0000000000000000 C0000000006C83C8 C0000000006C82E8
GPR12: 000000000000000A C0000000005CD000 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000230000 0000000003A10000 0000000000000060 0000000003F143C8
GPR24: C0000000005CD000 C0000000006BE208 C000000000577D68 0000000000008000
GPR28: 0000000000000000 00000000000080D0 C0000000005E2100 0000000000000001 NIP [c000000000099624] .interleave_nodes+0x38/0xd0
LR [c00000000009a014] .alloc_pages_current+0x100/0x134
Call Trace:
[c00000000057bc90] [000000000000001d] 0x1d (unreliable) [c00000000057bd20] [c00000000009a014] .alloc_pages_current+0x100/0x134 [c00000000057bdc0] [c00000000007abd4] .get_zeroed_page+0x28/0x90 [c00000000057be40] [c0000000004e2e68] .pidmap_init+0x24/0xa0 [c00000000057bed0] [c0000000004c7734] .start_kernel+0x21c/0x30c [c00000000057bf90] [c00000000000c010] .__setup_cpu_power3+0x0/0x4 Instruction dump:fba1ffe8 fbc1fff0 f8010010 f821ff71 60000000 ebcd0160 a93e0788 793f0020 7fe9fe70 7d20fa78 7c004850 54000ffe <0b000000> 3ba30010 38bf0001 38800001
<0>Kernel panic - not syncing: Attempted to kill the idle task! -- 0:conmux-control -- time-stamp -- May/16/05 10:00:30 --
Quoted 9 lines · --On Monday, May 16, 2005 14:36:21 -0700 "Martin J. Bligh" <[email protected]>…
Quoted 5 lines · --On Monday, May 16, 2005 14:25:04 -0700 Andrew Morton <[email protected]> wrote:
Quoted 3 lines · Martin J. Bligh
PPC64 NUMA box. Maybe this is the same NUMA slab problem you were hitting before ...
Quoted 6 lines
Quoted 2 lines
Probably. Christoph, this patch has crossed the grief threshold - I'll drop it.
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
OK, I backed out the slab patches from -mm2, and confirmed the problem went away.
M.
Quoted 6 lines · Martin J. Bligh
Quoted 3 lines
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
OK, I backed out the slab patches from -mm2, and confirmed the problem went away.
Is there any way I can access the system to figure out what is wrong? The failure is in the page allocator and it seems that a node id is wrong.
--Christoph Lameter <[email protected]> wrote (on Tuesday, May 17, 2005 18:07:16 -0700):
Quoted 11 lines · Martin J. Bligh
Quoted 7 lines
Quoted 4 lines
OK, fair enough. Christoph, I am interested in seeing your patch work ... is something that's needed. If you want, I can help you offline with some testing on a variety of platforms.
OK, I backed out the slab patches from -mm2, and confirmed the problem went away.
Is there any way I can access the system to figure out what is wrong? The failure is in the page allocator and it seems that a node id is wrong.
Not really - IBM doesn't tend to like letting outside parties into their network ;-) I think OSDL might have some power boxes now ... maybe it fails on there?
M.