2.6.14-rc1-git-now still dying in mm/slab - this time line 1849
Hello,
so now once crashes on UP system were sorted out, I tried to
put new kernel on my SMP host - and sorry to say, but it does not seem to work as advertised :-( It seems that we somehow got blocks from CPU#1 into memory blocks on CPU#0, and free_block complains that caller holds cachep->nodelists[0]->list_lock while nodeid for block passed to free_block() comes from processor (and node) #1...
I cannot find how this happened. Hopefully somebody else
will know... Meanwhile I'll try to get rid of PREEMPT, apparently although it is now masqueraded under 'Low-latency desktop' it is still somewhat dangerous. If it is triggered by preempt, that is.
Thanks,
Petr Vandrovec
ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 0) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA] io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pktcdvd: v0.2.0a 2004-07-14 Jens Axboe ([email protected]) and [email protected] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controller
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/slab.c:1849
invalid operand: 0000 [1] PREEMPT SMP
CPU 0
Modules linked in:
Pid: 8, comm: events/0 Not tainted 2.6.14-rc1-1619 #1
RIP: 0010:[<ffffffff8016e826>] <ffffffff8016e826>{free_block+294}
RSP: 0000:ffff81007ff21d88 EFLAGS: 00010002
RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000310
RDX: 0000000000000000 RSI: ffff81007ffddd10 RDI: ffff81007ffda080
RBP: ffff81007ffde000 R08: ffff81003ffaed90 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff81007ffc9b50
R13: ffff81007ffde048 R14: ffff81007ffda080 R15: ffff81007ffda080
FS: 0000000000000000(0000) GS:ffffffff805fb800(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
Process events/0 (pid: 8, threadinfo ffff81007ff20000, task ffff81003ff8c790)
Stack: 0000000000000000 0000000000000000 0000000000000213 0000000200000000
ffff81007ffddd10 ffff81007ffddd10 ffff81007ffddce8 0000000000000002
0000000000000000 ffff81007ffda080
Call Trace:<ffffffff8016fdc7>{drain_array_locked+167} <ffffffff8016feee>{cache_reap+206}
<ffffffff803a2374>{_spin_lock_irqsave+36} <ffffffff8016fe20>{cache_reap+0}
<ffffffff8014a1bc>{worker_thread+476} <ffffffff80132610>{default_wake_function+0}
<ffffffff80132610>{default_wake_function+0} <ffffffff80149fe0>{worker_thread+0}
<ffffffff8014ebc2>{kthread+146} <ffffffff8010ed12>{child_rip+8}
<ffffffff80149fe0>{worker_thread+0} <ffffffff8014eb30>{kthread+0}
<ffffffff8010ed0a>{child_rip+0}
Code: 0f 0b 68 bd aa 3d 80 c2 39 07 48 89 ee 4c 89 ff 4c 8d 75 30RIP <ffffffff8016e826>{free_block+294} RSP <ffff81007ff21d88>
<6>note: events/0[8] exited with preempt_count 1
hda: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: WDC WD1200JB-00CRA0, ATA DISK drive
ide1 at 0x170-0x177,0x376 on irq 15
hdc: max request size: 128KiB
hdc: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100)
hdc: cache flushes not supported
hdc: hdc1
libata version 1.12 loaded.sata_sil version 0.9
ACPI: PCI Interrupt 0000:01:0b.0[A] -> GSI 17 (level, low) -> IRQ 177 <and box is dead>
Quoted 9 lines · Petr Vandrovec
Hello,
so now once crashes on UP system were sorted out, I tried to
put new kernel on my SMP host - and sorry to say, but it does not seem to work as advertised :-( It seems that we somehow got blocks from CPU#1 into memory blocks on CPU#0, andfree_blockcomplains that caller holds cachep->nodelists[0]->list_lock while nodeid for block passed tofree_block()comes from processor (and node) #1...
Quoted 4 lines
I cannot find how this happened. Hopefully somebody else
will know... Meanwhile I'll try to get rid of PREEMPT, apparently although it is now masqueraded under 'Low-latency desktop' it is still somewhat dangerous. If it is triggered by preempt, that is.
It is not caused by preempt, non-preempt kernel crashes exactly same way.
Petr
Quoted 77 lines
Thanks,
Petr VandrovecttyS0 at I/O
0x3f8(irq = 0) is a 16550A
ttyS1 at I/O0x2f8(irq = 0) is a 16550A
ttyS0 at I/O0x3f8(irq = 4) is a 16550A
ttyS1 at I/O0x2f8(irq = 3) is a 16550A
parport: PnPBIOS parport detected.
parport0: PC-style at0x378(0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA] io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pktcdvd: v0.2.0a 2004-07-14 Jens Axboe ([email protected]) and [email protected] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controlleride0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at mm/slab.c:1849 invalid operand: 0000 [1] PREEMPT SMP CPU 0 Modules linked in: Pid: 8, comm: events/0 Not tainted 2.6.14-rc1-1619 #1 RIP: 0010:[<ffffffff8016e826>] <ffffffff8016e826>{free_block+294} RSP: 0000:ffff81007ff21d88 EFLAGS: 00010002 RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000310 RDX: 0000000000000000 RSI: ffff81007ffddd10 RDI: ffff81007ffda080 RBP: ffff81007ffde000 R08: ffff81003ffaed90 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff81007ffc9b50 R13: ffff81007ffde048 R14: ffff81007ffda080 R15: ffff81007ffda080 FS: 0000000000000000(0000) GS:ffffffff805fb800(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0 Process events/0 (pid: 8, threadinfo ffff81007ff20000, task ffff81003ff8c790) Stack: 0000000000000000 0000000000000000 0000000000000213 0000000200000000 ffff81007ffddd10 ffff81007ffddd10 ffff81007ffddce8 0000000000000002 0000000000000000 ffff81007ffda080 Call Trace:<ffffffff8016fdc7>{drain_array_locked+167} <ffffffff8016feee>{cache_reap+206} <ffffffff803a2374>{_spin_lock_irqsave+36} <ffffffff8016fe20>{cache_reap+0} <ffffffff8014a1bc>{worker_thread+476} <ffffffff80132610>{default_wake_function+0} <ffffffff80132610>{default_wake_function+0} <ffffffff80149fe0>{worker_thread+0} <ffffffff8014ebc2>{kthread+146} <ffffffff8010ed12>{child_rip+8} <ffffffff80149fe0>{worker_thread+0} <ffffffff8014eb30>{kthread+0} <ffffffff8010ed0a>{child_rip+0}Code: 0f 0b 68 bd aa 3d 80 c2 39 07 48 89 ee 4c 89 ff 4c 8d 75 30RIP <ffffffff8016e826>{free_block+294} RSP <
ffff81007ff21d88>
<6>note: events/0[8] exited withpreempt_count1
hda:_NECDVD_RWND-3500AG, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: WDC WD1200JB-00CRA0, ATA DISK drive
ide1 at 0x170-0x177,0x376 on irq 15
hdc: max request size: 128KiB
hdc: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100)
hdc: cache flushes not supported
hdc: hdc1
libata version 1.12 loaded.sata_silversion 0.9
ACPI: PCI Interrupt 0000:01:0b.0[A] -> GSI 17 (level, low) -> IRQ 177 <and box is dead>-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Andrew Morton
Quoted 7 lines · Petr Vandrovec
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Petr Vandrovec
so now once crashes on UP system were sorted out, I tried to
put new kernel on my SMP host - and sorry to say, but it does not seem to work as advertised :-(
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
.config (again), please.
Any SMP with NUMA. One which I'm trying to debug now is attached. It is available at http://vana.vc.cvut.cz/config as well.
I can get 2.6.14-rc1 to crash with your .config, but current -linus is OK.
It still dies for me, with current git (tree 7513cdadc661cfe0bd1625145a4876e54df191ca, commit 6c0741fbdee5bd0f8ed13ac287c4ab18e8ba7d83). Config is available at
http://platan.vc.cvut.cz/config-vana.txt. Box is dual opteron Tyan K8W, S2885.
Any idea how to track problem down? I'm not sure bisect will work without lot of interaction & patching, as almost all kernels after 2.6.13 were dying with some other problems on that box...
Thanks,
Petr Vandrovec
Bootdata ok (command line is BOOT_IMAGE=Linux ro root=801 ramdisk=0 console=ttyS0,115200 console=tty0 nmi_watchdog=2 psmouse_noext=1 verbose) Linux version 2.6.14-rc1-6c07 (root@vana) (gcc version 3.3.3 (Debian 20040401)) #4 SMP Mon Sep 19 17:44:44 CEST 2005 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
BIOS-e820: 000000007fff0000 - 000000007ffff000 (ACPI data)
BIOS-e820: 000000007ffff000 - 0000000080000000 (ACPI NVS)
BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: Node 0 PXM 0 100000-3fffffff
SRAT: Node 1 PXM 1 40000000-7fffffff
SRAT: Node 0 PXM 0 0-3fffffff
Bootmem setup node 0 0000000000000000-000000003fffffff
Bootmem setup node 1 0000000040000000-000000007ffeffff
ACPI: PM-Timer IO Port: 0x5008
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x03] address[0xff4ff000] gsi_base[24])
IOAPIC[1]: apic_id 3, version 17, address 0xff4ff000, GSI 24-27
ACPI: IOAPIC (id[0x04] address[0xff4fe000] gsi_base[28])
IOAPIC[2]: apic_id 4, version 17, address 0xff4fe000, GSI 28-31
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Setting APIC routing to flat
ACPI: HPET id: 0x102282a0 base: 0xfec01000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7f780000)
Checking aperture...
CPU 0: aperture @ c0000000 size 512 MB
CPU 1: aperture @ c0000000 size 512 MB
Built 2 zonelists
Kernel command line: BOOT_IMAGE=Linux ro root=801 ramdisk=0 console=ttyS0,115200 console=tty0 nmi_watchdog=2 psmouse_noext=1 verbose Parameter psmouse_noext is obsolete, ignored
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)time.c: Using 14.318180 MHz HPET timer.time.c: Detected 1993.374 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Memory: 2056328k/2097088k available (2616k kernel code, 40372k reserved, 1869k data, 248k init)
Calibrating delay using timer specific routine.. 3991.40 BogoMIPS (lpj=19957004)
Security Framework v1.0.0 initialized
SELinux: Initializing. SELinux: Starting in permissive mode Mount-cache hash table entries: 256 CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(1) -> Node 0 -> Core 0
mtrr: v2.0 (20020519)
Using local APIC timer interrupts.
Detected 12.458 MHz APIC timer.
softlockup thread 0 started up.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3986.60 BogoMIPS (lpj=19933040)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1(1) -> Node 1 -> Core 0
AMD Opteron(tm) Processor 246 stepping 0a
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff -8 cycles, maxerr 1095 cycles) Brought up 2 CPUs
softlockup thread 1 started up.time.c: Using HPET based timekeeping.
testing NMI watchdog ... OK.
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20050902
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: PCI Root Bridge [PCIB] (0000:04)
PCI: Probing PCI hardware (bus 04)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 7 9 10 11 12 14 15) Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report TC classifier action (bugs to [email protected] cc [email protected])
hpet0: at MMIO 0xfec01000, IRQs 2, 8, 0
hpet0: 69ns tick, 3 32-bit timers
agpgart: Detected AMD 8151 AGP Bridge rev B3
agpgart: AGP aperture is 512M @ 0xc0000000
PCI-DMA: Disabling IOMMU.
pnp: 00:09: ioport range 0x680-0x6ff has been reserved
pnp: 00:09: ioport range 0x295-0x296 has been reserved
pnp: 00:09: ioport range 0xb78-0xb7f has been reserved
pnp: 00:09: ioport range 0xf78-0xf7f has been reserved
PCI: Bridge: 0000:00:06.0
IO window: 9000-afff
MEM window: ff100000-ff2fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:0a.0
IO window: disabled.
MEM window: ff300000-ff3fffff
PREFETCH window: 9e900000-9e9fffff
PCI: Bridge: 0000:00:0b.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:04:01.0
IO window: c000-cfff
MEM window: ff500000-ff5fffff
PREFETCH window: 9eb00000-beafffff
IA-32 Microcode Update Driver: v1.14 <[email protected]>
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1127144995.320:1): initialized
Total HugeTLB memory allocated, 0
SELinux: Registering netfilter hooks
Initializing Cryptographic API
PCI: MSI quirk detected. pci_msi_quirk set.
PCI: MSI quirk detected. pci_msi_quirk set.
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 16 (level, low) -> IRQ 169
radeonfb: Found Intel x86 BIOS ROM Image
radeonfb: Retreived PLL infos from BIOS
radeonfb: Reference=27.00 MHz (RefDiv=12) Memory=200.00 Mhz, System=166.00 MHz
radeonfb: PLL min 20000 max 40000
radeonfb: Monitor 1 type DFP found
radeonfb: EDID probed
radeonfb: Monitor 2 type no found
Console: switching to colour frame buffer device 240x75
radeonfb (0000:05:00.0): ATI Radeon Yd
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
Using specific hotkey driver
ACPI: CPU0 (power states: C1[C1])
ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: CPU1 (power states: C1[C1])
Real Time Clock Driver v1.12hpet_acpi_add: no address or irqs in _CRS
Linux agpgart interface v0.101 (c) Dave Jones
[drm] Initialized drm 1.0.0 20040925
PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
PNP: PS/2 controller doesn't have AUX irq; using default 12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 0) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA] io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pktcdvd: v0.2.0a 2004-07-14 Jens Axboe ([email protected]) and [email protected] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controller
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/slab.c:1849
invalid operand: 0000 [1] SMP
CPU 0
Modules linked in:
Pid: 8, comm: events/0 Not tainted 2.6.14-rc1-6c07 #1
RIP: 0010:[<ffffffff8016d316>] <ffffffff8016d316>{free_block+294}
RSP: 0000:ffff81007ff21d88 EFLAGS: 00010002
RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000310
RDX: 0000000000000000 RSI: ffff81007ffddd10 RDI: ffff81007ffda080
RBP: ffff81007ffde000 R08: ffff81003ffa0d50 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff81007ffc9b50
R13: ffff81007ffde048 R14: ffff81007ffda080 R15: ffff81007ffda080
FS: 0000000000000000(0000) GS:ffffffff805f2800(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
Process events/0 (pid: 8, threadinfo ffff81007ff20000, task ffff81003ff8c790)
Stack: 0000000000000000 0000000000000000 0000000000000292 hda: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive
0000000200000000
ffff81007ffddd10 ffff81007ffddd10 ffff81007ffddce8 0000000000000002
0000000000000000 ffff81007ffda080
Call Trace:<ffffffff8016e8b7>{drain_array_locked+167} <ffffffff8016e9f7>{cache_reap+231}
<ffffffff80131e23>{__wake_up+67} <ffffffff8016e910>{cache_reap+0}
<ffffffff8014930c>{worker_thread+476} <ffffffff80131d60>{default_wake_function+0}
<ffffffff80131d60>{default_wake_function+0} <ffffffff80149130>{worker_thread+0}
<ffffffff8014db82>{kthread+146} <ffffffff8010ec22>{child_rip+8}
<ffffffff80149130>{worker_thread+0} <ffffffff8014daf0>{kthread+0}
<ffffffff8010ec1a>{child_rip+0}
Code: 0f 0b 68 9d 26 3d 80 c2 39 07 48 89 ee 4c 89 ff 4c 8d 75 30RIP <ffffffff8016d316>{free_block+294} RSP <ffff81007ff21d88>
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: WDC WD1200JB-00CRA0, ATA DISK drive
ide1 at 0x170-0x177,0x376 on irq 15
hdc: max request size: 128KiB
hdc: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100)
hdc: cache flushes not supported
hdc: hdc1
libata version 1.12 loaded.sata_sil version 0.9
ACPI: PCI Interrupt 0000:01:0b.0[A] -> GSI 17 (level, low) -> IRQ 177
Quoted 56 lines · Petr Vandrovec
Quoted 9 lines · Andrew Morton
Quoted 7 lines · Petr Vandrovec
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Petr Vandrovec
so now once crashes on UP system were sorted out, I tried to
put new kernel on my SMP host - and sorry to say, but it does not seem to work as advertised :-(Quoted 6 lines
Quoted 4 lines
Quoted 1 line
.config (again), please.
Any SMP with NUMA. One which I'm trying to debug now is attached. It is available at http://vana.vc.cvut.cz/config as well.
I can get 2.6.14-rc1 to crash with your .config, but current -linus is OK.
It still dies for me, with current git (tree
7513cdadc661cfe0bd1625145a4876e54df191ca, commit6c0741fbdee5bd0f8ed13ac287c4ab18e8ba7d83). Config is available at
http://platan.vc.cvut.cz/config-vana.txt. Box is dual opteron Tyan K8W, S2885....
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at mm/slab.c:1849 invalid operand: 0000 [1] SMP CPU 0 Modules linked in: Pid: 8, comm: events/0 Not tainted 2.6.14-rc1-6c07 #1 RIP: 0010:[<ffffffff8016d316>] <ffffffff8016d316>{free_block+294} RSP: 0000:ffff81007ff21d88 EFLAGS: 00010002 RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000310 RDX: 0000000000000000 RSI: ffff81007ffddd10 RDI: ffff81007ffda080 RBP: ffff81007ffde000 R08: ffff81003ffa0d50 R09: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff81007ffc9b50 R13: ffff81007ffde048 R14: ffff81007ffda080 R15: ffff81007ffda080 FS: 0000000000000000(0000) GS:ffffffff805f2800(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0 Process events/0 (pid: 8, threadinfo ffff81007ff20000, task ffff81003ff8c790) Stack: 0000000000000000 0000000000000000 0000000000000292 hda: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive 0000000200000000 ffff81007ffddd10 ffff81007ffddd10 ffff81007ffddce8 0000000000000002 0000000000000000 ffff81007ffda080 Call Trace:<ffffffff8016e8b7>{drain_array_locked+167} <ffffffff8016e9f7>{cache_reap+231} <ffffffff80131e23>{__wake_up+67} <ffffffff8016e910>{cache_reap+0} <ffffffff8014930c>{worker_thread+476} <ffffffff80131d60>{default_wake_function+0} <ffffffff80131d60>{default_wake_function+0} <ffffffff80149130>{worker_thread+0} <ffffffff8014db82>{kthread+146} <ffffffff8010ec22>{child_rip+8} <ffffffff80149130>{worker_thread+0} <ffffffff8014daf0>{kthread+0} <ffffffff8010ec1a>{child_rip+0} Code: 0f 0b 68 9d 26 3d 80 c2 39 07 48 89 ee 4c 89 ff 4c 8d 75 30
Quoted 2 lines
RIP <ffffffff8016d316>{free_block+294} RSP <
ffff81007ff21d88>
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Well. The CPU_UP_CANCELED locking in cpuup_callback() looks borked to me - it takes cachep->nodelists[node]->list_lock and then calls drain_alien_cache() which appears to take the same lock. But that's not the problem here.
The code in cache_reap() recalculates numa_node_id() multiple times, so if the caller changes CPUs then this assertion will trigger. However it's running under keventd here, which is pinned to a single CPU. Still, it would be useful if you could try putting preempt_disable()s in cache_reap(), or change cache_reap() to evaluate numa_node_id() just the once, and cache that in a local variable.
I wonder why numa_node_id() uses raw_smp_processor_id()? That's just asking for preempt non-atomicity bugs.
Quoted 11 lines · Andrew Morton
Well. The
CPU_UP_CANCELEDlocking incpuup_callback()looks borked to me - it takes cachep->nodelists[node]->list_lock and then callsdrain_alien_cache()which appears to take the same lock. But that's not the problem here.The code in
cache_reap()recalculatesnuma_node_id()multiple times, so if the caller changes CPUs then this assertion will trigger. However it's running under keventd here, which is pinned to a single CPU. Still, it would be useful if you could try putting preempt_disable()s incache_reap(), or changecache_reap()to evaluatenuma_node_id()just the once, and cache that in a local variable.
drain_array_cache_locked calls check_spinlock_acquired_node which is in turn insuring that interrupts are off. So no move to a different processor should be possible.
However, that is contradicted by __wake_up calling drain_array_cache_locked. The process just woke up?
Quoted 2 lines
I wonder why
numa_node_id()usesraw_smp_processor_id()? That's just asking for preempt non-atomicity bugs.
Accessing arrays indexed by node number even works if the process continues to be executed on another node.
Quoted 15 lines · Christoph Lameter
Quoted 11 lines · Andrew Morton
Well. The
CPU_UP_CANCELEDlocking incpuup_callback()looks borked to me - it takes cachep->nodelists[node]->list_lock and then callsdrain_alien_cache()which appears to take the same lock. But that's not the problem here.The code in
cache_reap()recalculatesnuma_node_id()multiple times, so if the caller changes CPUs then this assertion will trigger. However it's running under keventd here, which is pinned to a single CPU. Still, it would be useful if you could try putting preempt_disable()s incache_reap(), or changecache_reap()to evaluatenuma_node_id()just the once, and cache that in a local variable.
Quoted 3 lines
drain_array_cache_lockedcallscheck_spinlock_acquired_nodewhich is in turn insuring that interrupts are off. So no move to a different processor should be possible.
list_for_each(walk, &cache_chain) {
kmem_cache_t *searchp;
struct list_head* p;
int tofree;
struct slab *slabp;
searchp = list_entry(walk, kmem_cache_t, next);
if (searchp->flags & SLAB_NO_REAP)
goto next;
check_irq_on();
l3 = searchp->nodelists[numa_node_id()];
if (l3->alien)
drain_alien_cache(searchp, l3);
->preempt here
spin_lock_irq(&l3->list_lock);
drain_array_locked(searchp, ac_data(searchp), 0,
numa_node_id());->oops, wrong node.
Still, this should all be pinned to one CPU, by happenstance.
Quoted 2 lines
However, that is contradicted by
__wake_upcallingdrain_array_cache_locked. The process just woke up?
Not sure what you mean here.
Quoted 5 lines
Quoted 2 lines
I wonder why
numa_node_id()usesraw_smp_processor_id()? That's just asking for preempt non-atomicity bugs.Accessing arrays indexed by node number even works if the process continues to be executed on another node.
That's a special case and the callers should be changed to use a new raw_numa_node_id() in that case.
Code which calls numa_node_id() and then continues to use the result of that in preemptible code is often buggy. Code which reevaluates numa_node_id() in preemptible code and assumes that it returned the same thing is even buggier (unless it happens to be CPU pinned).
numa_node_id() is doing a bad thing and should be converted to use smp_processor_id() so we can identify all the possibly-buggy callsites.
Quoted 21 lines · Andrew Morton
list_for_each(walk, &cache_chain) {
kmem_cache_t *searchp;
struct list_head* p;
int tofree;
struct slab *slabp;
searchp = list_entry(walk, kmem_cache_t, next);
if (searchp->flags & SLAB_NO_REAP)
goto next;
check_irq_on();
l3 = searchp->nodelists[numa_node_id()];
if (l3->alien)
drain_alien_cache(searchp, l3);
->preempt here
spin_lock_irq(&l3->list_lock);
drain_array_locked(searchp, ac_data(searchp), 0,
numa_node_id());Quoted 1 line
->oops, wrong node.
This is called from keventd which exists per processor. Hmmm... This looks as if it can change processors after all but the slab allocator depends on it running on the right processor. So does the page allocator. sigh. What is the point of having per processor workqueues if they do not stay on the assigned processor?
The fast fix for this case is to get the node number once and then use it consistently. But we really need to audit the slab and page allocator for additional cases like this or disable preempt and check for the right processor in cache_reap().
mm/slab.c +4 -3
@@ -3262,6 +3262,7 @@ {struct list_head *walk;struct kmem_list3 *l3; +int node =numa_node_id ();if (down_trylock (&cache_chain_sem)) {/* Give up. Setup the next iteration. */
@@ -3282,13 +3283,13 @@check_irq_on (); - l3 = searchp->nodelists[numa_node_id ()]; + l3 = searchp->nodelists[node];if (l3->alien)drain_alien_cache (searchp, l3);spin_lock_irq (&l3->list_lock);drain_array_locked (searchp,ac_data (searchp),0 , -numa_node_id ()); + node);if (time_after (l3->next_reap, jiffies)) goto next_unlock;
@@ -3297,7 +3298,7 @@if (l3->shared)drain_array_locked (searchp, l3->shared,0 , -numa_node_id ()); + node);if (l3->free_touched) { l3->free_touched =0 ;
Quoted 22 lines · Christoph Lameter
Quoted 19 lines · Andrew Morton
list_for_each(walk, &cache_chain) { kmem_cache_t *searchp; struct list_head* p; int tofree; struct slab *slabp; searchp = list_entry(walk, kmem_cache_t, next); if (searchp->flags & SLAB_NO_REAP) goto next; check_irq_on(); l3 = searchp->nodelists[numa_node_id()]; if (l3->alien) drain_alien_cache(searchp, l3); ->preempt here spin_lock_irq(&l3->list_lock);
Quoted 6 lines
Quoted 3 lines
drain_array_locked(searchp, ac_data(searchp), 0, numa_node_id());->oops, wrong node.
This is called from keventd which exists per processor. Hmmm... This looks as if it can change processors after all
Well no, it would be a big bug if a keventd thread were to change CPUs.
It's OK to rely upon the pinnedness of keventd I guess - a comment would be nice.
Quoted 4 lines
but the slab allocator depends on
it running on the right processor. So does the page allocator. sigh. What is the point of having per processor workqueues if they do not stay on the assigned processor?
They do. I don't believe that preemption is the source of this BUG. (Petr, does CONFIG_PREEMPT=n fix it?)
Quoted 2 lines
The fast fix for this case is to get the node number once and then use it consistently.
If one is writing preempt-safe code then one should disable preemption before copying the current CPU number into a local variable.
Quoted 3 lines
But we really need to audit the slab and page allocator for additional cases like this or disable preempt and check for the right processor in
cache_reap().
numa_node_id() must use smp_processor_id(), not raw_smp_processor_id(). Then all the runtime squawks need to be audited and fixed, or switched to (new) raw_numa_node_id() if is is verified that a CPU/node switch at any time is OK.
Hi,
Attached is a patch which stores the numa_node_id in a local variable after disabling interrupts, in the cache_reap code path.
I was not able to reproduce the bug that Petr was talking about, but if the cache reap threads do schedule across cpu's which might be the problem here then this should just fix it.
Andrew, i also have a patch which fixes the CPU_DOWN code path, which i will send u later.
Thanks & Regards,
Alok.
Quoted 58 lines · Andrew Morton
Quoted 22 lines · Christoph Lameter
Quoted 19 lines · Andrew Morton
list_for_each(walk, &cache_chain) { kmem_cache_t *searchp; struct list_head* p; int tofree; struct slab *slabp; searchp = list_entry(walk, kmem_cache_t, next); if (searchp->flags & SLAB_NO_REAP) goto next; check_irq_on(); l3 = searchp->nodelists[numa_node_id()]; if (l3->alien) drain_alien_cache(searchp, l3); ->preempt here spin_lock_irq(&l3->list_lock);Quoted 6 lines
Quoted 3 lines
drain_array_locked(searchp, ac_data(searchp), 0, numa_node_id());->oops, wrong node.
This is called from keventd which exists per processor. Hmmm... This looks as if it can change processors after all
Well no, it would be a big bug if a keventd thread were to change CPUs.
It's OK to rely upon the pinnedness of keventd I guess - a comment would be nice.
Quoted 4 lines
but the slab allocator depends on
it running on the right processor. So does the page allocator. sigh. What is the point of having per processor workqueues if they do not stay on the assigned processor?They do. I don't believe that preemption is the source of this BUG. (Petr, does
CONFIG_PREEMPT=nfix it?)Quoted 2 lines
The fast fix for this case is to get the node number once and then use it consistently.
If one is writing preempt-safe code then one should disable preemption before copying the current CPU number into a local variable.
Quoted 3 lines
But we really need to audit the slab and page allocator for additional cases like this or disable preempt and check for the right processor in
cache_reap().
numa_node_id()must usesmp_processor_id(), notraw_smp_processor_id(). Then all the runtime squawks need to be audited and fixed, or switched to (new)raw_numa_node_id()if is is verified that a CPU/node switch at any time is OK.
Signature · 3 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
mm/slab.c +13 -6
@@ -3273,7 +3273,7 @@list_for_each (walk, &cache_chain) {kmem_cache_t *searchp;struct list_head * p; -int tofree; +int tofree, nodeid;struct slab *slabp; searchp =list_entry (walk, kmem_cache_t, next);
@@ -3283,13 +3283,19 @@check_irq_on (); - l3 = searchp->nodelists[numa_node_id ()]; + nodeid =numa_node_id (); + l3 = searchp->nodelists[nodeid];if (l3->alien)drain_alien_cache (searchp, l3); -spin_lock_irq (&l3->list_lock); + +local_irq_disable (); + nodeid =numa_node_id (); + l3 = searchp->nodelists[nodeid]; + +spin_lock (&l3->list_lock);drain_array_locked (searchp,ac_data (searchp),0 , -numa_node_id ()); + nodeid);if (time_after (l3->next_reap, jiffies)) goto next_unlock;
@@ -3298,7 +3304,7 @@if (l3->shared)drain_array_locked (searchp, l3->shared,0 , -numa_node_id ()); + nodeid);if (l3->free_touched) { l3->free_touched =0 ;
@@ -3327,7 +3333,8 @@spin_lock_irq (&l3->list_lock); }while (--tofree >0 ); next_unlock: -spin_unlock_irq (&l3->list_lock); +spin_unlock (&l3->list_lock); +local_irq_enable (); next:cond_resched (); }
Quoted 33 lines · Andrew Morton
Quoted 22 lines · Christoph Lameter
Quoted 19 lines · Andrew Morton
list_for_each(walk, &cache_chain) { kmem_cache_t *searchp; struct list_head* p; int tofree; struct slab *slabp; searchp = list_entry(walk, kmem_cache_t, next); if (searchp->flags & SLAB_NO_REAP) goto next; check_irq_on(); l3 = searchp->nodelists[numa_node_id()]; if (l3->alien) drain_alien_cache(searchp, l3); ->preempt here spin_lock_irq(&l3->list_lock);Quoted 5 lines
Quoted 3 lines
drain_array_locked(searchp, ac_data(searchp), 0, numa_node_id());->oops, wrong node.
This is called from keventd which exists per processor. Hmmm... This looks as if it can change processors after all
Well no, it would be a big bug if a keventd thread were to change CPUs.
Quoted 8 lines
It's OK to rely upon the pinnedness of keventd I guess - a comment would be nice.
Quoted 1 line
but the slab allocator depends on it running on the right processor. So does the page allocator. sigh. What is the point of having per processor workqueues if they do not stay on the assigned processor?
They do. I don't believe that preemption is the source of this BUG. (Petr, does
CONFIG_PREEMPT=nfix it?)
No, it does not. I've even added printks here and there to show node number, and everything works as it should. Maybe there are some problems with numa_node_id() and migrating between processors when memory gets released, I do not know.
Only thing I know that if I'll add WARN_ON below to the free_block(), it triggers...
@free_block
slabp = GET_PAGE_SLAB(virt_to_page(objp)); nodeid = slabp->nodeid;
+ WARN_ON(nodeid != numa_node_id()); <<<<<
l3 = cachep->nodelist[nodeid]; list_del(&slabp->list);
objnr = (objp - slabp->s_mem) / cachep->objsize; check_spinlock_acquired_node(cachep, nodeid); check_slabp(cachep, slabp);
... saying that keventd/0 tries to operate on
slab belonging to node#1, while having acquired lock for cachep belonging to node #0. Due to this check_spinlock_acquired_node(cachep, nodeid) fails (check_spinlock_acquired_node(cachep, 0) would succeed).
Petr
Quoted 3 lines · Petr Vandrovec
slab belonging to node#1, while having acquired lock for cachep belonging to node #0. Due to this
check_spinlock_acquired_node(cachep, nodeid)fails (check_spinlock_acquired_node(cachep, 0)would succeed).
Hmmm. If a node runs out of memory then pages from another node may end up on the slab list of a node. But it seems that free_block cannot handle that properly.
How are you producing the problem?
Could you try the following patch:
---
The numa slab allocator may allocate pages from foreign nodes onto the lists for a particular node if a node runs out of memory. Inspecting the slab->nodeid field will not reflect that the page is now in use for the slabs of another node.
This patch fixes that issue by adding a node field to free_block so that the caller can indicate which node currently uses a slab.
Also removes the check for the current node from kmalloc_cache_node since the process may shift later to another node which may lead to an allocation on another node than intended.
mm/slab.c +19 -19
@@ -639,7 +639,7 @@ static enum { static DEFINE_PER_CPU(struct work_struct, reap_work); -static void free_block (kmem_cache_t * cachep,void ** objpp,int len); +static void free_block (kmem_cache_t * cachep,void ** objpp,int len,int node);static void enable_cpucache (kmem_cache_t *cachep);static void cache_reap (void *unused);static int __node_shrink (kmem_cache_t *cachep,int node);
@@ -804,7 +804,7 @@ static inline void __drain_alien_cache(kif (ac->avail) {spin_lock (&rl3->list_lock); -free_block (cachep, ac->entry, ac->avail); +free_block (cachep, ac->entry, ac->avail, node); ac->avail =0 ;spin_unlock (&rl3->list_lock); }
@@ -925,7 +925,7 @@ static int __devinit cpuup_callback(stru/* Free limit for this kmem_list3 */ l3->free_limit -= cachep->batchcount;if (nc) -free_block (cachep, nc->entry, nc->avail); +free_block (cachep, nc->entry, nc->avail, node);if (!cpus_empty (mask)) {spin_unlock (&l3->list_lock);
@@ -934,7 +934,7 @@ static int __devinit cpuup_callback(struif (l3->shared) {free_block (cachep, l3->shared->entry, - l3->shared->avail); + l3->shared->avail, node);kfree (l3->shared); l3->shared =NULL ; }
@@ -1882,12 +1882,13 @@ static void do_drain(void *arg) {kmem_cache_t *cachep = (kmem_cache_t *)arg;struct array_cache *ac; +int node =numa_node_id ();check_irq_off (); ac =ac_data (cachep); -spin_lock (&cachep->nodelists[numa_node_id ()]->list_lock); -free_block (cachep, ac->entry, ac->avail); -spin_unlock (&cachep->nodelists[numa_node_id ()]->list_lock); +spin_lock (&cachep->nodelists[node]->list_lock); +free_block (cachep, ac->entry, ac->avail, node); +spin_unlock (&cachep->nodelists[node]->list_lock); ac->avail =0 ; }
@@ -2608,7 +2609,7 @@ done:/* * Caller needs to acquire correct kmem_list's list_lock */ -static void free_block (kmem_cache_t *cachep,void **objpp,int nr_objects) +static void free_block (kmem_cache_t *cachep,void **objpp,int nr_objects,int node) {int i;struct kmem_list3 *l3;
@@ -2617,14 +2618,12 @@ static void free_block(kmem_cache_t *cacvoid *objp = objpp[i];struct slab *slabp;unsigned int objnr; -int nodeid =0 ; slabp =GET_PAGE_SLAB (virt_to_page (objp)); - nodeid = slabp->nodeid; - l3 = cachep->nodelists[nodeid]; + l3 = cachep->nodelists[node];list_del (&slabp->list); objnr = (objp - slabp->s_mem) / cachep->objsize; -check_spinlock_acquired_node (cachep, nodeid); +check_spinlock_acquired_node (cachep, node);check_slabp (cachep, slabp);
@@ -2664,13 +2663,14 @@ static void cache_flusharray(kmem_cache_ {int batchcount;struct kmem_list3 *l3; +int node =numa_node_id (); batchcount = ac->batchcount;#if DEBUG BUG_ON (!batchcount || batchcount > ac->avail);#endif check_irq_off (); - l3 = cachep->nodelists[numa_node_id ()]; + l3 = cachep->nodelists[node];spin_lock (&l3->list_lock);if (l3->shared) {struct array_cache *shared_array = l3->shared;
@@ -2686,7 +2686,7 @@ static void cache_flusharray(kmem_cache_ } } -free_block (cachep, ac->entry, batchcount); +free_block (cachep, ac->entry, batchcount, node); free_done:#if STATS {
@@ -2751,7 +2751,7 @@ static inline void __cache_free(kmem_cac }else {spin_lock (&(cachep->nodelists[nodeid])-> list_lock); -free_block (cachep, &objp,1 ); +free_block (cachep, &objp,1 , nodeid);spin_unlock (&(cachep->nodelists[nodeid])-> list_lock); }
@@ -2844,7 +2844,7 @@ void *kmem_cache_alloc_node(kmem_cache_tunsigned long save_flags;void *ptr; -if (nodeid ==numa_node_id () || nodeid ==-1 ) +if (nodeid ==-1 )return __cache_alloc (cachep, flags);if (unlikely (!cachep->nodelists[nodeid])) {
@@ -3079,7 +3079,7 @@ static int alloc_kmemlist(kmem_cache_t *if ((nc = cachep->nodelists[node]->shared))free_block (cachep, nc->entry, - nc->avail); + nc->avail, node); l3->shared = new;if (!cachep->nodelists[node]->alien) {
@@ -3160,7 +3160,7 @@ static int do_tune_cpucache(kmem_cache_tif (!ccold)continue ;spin_lock_irq (&cachep->nodelists[cpu_to_node (i)]->list_lock); -free_block (cachep, ccold->entry, ccold->avail); +free_block (cachep, ccold->entry, ccold->avail,cpu_to_node (i));spin_unlock_irq (&cachep->nodelists[cpu_to_node (i)]->list_lock);kfree (ccold); }
@@ -3240,7 +3240,7 @@ static void drain_array_locked(kmem_cachif (tofree > ac->avail) { tofree = (ac->avail+1 )/2 ; } -free_block (cachep, ac->entry, tofree); +free_block (cachep, ac->entry, tofree, node); ac->avail -= tofree;memmove (ac->entry, &(ac->entry[tofree]),sizeof (void *)*ac->avail);
Quoted 11 lines · Christoph Lameter
Quoted 3 lines · Petr Vandrovec
slab belonging to node#1, while having acquired lock for cachep belonging to node #0. Due to this
check_spinlock_acquired_node(cachep, nodeid)fails (check_spinlock_acquired_node(cachep, 0)would succeed).Hmmm. If a node runs out of memory then pages from another node may end up on the slab list of a node. But it seems that
free_blockcannot handle that properly.How are you producing the problem?
Simple... I just boot any kernel after 2.6.13, and it dies in front of me. Currently I'm using config below, which I boot with 'rootdelay=60' so panic in keventd happens before panic due to no root filesystem. No ACPI. Nothing. 100% reproducible. Maybe I should enable embedded options and remove all other device drivers still present in the kernel.
Below config is dmesg from 2.6.13, which has no problems with comming up. Maybe you'll find some clue there, but I see none. Node #0 has 1GB of memory, so it should have no need to borrow blocks from node #1 when this kernel is able to boot in 16MB of memory...
Petr
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.14-rc1-6c07
# Wed Sep 21 03:03:20 2005
#CONFIG_X86_64=yCONFIG_64BIT=yCONFIG_X86=yCONFIG_SEMAPHORE_SLEEPERS=yCONFIG_MMU=yCONFIG_RWSEM_GENERIC_SPINLOCK=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_X86_CMPXCHG=yCONFIG_EARLY_PRINTK=yCONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_IOMAP=yCONFIG_ARCH_MAY_HAVE_PC_FDC=y
#
# Code maturity level options
#
# CONFIG_EXPERIMENTAL is not setCONFIG_CLEAN_COMPILE=yCONFIG_LOCK_KERNEL=yCONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_SHOW_LOGO is not set
#
# General setup
#CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
# CONFIG_SWAP is not set
# CONFIG_SYSVIPC is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_SYSCTL is not set
# CONFIG_HOTPLUG is not set
# CONFIG_IKCONFIG is not set
# CONFIG_CPUSETS is not setCONFIG_INITRAMFS_SOURCE=""
# CONFIG_EMBEDDED is not setCONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not setCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_SHMEM=yCONFIG_CC_ALIGN_FUNCTIONS=0CONFIG_CC_ALIGN_LABELS=0CONFIG_CC_ALIGN_LOOPS=0CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not setCONFIG_BASE_SMALL=0
#
# Loadable module support
#
# CONFIG_MODULES is not set
#
# Processor type and features
#CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_GENERIC_CPU is not setCONFIG_X86_L1_CACHE_BYTES=64CONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_TSC=yCONFIG_X86_GOOD_APIC=y
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not setCONFIG_X86_IO_APIC=yCONFIG_X86_LOCAL_APIC=y
# CONFIG_MTRR is not setCONFIG_SMP=y
# CONFIG_SCHED_SMT is not setCONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_BKL is not setCONFIG_K8_NUMA=y
# CONFIG_NUMA_EMU is not setCONFIG_ARCH_DISCONTIGMEM_ENABLE=yCONFIG_NUMA=yCONFIG_ARCH_DISCONTIGMEM_DEFAULT=yCONFIG_ARCH_SPARSEMEM_ENABLE=yCONFIG_DISCONTIGMEM=yCONFIG_FLAT_NODE_MEM_MAP=yCONFIG_NEED_MULTIPLE_NODES=y
# CONFIG_SPARSEMEM_STATIC is not setCONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=yCONFIG_NR_CPUS=8CONFIG_HPET_TIMER=yCONFIG_DUMMY_IOMMU=yCONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not setCONFIG_PHYSICAL_START=0x100000
# CONFIG_SECCOMP is not setCONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_1000 is not setCONFIG_HZ=100CONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_ISA_DMA_API=yCONFIG_GENERIC_PENDING_IRQ=y
#
# Power management options
#
# CONFIG_PM is not set
#
# ACPI (Advanced Configuration and Power Interface) Support #
# CONFIG_ACPI is not set
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
#
# Bus options (PCI etc.)
#
# CONFIG_PCI is not set
#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set
#
# PCI Hotplug Support
#
#
# Executable file formats / Emulations
#
# CONFIG_BINFMT_ELF is not set
# CONFIG_BINFMT_MISC is not set
# CONFIG_IA32_EMULATION is not set
#
# Networking
#
# CONFIG_NET is not set
#
# Device Drivers
#
#
# Generic Driver Options
#CONFIG_STANDALONE=y
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
# CONFIG_FW_LOADER is not set
# CONFIG_DEBUG_DRIVER is not set
#
# Connector - unified userspace <-> kernelspace linker #
#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set
#
# Parallel port support
#
# CONFIG_PARPORT is not set
#
# Plug and Play support
#
#
# Block devices
#
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_RAM is not setCONFIG_BLK_DEV_RAM_COUNT=16
# CONFIG_LBD is not set
# CONFIG_CDROM_PKTCDVD is not set
#
# IO Schedulers
#CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
#
# ATA/ATAPI/MFM/RLL support
#
# CONFIG_IDE is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
# CONFIG_SCSI is not set
#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set
#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
#
# I2O device support
#
#
# Network device support
#
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
#
# ISDN subsystem
#
#
# Telephony Support
#
# CONFIG_PHONE is not set
#
# Input device support
#CONFIG_INPUT=y
#
# Userland interfaces
#CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not setCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#CONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#CONFIG_SERIO=yCONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not setCONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#CONFIG_VT=yCONFIG_VT_CONSOLE=y
# CONFIG_CONSOLE_KNOWS_9B is not setCONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
#
# Non-8250 serial port support
#CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set
#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_NVRAM is not set
# CONFIG_RTC is not set
# CONFIG_GEN_RTC is not set
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
#
# Ftape, the floppy tape device driver
#
# CONFIG_AGP is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HANGCHECK_TIMER is not set
#
# TPM devices
#
#
# I2C support
#
# CONFIG_I2C is not set
#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set
#
# Hardware Monitoring support
#
# CONFIG_HWMON is not set
# CONFIG_HWMON_VID is not set
#
# Misc devices
#
#
# Multimedia Capabilities Port drivers
#
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
#
# Digital Video Broadcasting Devices
#
#
# Graphics support
#
# CONFIG_FB is not set
# CONFIG_VIDEO_SELECT is not set
#
# Console display driver support
#CONFIG_VGA_CONSOLE=yCONFIG_DUMMY_CONSOLE=y
#
# Sound
#
# CONFIG_SOUND is not set
#
# USB support
#
# CONFIG_USB_ARCH_HAS_HCD is not set
# CONFIG_USB_ARCH_HAS_OHCI is not set
#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set
#
# MMC/SD Card support
#
# CONFIG_MMC is not set
#
# InfiniBand support
#
#
# SN Devices
#
#
# Firmware Drivers
#
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
#
# File systems
#
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
# CONFIG_JBD is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_INOTIFY is not set
# CONFIG_QUOTA is not setCONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#CONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_SYSFS=y
# CONFIG_TMPFS is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not setCONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set
#
# Miscellaneous filesystems
#
# CONFIG_HFSPLUS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not setCONFIG_MSDOS_PARTITION=y
#
# Native Language Support
#
# CONFIG_NLS is not set
#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not setCONFIG_DEBUG_KERNEL=yCONFIG_MAGIC_SYSRQ=yCONFIG_LOG_BUF_SHIFT=17CONFIG_DETECT_SOFTLOCKUP=yCONFIG_SCHEDSTATS=yCONFIG_DEBUG_SLAB=yCONFIG_DEBUG_SPINLOCK=yCONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_KOBJECT is not setCONFIG_DEBUG_INFO=yCONFIG_DEBUG_FS=y
# CONFIG_FRAME_POINTER is not setCONFIG_INIT_DEBUG=y
# CONFIG_KPROBES is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
#
# Cryptographic options
#
# CONFIG_CRYPTO is not set
#
# Hardware crypto devices
#
#
# Library routines
#
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not setCONFIG_CRC32=y
# CONFIG_LIBCRC32C is not set
Bootdata ok (command line is BOOT_IMAGE=2.6.13-64 ro root=801 ramdisk=0 console=ttyS0,115200 console=tty0 nmi_watchdog=2 psmouse_noext=1 verbose) Linux version 2.6.13 (root@vana) (gcc version 3.3.3 (Debian 20040401)) #2 SMP Tue Aug 30 02:41:20 CEST 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
BIOS-e820: 000000007fff0000 - 000000007ffff000 (ACPI data)
BIOS-e820: 000000007ffff000 - 0000000080000000 (ACPI NVS)
BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
ACPI: RSDP (v002 ACPIAM ) @ 0x00000000000f6e10
ACPI: XSDT (v001 A M I OEMXSDT 0x06000514 MSFT 0x00000097) @ 0x000000007fff0100
ACPI: FADT (v001 A M I OEMFACP 0x06000514 MSFT 0x00000097) @ 0x000000007fff0281
ACPI: MADT (v001 A M I OEMAPIC 0x06000514 MSFT 0x00000097) @ 0x000000007fff0380
ACPI: OEMB (v001 A M I OEMBIOS 0x06000514 MSFT 0x00000097) @ 0x000000007ffff040
ACPI: SRAT (v001 A M I OEMSRAT 0x06000514 MSFT 0x00000097) @ 0x000000007fff4260
ACPI: HPET (v001 A M I OEMHPET 0x06000514 MSFT 0x00000097) @ 0x000000007fff4370
ACPI: ASF! (v001 AMIASF AMDSTRET 0x00000001 INTL 0x02002026) @ 0x000000007fff43b0
ACPI: DSDT (v001 0AAAA 0AAAA001 0x00000001 INTL 0x02002026) @ 0x0000000000000000
SRAT: PXM 0 -> APIC 0 -> CPU 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> CPU 1 -> Node 1
SRAT: Node 0 PXM 0 100000-3fffffff
SRAT: Node 1 PXM 1 40000000-7fffffff
SRAT: Node 0 PXM 0 0-3fffffff
Using 24 for the hash shift. Max adder is 7fffffff
Bootmem setup node 0 0000000000000000-000000003fffffff
Bootmem setup node 1 0000000040000000-000000007ffeffff
On node 0 totalpages: 262046
DMA zone: 3999 pages, LIFO batch:1
Normal zone: 258047 pages, LIFO batch:31
HighMem zone: 0 pages, LIFO batch:1
On node 1 totalpages: 262127
DMA zone: 0 pages, LIFO batch:1
Normal zone: 262127 pages, LIFO batch:31
HighMem zone: 0 pages, LIFO batch:1
ACPI: PM-Timer IO Port: 0x5008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x03] address[0xff4ff000] gsi_base[24])
IOAPIC[1]: apic_id 3, version 17, address 0xff4ff000, GSI 24-27
ACPI: IOAPIC (id[0x04] address[0xff4fe000] gsi_base[28])
IOAPIC[2]: apic_id 4, version 17, address 0xff4fe000, GSI 28-31
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
ACPI: HPET id: 0x102282a0 base: 0xfec01000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 80000000 (gap: 80000000:7f780000)
Checking aperture...
CPU 0: aperture @ c0000000 size 512 MB
CPU 1: aperture @ c0000000 size 512 MB
Built 2 zonelists
Kernel command line: BOOT_IMAGE=2.6.13-64 ro root=801 ramdisk=0 console=ttyS0,115200 console=tty0 nmi_watchdog=2 psmouse_noext=1 verbose Parameter psmouse_noext is obsolete, ignored
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)time.c: Using 14.318180 MHz HPET timer.time.c: Detected 1991.621 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Memory: 2055660k/2097088k available (2668k kernel code, 0k reserved, 2318k data, 232k init)
Calibrating delay using timer specific routine.. 3988.04 BogoMIPS (lpj=19940224)
Security Framework v1.0.0 initialized
SELinux: Initializing. SELinux: Starting in permissive mode Mount-cache hash table entries: 256 CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(1) -> Node 0 -> Core 0
mtrr: v2.0 (20020519)
Using local APIC timer interrupts.
Detected 12.447 MHz APIC timer.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3983.13 BogoMIPS (lpj=19915667)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1(1) -> Node 1 -> Core 0
AMD Opteron(tm) Processor 246 stepping 0a
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff -136 cycles, maxerr 901 cycles) Brought up 2 CPUstime.c: Using HPET based timekeeping.
testing NMI watchdog ... OK.
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20050408
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] segment is 0
ACPI: Assume root bridge [\_SB_.PCIB] segment is 0
PCI: Scanning bus 0000:00
PCI: Found 0000:00:06.0 [1022/7460] 000604 01
PCI: Found 0000:00:07.0 [1022/7468] 000601 00
PCI: Found 0000:00:07.1 [1022/7469] 000101 00
PCI: Found 0000:00:07.2 [1022/746a] 000c05 00
PCI: Found 0000:00:07.3 [1022/746b] 000680 00
PCI: Found 0000:00:07.5 [1022/746d] 000401 00
PCI: Found 0000:00:0a.0 [1022/7450] 000604 01
PCI: Found 0000:00:0a.1 [1022/7451] 000800 00
PCI: Found 0000:00:0b.0 [1022/7450] 000604 01
PCI: Found 0000:00:0b.1 [1022/7451] 000800 00
PCI: Found 0000:00:18.0 [1022/1100] 000600 00
PCI: Found 0000:00:18.1 [1022/1101] 000600 00
PCI: Found 0000:00:18.2 [1022/1102] 000600 00
PCI: Found 0000:00:18.3 [1022/1103] 000600 00
PCI: Found 0000:00:19.0 [1022/1100] 000600 00
PCI: Found 0000:00:19.1 [1022/1101] 000600 00
PCI: Found 0000:00:19.2 [1022/1102] 000600 00
PCI: Found 0000:00:19.3 [1022/1103] 000600 00
PCI: Fixups for bus 0000:00
PCI: Scanning behind PCI bridge 0000:00:06.0, config 010100, pass 0
PCI: Scanning bus 0000:01
PCI: Found 0000:01:00.0 [1022/7464] 000c03 00
PCI: Found 0000:01:00.1 [1022/7464] 000c03 00
PCI: Found 0000:01:0b.0 [1095/3114] 000180 00
PCI: Found 0000:01:0c.0 [104c/8023] 000c00 00
PCI: Fixups for bus 0000:01
PCI: Bus scan for 0000:01 returning with max=01
PCI: Scanning behind PCI bridge 0000:00:0a.0, config 020200, pass 0
PCI: Scanning bus 0000:02
PCI: Found 0000:02:07.0 [1131/7146] 000480 00
PCI: Found 0000:02:09.0 [14e4/16a7] 000200 00
PCI: Fixups for bus 0000:02
PCI: Bus scan for 0000:02 returning with max=02
PCI: Scanning behind PCI bridge 0000:00:0b.0, config 030300, pass 0
PCI: Scanning bus 0000:03
PCI: Fixups for bus 0000:03
PCI: Bus scan for 0000:03 returning with max=03
PCI: Scanning behind PCI bridge 0000:00:06.0, config 010100, pass 1
PCI: Scanning behind PCI bridge 0000:00:0a.0, config 020200, pass 1
PCI: Scanning behind PCI bridge 0000:00:0b.0, config 030300, pass 1
PCI: Bus scan for 0000:00 returning with max=03
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.GOLA._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.GOLB._PRT]
ACPI: PCI Root Bridge [PCIB] (0000:04)
PCI: Probing PCI hardware (bus 04)
ACPI: Assume root bridge [\_SB_.PCI0] segment is 0
ACPI: Assume root bridge [\_SB_.PCIB] segment is 0
PCI: Scanning bus 0000:04
PCI: Found 0000:04:00.0 [1022/7454] 000600 00
PCI: Found 0000:04:01.0 [1022/7455] 000604 01
PCI: Fixups for bus 0000:04
PCI: Scanning behind PCI bridge 0000:04:01.0, config 050504, pass 0
PCI: Scanning bus 0000:05
PCI: Found 0000:05:00.0 [1002/5964] 000300 00
Boot video device is 0000:05:00.0
PCI: Found 0000:05:00.1 [1002/5d44] 000380 00
PCI: Fixups for bus 0000:05
PCI: Bus scan for 0000:05 returning with max=05
PCI: Scanning behind PCI bridge 0000:04:01.0, config 050504, pass 1
PCI: Bus scan for 0000:04 returning with max=05
ACPI: PCI Interrupt Routing Table [\_SB_.PCIB.PBP2._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 7 9 10 11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
got res [80000000:8007ffff] bus [80000000:8007ffff] flags 7200 for BAR 6 of 0000:01:0b.0
PCI: Bridge: 0000:00:06.0
IO window: 9000-afff
MEM window: ff100000-ff2fffff
PREFETCH window: 80000000-800fffff
got res [9e900000:9e90ffff] bus [9e900000:9e90ffff] flags 7200 for BAR 6 of 0000:02:09.0
PCI: Bridge: 0000:00:0a.0
IO window: disabled.
MEM window: ff300000-ff3fffff
PREFETCH window: 9e900000-9e9fffff
PCI: Bridge: 0000:00:0b.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
got res [9eb00000:9eb1ffff] bus [9eb00000:9eb1ffff] flags 7202 for BAR 6 of 0000:05:00.0
PCI: Bridge: 0000:04:01.0
IO window: c000-cfff
MEM window: ff500000-ff5fffff
PREFETCH window: 9eb00000-beafffff
TC classifier action (bugs to [email protected] cc [email protected])
hpet0: at MMIO 0xfec01000, IRQs 2, 8, 0
hpet0: 69ns tick, 3 32-bit timers
agpgart: Detected AMD 8151 AGP Bridge rev B3
agpgart: AGP aperture is 512M @ 0xc0000000
PCI-DMA: Disabling IOMMU.
pnp: 00:09: ioport range 0x680-0x6ff has been reserved
pnp: 00:09: ioport range 0x295-0x296 has been reserved
pnp: 00:09: ioport range 0xb78-0xb7f has been reserved
pnp: 00:09: ioport range 0xf78-0xf7f has been reserved
IA-32 Microcode Update Driver: v1.14 <[email protected]>
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1127264758.390:1): initialized
Total HugeTLB memory allocated, 0
SELinux: Registering netfilter hooks
Initializing Cryptographic API
PCI: MSI quirk detected. pci_msi_quirk set.
PCI: MSI quirk detected. pci_msi_quirk set.
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 16 (level, low) -> IRQ 169
radeonfb: Found Intel x86 BIOS ROM Image
radeonfb: Retreived PLL infos from BIOS
radeonfb: Reference=27.00 MHz (RefDiv=12) Memory=200.00 Mhz, System=166.00 MHz
radeonfb: PLL min 20000 max 40000
radeonfb: Monitor 1 type DFP found
radeonfb: EDID probed
radeonfb: Monitor 2 type no found
Console: switching to colour frame buffer device 240x75
radeonfb (0000:05:00.0): ATI Radeon Yd
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
Using specific hotkey driver
ACPI: CPU0 (power states: C1[C1])
ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: CPU1 (power states: C1[C1])
Real Time Clock Driver v1.12hpet_acpi_add: no address or irqs in _CRS
Linux agpgart interface v0.101 (c) Dave Jones
[drm] Initialized drm 1.0.0 20040925
PNP: PS/2 controller doesn't have AUX irq; using default 0xc
PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 112
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 0) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA] io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pktcdvd: v0.2.0a 2004-07-14 Jens Axboe ([email protected]) and [email protected] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controller
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pioProbing IDE interface ide0...
hda: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: WDC WD1200JB-00CRA0, ATA DISK drive
ide1 at 0x170-0x177,0x376 on irq 15
hdc: max request size: 128KiB
hdc: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100)
hdc: cache flushes not supported
hdc: hdc1
libata version 1.12 loaded.sata_sil version 0.9
ACPI: PCI Interrupt 0000:01:0b.0[A] -> GSI 17 (level, low) -> IRQ 177
ata1: SATA max UDMA/100 cmd 0xFFFFC20000004C80 ctl 0xFFFFC20000004C8A bmdma 0xFFFFC20000004C00 irq 177
ata2: SATA max UDMA/100 cmd 0xFFFFC20000004CC0 ctl 0xFFFFC20000004CCA bmdma 0xFFFFC20000004C08 irq 177
ata3: SATA max UDMA/100 cmd 0xFFFFC20000004E80 ctl 0xFFFFC20000004E8A bmdma 0xFFFFC20000004E00 irq 177
ata4: SATA max UDMA/100 cmd 0xFFFFC20000004EC0 ctl 0xFFFFC20000004ECA bmdma 0xFFFFC20000004E08 irq 177
ata1: dev 0 cfg 49:2f00 82:74eb 83:7feb 84:4123 85:74e8 86:3c03 87:4123 88:207f
ata1: dev 0 ATA, max UDMA/133, 781422768 sectors: lba48
ata1: dev 0 configured for UDMA/100
scsi0 : sata_sil
ata2: no device found (phy stat 00000000)
scsi1 : sata_sil
ata3: no device found (phy stat 00000000)
scsi2 : sata_sil
ata4: no device found (phy stat 00000000)
scsi3 : sata_sil
Vendor: ATA Model: HDS724040KLSA80 Rev: KFAO
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Fusion MPT base driver 3.03.02
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SPI Host driver 3.03.02
Fusion MPT misc device (ioctl) driver 3.03.02
mptctl: Registered with Fusion MPT base driver
mptctl: /dev/mptctl @ (major,minor=10,220)
aoe: aoe_init: AoE v2.6-10 initialised.
mice: PS/2 mouse device common for all mice
input: PC Speaker
i2c /dev entries driver
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 131072 (order: 9, 3145728 bytes)
TCP bind hash table entries: 65536 (order: 8, 1572864 bytes)
input: AT Translated Set 2 keyboard on isa0060/serio0
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
NET: Registered protocol family 17
BIOS EDD facility v0.16 2004-Jun-25, 2 devices found
Found 512b device! Using larger block size...
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 232k freed
kjournald starting. Commit interval 5 seconds
Found 512b device! Using larger block size...
Found 512b device! Using larger block size...
Adding 1959916k swap on /dev/sda2. Priority:-1 extents:1
EXT3 FS on sda1, internal journal
i2c_adapter i2c-6: Detecting device at 6,0x2e with COMPANY: 0x41 and VERSTEP: 0x62 i2c_adapter i2c-6: Autodetecting device at 6,0x2e ... lm85 6-002e: Initializing device lm85 6-002e: LM85_REG_CONFIG is: 0x05 lm85 6-002e: Setting CONFIG to: 0x05 powernow-k8: Found 2 AMD Athlon 64 / Opteron processors (version 1.50.3) powernow-k8: 0 : fid 0xc (2000 MHz), vid 0x2 (1500 mV) powernow-k8: 1 : fid 0xa (1800 MHz), vid 0x6 (1400 mV) powernow-k8: 2 : fid 0x2 (1000 MHz), vid 0xe (1200 mV) cpu_init done, current fid 0xc, vid 0x2 powernow-k8: 0 : fid 0xc (2000 MHz), vid 0x2 (1500 mV) powernow-k8: 1 : fid 0xa (1800 MHz), vid 0x6 (1400 mV) powernow-k8: 2 : fid 0x2 (1000 MHz), vid 0xe (1200 mV)
cpu_init done, current fid 0xc, vid 0x2
tg3.c:v3.37 (August 25, 2005)
ACPI: PCI Interrupt 0000:02:09.0[A] -> GSI 24 (level, low) -> IRQ 185
eth0: Tigon3 [partno(BCM95703A30) rev 1002 PHY(5703)] (PCI:33MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:2c:90:0a
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1]
eth0: dma_rwctrl[763f0000]
Found 512b device! Using larger block size...
Found 512b device! Using larger block size...
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Found 512b device! Using larger block size...
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda4, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Intel 810 + AC97 Audio, version 1.01, 19:40:55 Aug 29 2005
ACPI: PCI Interrupt 0000:00:07.5[B] -> GSI 17 (level, low) -> IRQ 177
i810: AMD-8111 IOHub found at IO 0xbc00 and 0xb800, MEM 0x0000 and 0x0000, IRQ 177i810_audio: Audio Controller supports 6 channels.i810_audio: Defaulting to base 2 channel mode.i810_audio: Resetting connection 0ac97_codec: AC97 Audio codec, id: ADS116 (Analog Devices AD1981B)i810_audio: AC'97 codec 0 supports AMAP, total channels = 2
usbcore: registered new driver usbfs
usbcore: registered new driver hubohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ACPI: PCI Interrupt 0000:01:00.0[D] -> GSI 19 (level, low) -> IRQ 193ohci_hcd 0000:01:00.0: Advanced Micro Devices [AMD] AMD-8111 USBohci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1ohci_hcd 0000:01:00.0: irq 193, io mem 0xff2fd000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ACPI: PCI Interrupt 0000:01:00.1[D] -> GSI 19 (level, low) -> IRQ 193ohci_hcd 0000:01:00.1: Advanced Micro Devices [AMD] AMD-8111 USB (#2)ohci_hcd 0000:01:00.1: new USB bus registered, assigned bus number 2ohci_hcd 0000:01:00.1: irq 193, io mem 0xff2fe000
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
ieee1394: Initialized config rom entry `ip1394'
ohci1394: $Rev: 1299 $ Ben Collins <[email protected]>
ACPI: PCI Interrupt 0000:01:0c.0[A] -> GSI 19 (level, low) -> IRQ 193
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[193] MMIO=[ff2ff000-ff2ff7ff] Max Packet=[2048]
ieee1394: Host added: ID:BUS[0-00:1023] GUID[00e0810000303bdf]
lm85 6-002e: Reading sensor values
eth1394: $Rev: 1264 $ Ben Collins <[email protected]>
eth1394: eth1: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
lm85 6-002e: Reading config values
NET: Registered protocol family 15pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shpchp: HPC vendor_id 1022 device_id 7460 ss_vid 0 ss_did 0
shpchp: shpc_init: cannot reserve MMIO region
shpchp: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
shpchp: shpc_init: cannot reserve MMIO region
shpchp: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
shpchp: shpc_init: cannot reserve MMIO region
shpchp: HPC vendor_id 1022 device_id 7455 ss_vid 0 ss_did 0
shpchp: shpc_init: cannot reserve MMIO region
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4hw_random: AMD768 system management I/O registers at 0x5000.hw_random hardware driver 1.0.0 loaded
saa7146: register extension 'budget_ci dvb'.
ACPI: PCI Interrupt 0000:02:07.0[A] -> GSI 26 (level, low) -> IRQ 201
saa7146: found saa7146 @ mem ffffc20000198c00 (revision 1, irq 201) (0x13c2,0x1011).
DVB: registering new adapter (TT-Budget/WinTV-NOVA-T PCI).
adapter has MAC addr = 00:d0:5c:03:23:34
DVB: registering frontend 0 (Philips TDA10045H DVB-T)...
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
hda: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
NET: Registered protocol family 10
Disabled Privacy Extensions on device ffffffff8052efc0(lo)
IPv6 over IPv4 tunneling driver
WDT driver for the Winbond(TM) W83627HF Super I/O chip initialising.
w83627hf WDT: initialized. timeout=60 sec (nowayout=0)selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Installing knfsd (copyright (C) 1996 [email protected]).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
crc32c: Unknown symbol crc32c_le
Initializing IPsec netlink socket
ipcomp6: Unknown symbol xfrm6_tunnel_free_spi
ipcomp6: Unknown symbol xfrm6_tunnel_alloc_spi
ipcomp6: Unknown symbol xfrm6_tunnel_spi_lookup
NET: Registered protocol family 4ioctl32(ipx_configure:6768): Unknown cmd fd(3) cmd(000089e1){00} arg(ffffd85b) on socket:[13557] Process accounting paused
lm85 6-002e: Reading sensor values
Quoted 11 lines · Petr Vandrovec
Simple... I just boot any kernel after 2.6.13, and it dies in front of me. Currently I'm using config below, which I boot with 'rootdelay=60' so panic in keventd happens before panic due to no root filesystem. No ACPI. Nothing. 100% reproducible. Maybe I should enable embedded options and remove all other device drivers still present in the kernel.
Below config is dmesg from 2.6.13, which has no problems with comming up. Maybe
you'll find some clue there, but I see none. Node #0 has 1GB of memory, so it should have no need to borrow blocks from node #1 when this kernel is able to boot in 16MB of memory...
Hmm. This likely has something to do with debugging code. I was unable to reproduce this on amd64 with your config. I get another failure with 2.6.14-rc2 on ia64 if I enable all the debugging features that you have. The system works fine if no debugging is configured:
kernel BUG at kernel/workqueue.c:541!
swapper[1]: bugcheck! 0 [1]
Modules linked in:
Pid: 1, CPU 0, comm: swapper
psr : 00001010085a6010 ifs : 8000000000000105 ip : [<a0000001000e5b10>] Not tainted
ip is at init_workqueues+0x90/0xa0
unat: 0000000000000000 pfs : 0000000000000105 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 000000000001003e pr : 000000000000a541
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f csd : 0000000000000000 ssd : 0000000000000000
b0 : a0000001000e5b10 b6 : e000003002385ad0 b7 : e000000001fffc00 f6 : 1003e000000003f34da65 f7 : 1003e20c49ba5e353f7cf
f8 : 1003e00000000000000c8 f9 : 10006c7fffffffd73ea5c
f10 : 0fffdfffffffffaf00000 f11 : 1003e0000000000000000
r1 : a000000100cd7670 r2 : 0000000000000001 r3 : e00000b0057c0e00 r8 : 0000000000000029 r9 : 0000000000004000 r10 : 0000000000000001 r11 : 0000000000000002 r12 : e00000b0057c7de0 r13 : e00000b0057c0000 r14 : a0000001009690b0 r15 : e00000b0057c0df4 r16 : e00000b0057c0e00 r17 : 0000000000000001 r18 : 0000000000000002 r19 : a0000001009690b0 r20 : a000000100ad78f8 r21 : ffffffffffffffff r22 : e000000001fffc00 r23 : a000000100b11748 r24 : 0000000000000000 r25 : 0000000000000004 r26 : e00000b0057c0df0 r27 : e00000b0057c7cc8 r28 : e00000b0057c7cd0 r29 : 0000000000000c46 r30 : 0000000000000c46 r31 : 0000000000000308
Quoted 7 lines · Christoph Lameter
Hmm. This likely has something to do with debugging code. I was unable to reproduce this on amd64 with your config. I get another failure with 2.6.14-rc2 on ia64 if I enable all the debugging features that you have. The system works fine if no debugging is configured:
kernel BUG at
kernel/workqueue.c:541!
swapper[1]: bugcheck! 0 [1]
I fixed the above issue (a structure became larger than the maximum allowed by the slab allocator) and the kernel boots fine now on an 8 way ia64. Cannot reproduce the problem.
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.14-rc2-git1
# Thu Sep 22 10:31:35 2005
#
#
# Code maturity level options
#CONFIG_EXPERIMENTAL=yCONFIG_CLEAN_COMPILE=yCONFIG_LOCK_KERNEL=yCONFIG_INIT_ENV_ARG_LIMIT=32
#
# General setup
#CONFIG_LOCALVERSION=""CONFIG_LOCALVERSION_AUTO=yCONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not setCONFIG_SYSCTL=y
# CONFIG_AUDIT is not setCONFIG_HOTPLUG=yCONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not setCONFIG_CPUSETS=yCONFIG_INITRAMFS_SOURCE=""
# CONFIG_EMBEDDED is not setCONFIG_KALLSYMS=yCONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not setCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_SHMEM=yCONFIG_CC_ALIGN_FUNCTIONS=0CONFIG_CC_ALIGN_LABELS=0CONFIG_CC_ALIGN_LOOPS=0CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not setCONFIG_BASE_SMALL=0
#
# Loadable module support
#CONFIG_MODULES=yCONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not setCONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not setCONFIG_KMOD=yCONFIG_STOP_MACHINE=y
#
# Processor type and features
#CONFIG_IA64=yCONFIG_64BIT=yCONFIG_MMU=yCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_TIME_INTERPOLATION=yCONFIG_EFI=yCONFIG_GENERIC_IOMAP=yCONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=yCONFIG_IA64_UNCACHED_ALLOCATOR=y
# CONFIG_IA64_GENERIC is not set
# CONFIG_IA64_DIG is not set
# CONFIG_IA64_HP_ZX1 is not set
# CONFIG_IA64_HP_ZX1_SWIOTLB is not setCONFIG_IA64_SGI_SN2=y
# CONFIG_IA64_HP_SIM is not set
# CONFIG_ITANIUM is not setCONFIG_MCKINLEY=y
# CONFIG_IA64_PAGE_SIZE_4KB is not set
# CONFIG_IA64_PAGE_SIZE_8KB is not setCONFIG_IA64_PAGE_SIZE_16KB=y
# CONFIG_IA64_PAGE_SIZE_64KB is not set
# CONFIG_HZ_100 is not setCONFIG_HZ_250=y
# CONFIG_HZ_1000 is not setCONFIG_HZ=250CONFIG_IA64_L1_CACHE_SHIFT=7CONFIG_NUMA=yCONFIG_VIRTUAL_MEM_MAP=yCONFIG_HOLES_IN_ZONE=yCONFIG_ARCH_DISCONTIGMEM_ENABLE=y
# CONFIG_IA64_CYCLONE is not setCONFIG_IOSAPIC=yCONFIG_IA64_SGI_SN_XP=mCONFIG_FORCE_MAX_ZONEORDER=18CONFIG_SMP=yCONFIG_NR_CPUS=512
# CONFIG_HOTPLUG_CPU is not setCONFIG_SCHED_SMT=yCONFIG_PREEMPT=yCONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not setCONFIG_DISCONTIGMEM_MANUAL=y
# CONFIG_SPARSEMEM_MANUAL is not setCONFIG_DISCONTIGMEM=yCONFIG_FLAT_NODE_MEM_MAP=yCONFIG_NEED_MULTIPLE_NODES=y
# CONFIG_SPARSEMEM_STATIC is not setCONFIG_IA32_SUPPORT=yCONFIG_COMPAT=yCONFIG_IA64_MCA_RECOVERY=yCONFIG_PERFMON=yCONFIG_IA64_PALINFO=y
#
# Firmware Drivers
#CONFIG_EFI_VARS=yCONFIG_EFI_PCDP=y
# CONFIG_DELL_RBU is not setCONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
#
# Power management and ACPI
#CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
#
# ACPI (Advanced Configuration and Power Interface) Support #CONFIG_ACPI=y
# CONFIG_ACPI_BUTTON is not set
# CONFIG_ACPI_FAN is not set
# CONFIG_ACPI_PROCESSOR is not set CONFIG_ACPI_NUMA=yCONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not setCONFIG_ACPI_POWER=yCONFIG_ACPI_SYSTEM=y
# CONFIG_ACPI_CONTAINER is not set
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
#
# Bus options (PCI, PCMCIA)
#CONFIG_PCI=yCONFIG_PCI_DOMAINS=y
# CONFIG_PCI_MSI is not setCONFIG_PCI_LEGACY_PROC=y
# CONFIG_PCI_DEBUG is not set
#
# PCI Hotplug Support
#CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_FAKE is not set
# CONFIG_HOTPLUG_PCI_ACPI is not set
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not setCONFIG_HOTPLUG_PCI_SGI=y
#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set
#
# Networking
#CONFIG_NET=y
#
# Networking options
#CONFIG_PACKET=yCONFIG_PACKET_MMAP=yCONFIG_UNIX=y
# CONFIG_NET_KEY is not setCONFIG_INET=yCONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not setCONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not setCONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not setCONFIG_INET_DIAG=yCONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not setCONFIG_TCP_CONG_BIC=yCONFIG_IPV6=m
# CONFIG_IPV6_PRIVACY is not set
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_NETFILTER is not set
#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set
#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set
#
# Device Drivers
#
#
# Generic Driver Options
#CONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set
#
# Connector - unified userspace <-> kernelspace linker #
# CONFIG_CONNECTOR is not set
#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set
#
# Parallel port support
#
# CONFIG_PARPORT is not set
#
# Plug and Play support
#
# CONFIG_PNP is not set
#
# Block devices
#
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not setCONFIG_BLK_DEV_LOOP=yCONFIG_BLK_DEV_CRYPTOLOOP=mCONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not setCONFIG_BLK_DEV_RAM=yCONFIG_BLK_DEV_RAM_COUNT=16CONFIG_BLK_DEV_RAM_SIZE=4096CONFIG_BLK_DEV_INITRD=y
# CONFIG_CDROM_PKTCDVD is not set
#
# IO Schedulers
#CONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_AS=yCONFIG_IOSCHED_DEADLINE=yCONFIG_IOSCHED_CFQ=yCONFIG_ATA_OVER_ETH=m
#
# ATA/ATAPI/MFM/RLL support
#CONFIG_IDE=yCONFIG_BLK_DEV_IDE=y
#
# Please see Documentation/ide.txt for help/info on IDE drives #
# CONFIG_BLK_DEV_IDE_SATA is not set CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
#
# IDE chipset support/bugfixes
#CONFIG_IDE_GENERIC=yCONFIG_BLK_DEV_IDEPCI=y
# CONFIG_IDEPCI_SHARE_IRQ is not set
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not setCONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not setCONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not setCONFIG_BLK_DEV_SGIIOC4=y
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not setCONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not setCONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not setCONFIG_SCSI=yCONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#CONFIG_BLK_DEV_SD=yCONFIG_CHR_DEV_ST=m
# CONFIG_CHR_DEV_OSST is not setCONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not setCONFIG_CHR_DEV_SG=mCONFIG_CHR_DEV_SCH=m
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
# CONFIG_SCSI_MULTI_LUN is not set CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set
#
# SCSI Transport Attributes
#CONFIG_SCSI_SPI_ATTRS=yCONFIG_SCSI_FC_ATTRS=y
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not setCONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_AHCI is not set
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_ATA_PIIX is not set
# CONFIG_SCSI_SATA_MV is not set
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_QSTOR is not set
# CONFIG_SCSI_SATA_SX4 is not set
# CONFIG_SCSI_SATA_SIL is not set
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not setCONFIG_SCSI_SATA_VITESSE=y
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_FC is not setCONFIG_SCSI_QLOGIC_1280=y
# CONFIG_SCSI_QLOGIC_1280_1040 is not setCONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not setCONFIG_SCSI_QLA22XX=yCONFIG_SCSI_QLA2300=yCONFIG_SCSI_QLA2322=y
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA24XX is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
#
# Multi-device support (RAID and LVM)
#CONFIG_MD=yCONFIG_BLK_DEV_MD=yCONFIG_MD_LINEAR=yCONFIG_MD_RAID0=yCONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not setCONFIG_MD_RAID5=y
# CONFIG_MD_RAID6 is not setCONFIG_MD_MULTIPATH=y
# CONFIG_MD_FAULTY is not setCONFIG_BLK_DEV_DM=yCONFIG_DM_CRYPT=mCONFIG_DM_SNAPSHOT=mCONFIG_DM_MIRROR=mCONFIG_DM_ZERO=mCONFIG_DM_MULTIPATH=mCONFIG_DM_MULTIPATH_EMC=m
#
# Fusion MPT device support
#CONFIG_FUSION=yCONFIG_FUSION_SPI=yCONFIG_FUSION_FC=y
# CONFIG_FUSION_SAS is not setCONFIG_FUSION_MAX_SGE=128CONFIG_FUSION_CTL=m
#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set
#
# I2O device support
#
# CONFIG_I2O is not set
#
# Network device support
#CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
#
# ARCnet devices
#
# CONFIG_ARCNET is not set
#
# PHY device support
#
#
# Ethernet (10 or 100Mbit)
#
# CONFIG_NET_ETHERNET is not set
#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SK98LIN is not setCONFIG_TIGON3=y
# CONFIG_BNX2 is not set
#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not setCONFIG_S2IO=m
# CONFIG_S2IO_NAPI is not set
# CONFIG_2BUFF_MODE is not set
#
# Token Ring devices
#
# CONFIG_TR is not set
#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set
#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not setCONFIG_NETCONSOLE=yCONFIG_NETPOLL=y
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not setCONFIG_NET_POLL_CONTROLLER=y
#
# ISDN subsystem
#
# CONFIG_ISDN is not set
#
# Telephony Support
#
# CONFIG_PHONE is not set
#
# Input device support
#CONFIG_INPUT=y
#
# Userland interfaces
#CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not setCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
# CONFIG_INPUT_KEYBOARD is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#CONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_HW_CONSOLE=yCONFIG_SERIAL_NONSTANDARD=y
# CONFIG_ROCKETPORT is not set
# CONFIG_CYCLADES is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
# CONFIG_SYNCLINKMP is not set
# CONFIG_N_HDLC is not set
# CONFIG_SPECIALIX is not set
# CONFIG_SX is not set
# CONFIG_STALDRV is not setCONFIG_SGI_SNSC=yCONFIG_SGI_TIOCX=yCONFIG_SGI_MBCS=m
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
#
# Non-8250 serial port support
#CONFIG_SERIAL_CORE=yCONFIG_SERIAL_CORE_CONSOLE=yCONFIG_SERIAL_SGI_L1_CONSOLE=y
# CONFIG_SERIAL_JSM is not setCONFIG_SERIAL_SGI_IOC4=yCONFIG_UNIX98_PTYS=yCONFIG_LEGACY_PTYS=yCONFIG_LEGACY_PTY_COUNT=256
#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set
#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not setCONFIG_EFI_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
#
# Ftape, the floppy tape device driver
#
# CONFIG_AGP is not set
# CONFIG_DRM is not setCONFIG_RAW_DRIVER=m
# CONFIG_HPET is not setCONFIG_MAX_RAW_DEVS=256
# CONFIG_HANGCHECK_TIMER is not setCONFIG_MMTIMER=y
#
# TPM devices
#
# CONFIG_TCG_TPM is not set
#
# I2C support
#
# CONFIG_I2C is not set
#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set
#
# Hardware Monitoring support
#
# CONFIG_HWMON is not set
# CONFIG_HWMON_VID is not set
#
# Misc devices
#
#
# Multimedia Capabilities Port drivers
#
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
#
# Graphics support
#
# CONFIG_FB is not set
#
# Console display driver support
#CONFIG_VGA_CONSOLE=yCONFIG_DUMMY_CONSOLE=y
#
# Sound
#
# CONFIG_SOUND is not set
#
# USB support
#CONFIG_USB_ARCH_HAS_HCD=yCONFIG_USB_ARCH_HAS_OHCI=yCONFIG_USB=m
# CONFIG_USB_DEBUG is not set
#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_ISP116X_HCD is not setCONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN is not setCONFIG_USB_OHCI_LITTLE_ENDIAN=yCONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
#
# USB Device Class drivers
#
# CONFIG_USB_BLUETOOTH_TTY is not set
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information #
# CONFIG_USB_STORAGE is not set
#
# USB Input Devices
#CONFIG_USB_HID=mCONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set
#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_ITMTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set
#
# Video4Linux support is needed for USB Multimedia device support #
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not setCONFIG_USB_MON=y
#
# USB port drivers
#
#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
#
# USB DSL modem support
#
#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set
#
# MMC/SD Card support
#
# CONFIG_MMC is not set
#
# InfiniBand support
#CONFIG_INFINIBAND=m
# CONFIG_INFINIBAND_USER_MAD is not set
# CONFIG_INFINIBAND_USER_ACCESS is not setCONFIG_INFINIBAND_MTHCA=m
# CONFIG_INFINIBAND_MTHCA_DEBUG is not setCONFIG_INFINIBAND_IPOIB=m
# CONFIG_INFINIBAND_IPOIB_DEBUG is not set
#
# SN Devices
#CONFIG_SGI_IOC4=y
#
# File systems
#CONFIG_EXT2_FS=yCONFIG_EXT2_FS_XATTR=yCONFIG_EXT2_FS_POSIX_ACL=yCONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not setCONFIG_EXT3_FS=yCONFIG_EXT3_FS_XATTR=yCONFIG_EXT3_FS_POSIX_ACL=yCONFIG_EXT3_FS_SECURITY=yCONFIG_JBD=y
# CONFIG_JBD_DEBUG is not setCONFIG_FS_MBCACHE=yCONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not setCONFIG_REISERFS_FS_XATTR=yCONFIG_REISERFS_FS_POSIX_ACL=yCONFIG_REISERFS_FS_SECURITY=y
# CONFIG_JFS_FS is not setCONFIG_FS_POSIX_ACL=yCONFIG_XFS_FS=yCONFIG_XFS_EXPORT=yCONFIG_XFS_QUOTA=y
# CONFIG_XFS_SECURITY is not setCONFIG_XFS_POSIX_ACL=yCONFIG_XFS_RT=y
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not setCONFIG_INOTIFY=yCONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not setCONFIG_QUOTACTL=yCONFIG_DNOTIFY=yCONFIG_AUTOFS_FS=mCONFIG_AUTOFS4_FS=m
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#CONFIG_ISO9660_FS=yCONFIG_JOLIET=y
# CONFIG_ZISOFS is not setCONFIG_UDF_FS=mCONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not setCONFIG_VFAT_FS=yCONFIG_FAT_DEFAULT_CODEPAGE=437CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#CONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_SYSFS=yCONFIG_TMPFS=yCONFIG_HUGETLBFS=yCONFIG_HUGETLB_PAGE=yCONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
#
# Network File Systems
#CONFIG_NFS_FS=mCONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not setCONFIG_NFS_V4=yCONFIG_NFS_DIRECTIO=yCONFIG_NFSD=mCONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not setCONFIG_NFSD_V4=yCONFIG_NFSD_TCP=yCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_EXPORTFS=yCONFIG_NFS_COMMON=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=mCONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_RPCSEC_GSS_SPKM3 is not setCONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not setCONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set
#
# Partition Types
#CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not setCONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not setCONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not setCONFIG_EFI_PARTITION=y
#
# Native Language Support
#CONFIG_NLS=yCONFIG_NLS_DEFAULT="iso8859-1"CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not setCONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not setCONFIG_NLS_UTF8=y
#
# Library routines
#
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not setCONFIG_CRC32=y
# CONFIG_LIBCRC32C is not setCONFIG_ZLIB_INFLATE=mCONFIG_ZLIB_DEFLATE=mCONFIG_GENERIC_ALLOCATOR=yCONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_PENDING_IRQ=y
#
# Profiling support
#
# CONFIG_PROFILING is not set
#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not setCONFIG_DEBUG_KERNEL=yCONFIG_MAGIC_SYSRQ=yCONFIG_LOG_BUF_SHIFT=20CONFIG_DETECT_SOFTLOCKUP=yCONFIG_SCHEDSTATS=yCONFIG_DEBUG_SLAB=yCONFIG_DEBUG_PREEMPT=yCONFIG_DEBUG_SPINLOCK=yCONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_KOBJECT is not setCONFIG_DEBUG_INFO=yCONFIG_DEBUG_FS=y
# CONFIG_KPROBES is not setCONFIG_IA64_GRANULE_16MB=y
# CONFIG_IA64_GRANULE_64MB is not set
# CONFIG_IA64_PRINT_HAZARDS is not set
# CONFIG_DISABLE_VHPT is not set
# CONFIG_IA64_DEBUG_CMPXCHG is not set
# CONFIG_IA64_DEBUG_IRQ is not setCONFIG_SYSVIPC_COMPAT=y
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
#
# Cryptographic options
#CONFIG_CRYPTO=yCONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_MD4 is not setCONFIG_CRYPTO_MD5=yCONFIG_CRYPTO_SHA1=m
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_TGR192 is not setCONFIG_CRYPTO_DES=m
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_AES is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_ANUBIS is not setCONFIG_CRYPTO_DEFLATE=m
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_TEST is not set
#
# Hardware crypto devices
#
Quoted 8 lines · Christoph Lameter
Quoted 5 lines · Christoph Lameter
Hmm. This likely has something to do with debugging code. I was unable to reproduce this on amd64 with your config. I get another failure with 2.6.14-rc2 on ia64 if I enable all the debugging features that you have. The system works fine if no debugging is configured:
Quoted 6 lines
Quoted 2 lines
kernel BUG at
kernel/workqueue.c:541!
swapper[1]: bugcheck! 0 [1]I fixed the above issue (a structure became larger than the maximum allowed by the slab allocator) and the kernel boots fine now on an 8 way ia64. Cannot reproduce the problem.
Petr can. I think we're still waiting for him to test the below (please):
Begin forwarded message:
Date: Tue, 20 Sep 2005 18:03:54 -0700 (PDT)
From: Christoph Lameter <[email protected]>
To: Petr Vandrovec <[email protected]>
Cc: Andrew Morton <[email protected]>, [email protected], [email protected], [email protected]
Subject: Re: 2.6.14-rc1-git-now still dying in mm/slab - this time line 1849
Quoted 3 lines · Petr Vandrovec
slab belonging to node#1, while having acquired lock for cachep belonging to node #0. Due to this
check_spinlock_acquired_node(cachep, nodeid)fails (check_spinlock_acquired_node(cachep, 0)would succeed).
Hmmm. If a node runs out of memory then pages from another node may end up on the slab list of a node. But it seems that free_block cannot handle that properly.
How are you producing the problem?
Could you try the following patch:
---
The numa slab allocator may allocate pages from foreign nodes onto the lists for a particular node if a node runs out of memory. Inspecting the slab->nodeid field will not reflect that the page is now in use for the slabs of another node.
This patch fixes that issue by adding a node field to free_block so that the caller can indicate which node currently uses a slab.
Also removes the check for the current node from kmalloc_cache_node since the process may shift later to another node which may lead to an allocation on another node than intended.
mm/slab.c +19 -19
@@ -639,7 +639,7 @@ static enum { static DEFINE_PER_CPU(struct work_struct, reap_work); -static void free_block (kmem_cache_t * cachep,void ** objpp,int len); +static void free_block (kmem_cache_t * cachep,void ** objpp,int len,int node);static void enable_cpucache (kmem_cache_t *cachep);static void cache_reap (void *unused);static int __node_shrink (kmem_cache_t *cachep,int node);
@@ -804,7 +804,7 @@ static inline void __drain_alien_cache(kif (ac->avail) {spin_lock (&rl3->list_lock); -free_block (cachep, ac->entry, ac->avail); +free_block (cachep, ac->entry, ac->avail, node); ac->avail =0 ;spin_unlock (&rl3->list_lock); }
@@ -925,7 +925,7 @@ static int __devinit cpuup_callback(stru/* Free limit for this kmem_list3 */ l3->free_limit -= cachep->batchcount;if (nc) -free_block (cachep, nc->entry, nc->avail); +free_block (cachep, nc->entry, nc->avail, node);if (!cpus_empty (mask)) {spin_unlock (&l3->list_lock);
@@ -934,7 +934,7 @@ static int __devinit cpuup_callback(struif (l3->shared) {free_block (cachep, l3->shared->entry, - l3->shared->avail); + l3->shared->avail, node);kfree (l3->shared); l3->shared =NULL ; }
@@ -1882,12 +1882,13 @@ static void do_drain(void *arg) {kmem_cache_t *cachep = (kmem_cache_t *)arg;struct array_cache *ac; +int node =numa_node_id ();check_irq_off (); ac =ac_data (cachep); -spin_lock (&cachep->nodelists[numa_node_id ()]->list_lock); -free_block (cachep, ac->entry, ac->avail); -spin_unlock (&cachep->nodelists[numa_node_id ()]->list_lock); +spin_lock (&cachep->nodelists[node]->list_lock); +free_block (cachep, ac->entry, ac->avail, node); +spin_unlock (&cachep->nodelists[node]->list_lock); ac->avail =0 ; }
@@ -2608,7 +2609,7 @@ done:/* * Caller needs to acquire correct kmem_list's list_lock */ -static void free_block (kmem_cache_t *cachep,void **objpp,int nr_objects) +static void free_block (kmem_cache_t *cachep,void **objpp,int nr_objects,int node) {int i;struct kmem_list3 *l3;
@@ -2617,14 +2618,12 @@ static void free_block(kmem_cache_t *cacvoid *objp = objpp[i];struct slab *slabp;unsigned int objnr; -int nodeid =0 ; slabp =GET_PAGE_SLAB (virt_to_page (objp)); - nodeid = slabp->nodeid; - l3 = cachep->nodelists[nodeid]; + l3 = cachep->nodelists[node];list_del (&slabp->list); objnr = (objp - slabp->s_mem) / cachep->objsize; -check_spinlock_acquired_node (cachep, nodeid); +check_spinlock_acquired_node (cachep, node);check_slabp (cachep, slabp);
@@ -2664,13 +2663,14 @@ static void cache_flusharray(kmem_cache_ {int batchcount;struct kmem_list3 *l3; +int node =numa_node_id (); batchcount = ac->batchcount;#if DEBUG BUG_ON (!batchcount || batchcount > ac->avail);#endif check_irq_off (); - l3 = cachep->nodelists[numa_node_id ()]; + l3 = cachep->nodelists[node];spin_lock (&l3->list_lock);if (l3->shared) {struct array_cache *shared_array = l3->shared;
@@ -2686,7 +2686,7 @@ static void cache_flusharray(kmem_cache_ } } -free_block (cachep, ac->entry, batchcount); +free_block (cachep, ac->entry, batchcount, node); free_done:#if STATS {
@@ -2751,7 +2751,7 @@ static inline void __cache_free(kmem_cac }else {spin_lock (&(cachep->nodelists[nodeid])-> list_lock); -free_block (cachep, &objp,1 ); +free_block (cachep, &objp,1 , nodeid);spin_unlock (&(cachep->nodelists[nodeid])-> list_lock); }
@@ -2844,7 +2844,7 @@ void *kmem_cache_alloc_node(kmem_cache_tunsigned long save_flags;void *ptr; -if (nodeid ==numa_node_id () || nodeid ==-1 ) +if (nodeid ==-1 )return __cache_alloc (cachep, flags);if (unlikely (!cachep->nodelists[nodeid])) {
@@ -3079,7 +3079,7 @@ static int alloc_kmemlist(kmem_cache_t *if ((nc = cachep->nodelists[node]->shared))free_block (cachep, nc->entry, - nc->avail); + nc->avail, node); l3->shared = new;if (!cachep->nodelists[node]->alien) {
@@ -3160,7 +3160,7 @@ static int do_tune_cpucache(kmem_cache_tif (!ccold)continue ;spin_lock_irq (&cachep->nodelists[cpu_to_node (i)]->list_lock); -free_block (cachep, ccold->entry, ccold->avail); +free_block (cachep, ccold->entry, ccold->avail,cpu_to_node (i));spin_unlock_irq (&cachep->nodelists[cpu_to_node (i)]->list_lock);kfree (ccold); }
@@ -3240,7 +3240,7 @@ static void drain_array_locked(kmem_cachif (tofree > ac->avail) { tofree = (ac->avail+1 )/2 ; } -free_block (cachep, ac->entry, tofree); +free_block (cachep, ac->entry, tofree, node); ac->avail -= tofree;memmove (ac->entry, &(ac->entry[tofree]),sizeof (void *)*ac->avail);
Quoted 6 lines · Andrew Morton
Quoted 4 lines · Christoph Lameter
Quoted 2 lines · Christoph Lameter
Hmm. This likely has something to do with debugging code. I was unable to reproduce this on amd64 with your config. I get another failure with 2.6.14-rc2 on ia64 if I enable all the debugging features that you have. The system works fine if no debugging is configured:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
kernel BUG at
kernel/workqueue.c:541!
swapper[1]: bugcheck! 0 [1]I fixed the above issue (a structure became larger than the maximum allowed by the slab allocator) and the kernel boots fine now on an 8 way ia64. Cannot reproduce the problem.
Petr can. I think we're still waiting for him to test the below (please):
Sorry, I've missed that half of email completely. Yes, it seems to fix problem, box has currently 8 min uptime, which is 7:55 more than it survived before.
Thanks,
Petr Vandrovec
Quoted 16 lines
Could you try the following patch:
---
The numa slab allocator may allocate pages from foreign nodes onto the lists for a particular node if a node runs out of memory. Inspecting the
slab->nodeidfield will not reflect that the page is now in use for the slabs of another node.This patch fixes that issue by adding a node field to
free_blockso that the caller can indicate which node currently uses a slab.Also removes the check for the current node from
kmalloc_cache_nodesince the process may shift later to another node which may lead to an allocation on another node than intended.Signed-off-by Christoph Lameter <[email protected]>
Quoted 3 lines · Petr Vandrovec
Sorry, I've missed that half of email completely. Yes, it seems to fix problem,
box has currently 8 min uptime, which is 7:55 more than it survived before.
I thought the box did not boot at all? The problem appears on an otherwise idle machine after bootup?
Quoted 19 lines · Petr Vandrovec
Quoted 17 lines · Andrew Morton
Quoted 13 lines · Christoph Lameter
Quoted 7 lines · Christoph Lameter
Hmm. This likely has something to do with debugging code. I was unable to reproduce this on amd64 with your config. I get another failure with 2.6.14-rc2 on ia64 if I enable all the debugging features that you have. The system works fine if no debugging is configured:
kernel BUG at
kernel/workqueue.c:541!
swapper[1]: bugcheck! 0 [1]I fixed the above issue (a structure became larger than the maximum allowed by the slab allocator) and the kernel boots fine now on an 8 way ia64. Cannot reproduce the problem.
Quoted 4 lines
Quoted 1 line
Petr can. I think we're still waiting for him to test the below (please):
Sorry, I've missed that half of email completely. Yes, it seems to fix problem, box has currently 8 min uptime, which is 7:55 more than it survived before.
Great, thanks. Christoph, was that patch the final official version?
Quoted 1 line · Andrew Morton
Great, thanks. Christoph, was that patch the final official version?
This should deal with the node ownership issue. So yes.
I still have some open question on how pages ended up on the wrong node. This should only happen if a zone / node has run out of memory. If pages ended up on the wrong node without that then there may be a different issue still to be fixed.
Maybe Petr can give us some more details on when the problem occurs?
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Andrew Morton
Great, thanks. Christoph, was that patch the final official version?
Quoted 6 lines
This should deal with the node ownership issue. So yes.
I still have some open question on how pages ended up on the wrong node.
This should only happen if a zone / node has run out of memory. If pages ended up on the wrong node without that then there may be a different issue still to be fixed.Maybe Petr can give us some more details on when the problem occurs?
Problem seems to happen immediately, and just first run of cache_reap (2 seconds after eventd initializes if I understand it correctly) already finds problem.
But I'm confused. I've just added code which is supposed to verify all additions to the cache entry[] (http://platan.vc.cvut.cz/verify-all-entry-add.diff)
on the top of Christoph patch to catch one which later causes problem in cache_reap, and it logs nothing at the time crash was happening :-( Only incident it logs is "while (batchcount > 0)" loop in cache_alloc_refill, saying that
objp ffff81007ffd9430 belonging to the slab ffff81007ffd9000 which belongs
to node 1 was added to array_cache belonging to node 0 (called from ffffffff8016e4a9) (mm/slab.c ~ line 2430)
... cache avc_node
This repeats couple of times, for avc_node, mnt_cache, proc_inode_cache and bdev_cache. Nothing else.
So I've reverted your fix, and still I did not catch offender, so I'm probably missing some place which populates array_cache entry[] :-(
Only if after I added logging to free_block() I was able to find that offender is proc_inode_cache. But I have no idea how this object appeared in the incorrect node cache...
Petr
Quoted 26 lines · Petr Vandrovec
Quoted 5 lines · Andrew Morton
Christoph Lameter <[email protected]> wrote:
...
They do. I don't believe that preemption is the source of this BUG. (Petr, doesCONFIG_PREEMPT=nfix it?)No, it does not. I've even added printks here and there to show node number,
and everything works as it should. Maybe there are some problems withnuma_node_id()and migrating between processors when memory gets released, I do not know.Only thing I know that if I'll add
WARN_ONbelow to thefree_block(), it triggers...@free_block
slabp = GET_PAGE_SLAB(virt_to_page(objp)); nodeid =slabp->nodeid;
+WARN_ON(nodeid != numa_node_id()); <<<<<
l3 = cachep->nodelist[nodeid];list_del(&slabp->list);
objnr = (objp -slabp->s_mem) /cachep->objsize;check_spinlock_acquired_node(cachep, nodeid);check_slabp(cachep, slabp);
Quoted 2 lines
... saying that keventd/0 tries to operate on
slab belonging to node#1, while having acquired lock for cachep belonging
^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line
to node #0
^^^^^^^^^^
Just might be relevant here, I found a bug with the recentx86_64 changes to 2.6.14-rc* which causes the node_to_cpumask[] to go bad for the boot processor. This happens on both amd and em64t boxes. I guess the kevent/0 cpus_allowed mask might be changed by the bad node_to_cpumask[] here?
On a opteron box (courtesy Ananth M)
# cat /sys/devices/system/node/node0/cpumap 00000000
# cat /sys/devices/system/node/node1/cpumap 00000003
On our em64t IBM x460 NUMA,
# cat /sys/devices/system/node/node0/cpumap 0000000e
# cat /sys/devices/system/node/node1/cpumap 000000f1
Here is a fix for that, I have sounded out Andi Kleen on this and waiting for his comments. Maybe somebody can test the patch below on amds?
Thanks,
Kiran
---
Patch to fix the BP node_to_cpumask. 2.6.14-rc* broke the boot cpu bit as the cpu_to_node(0) is now not setup early enough for numa_init_array. cpu_to_node[] is setup much later at srat_detect_node on acpi srat based em64t machines. This seems like a problem on amd machines too, Tested on em64t though. /sys/devices/system/node/node0/cpumap shows up sanely after this patch.
Signed off by: Ravikiran Thirumalai <[email protected]>
arch/x86_64/mm/numa.c +1 -4
@@ -178,7 +178,6 @@ rr++; } -set_bit (0 , &node_to_cpumask[cpu_to_node (0 )]); }#ifdef CONFIG_NUMA_EMU
@@ -266,9 +265,7 @@__cpuinit void numa_add_cpu (int cpu) { -/* BP is initialized elsewhere */ -if (cpu) -set_bit (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +set_bit (cpu, &node_to_cpumask[cpu_to_node (cpu)]); }unsigned long __init numa_free_all_bootmem (void )
Quoted 5 lines · Ravikiran G Thirumalai
Just might be relevant here, I found a bug with the recent
x86_64changes to 2.6.14-rc* which causes the node_to_cpumask[] to go bad for the boot processor. This happens on both amd and em64t boxes. I guess the kevent/0cpus_allowedmask might be changed by the bad node_to_cpumask[] here?
Andrew, could we add the following patch to the kernel to detect conditions in the future? This code will only be compiled in if slab debugging is enabled.
---
[SLAB] Add additional debugging to detect slabs from the wrong node
This patch adds some stack dumps if the slab logic is processing slab blocks from the wrong node. This is necessary in order to detect situations as encountered by Petr.
mm/slab.c +4 -1
@@ -2421,6 +2421,7 @@ retry: next =slab_bufctl (slabp)[slabp->free];#if DEBUG slab_bufctl (slabp)[slabp->free] =BUFCTL_FREE ; +WARN_ON (numa_node_id () != slabp->nodeid);#endif slabp->free = next; }
@@ -2635,8 +2636,10 @@ static void free_block(kmem_cache_t *caccheck_spinlock_acquired_node (cachep, node);check_slabp (cachep, slabp); -#if DEBUG +/* Verify that the slab belongs to the intended node */ +WARN_ON (slabp->nodeid != node); +if (slab_bufctl (slabp)[objnr] !=BUFCTL_FREE ) {printk (KERN_ERR "slab: double free detected in cache " "'%s', objp %p\n" , cachep->name, objp);
Ravikiran G Thirumalai wrote:
Unfortunately I must confirm that it does not fix problem. But it pointed out to me another thing - proc_inode_cache stuff is put into caches BEFORE this code is executed. So if anything in mm/slab.c relies on node_to_mask[] being valid (and if it relies on some other things which are set this late), it probably won't work.
Petr
Quoted 25 lines · Petr Vandrovec
Index:
linux-2.6.14-rc1/arch/x86_64/mm/numa.c
===================================================================arch/x86_64/mm/numa.c +1
-4@@ -178,7 +178,6 @@ rr++; } -set_bit (0 , &node_to_cpumask[cpu_to_node (0 )]); }#ifdef CONFIG_NUMA_EMU @@ -266,9 +265,7 @@__cpuinit void numa_add_cpu (int cpu) { -/* BP is initialized elsewhere */ -if (cpu) -set_bit (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +set_bit (cpu, &node_to_cpumask[cpu_to_node (cpu)]); }unsigned long __init numa_free_all_bootmem (void )
Quoted 7 lines · Petr Vandrovec
Ravikiran G Thirumalai wrote:
Unfortunately I must confirm that it does not fix problem. But it pointed out to me another thing -
proc_inode_cachestuff is put into caches BEFORE this code is executed. So if anything inmm/slab.crelies on node_to_mask[] being valid (and if it relies on some other things which are set this late), it probably won't work.
Hmmm. Another data point for this bug. Bryan, who encountered the same bug on his box just tried 2.6.13 stock + numa slab patches from 2.6.13-mm s, and apparently, the kernel booted up on his opteron. So I guess we should concentrate on the x86_64 bootup part.
Thanks,
Kiran
Quoted 4 lines · Ravikiran G Thirumalai
Hmmm. Another data point for this bug. Bryan, who encountered the same bug on his box just tried 2.6.13 stock + numa slab patches from 2.6.13-mm s, and apparently, the kernel booted up on his opteron. So I guess we should concentrate on the
x86_64bootup part.
Careful with the patchsets. Some of them contain my fix that masks the problem. Be sure to either have the WARN_ON statements in there that check for valid node numers or use a version before I added the node parameter to free_block.
Quoted 7 lines · Petr Vandrovec
Ravikiran G Thirumalai wrote:
Unfortunately I must confirm that it does not fix problem. But it pointed out to me another thing -
proc_inode_cachestuff is put into caches BEFORE this code is executed. So if anything inmm/slab.crelies on node_to_mask[] being valid (and if it relies on some other things which are set this late), it probably won't work.
The tests Alok carried out on Petr's box confirmed that cpu_to_node[BP] is not setup early enough by numa_init_array due to the x86_64 changes in
2.6.14-rc*, and unfortunately set wrongly by the work around code innuma_init_array(). cpu_to_node[0] gets set with 1 early and later gets set properly to 0 during identify_cpu() when all cpus are brought up, but confusing the numa slab in the process.
Here is a quick fix for this. The right fix obviously is to have cpu_to_node[bsp] setup early for numa_init_array(). The following patch will fix the problem now, and the code can stay on even when cpu_to_node{BP] gets fixed early correctly.
Thanks to Petr for access to his box.
Signed off by: Ravikiran Thirumalai <[email protected]>
arch/x86_64/mm/numa.c +2 -3
@@ -167,15 +167,14 @@mapping. To avoid this fill in the mapping for all possible CPUs, as the number of CPUs is not known yet. We round robin the existing nodes. */ - rr =0 ; + rr =first_node (node_online_map);for (i =0 ; i <NR_CPUS ; i++) {if (cpu_to_node[i] !=NUMA_NO_NODE )continue ; + cpu_to_node[i] = rr; rr =next_node (rr, node_online_map);if (rr ==MAX_NUMNODES ) rr =first_node (node_online_map); - cpu_to_node[i] = rr; - rr++; } }
Quoted 39 lines · Ravikiran G Thirumalai
Quoted 7 lines · Petr Vandrovec
Ravikiran G Thirumalai wrote:
Unfortunately I must confirm that it does not fix problem. But it pointed out to me another thing -
proc_inode_cachestuff is put into caches BEFORE this code is executed. So if anything inmm/slab.crelies on node_to_mask[] being valid (and if it relies on some other things which are set this late), it probably won't work.The tests Alok carried out on Petr's box confirmed that cpu_to_node[BP] is not setup early enough by
numa_init_arraydue to thex86_64changes in
2.6.14-rc*, and unfortunately set wrongly by the work around code innuma_init_array(). cpu_to_node[0] gets set with 1 early and later gets set properly to 0 duringidentify_cpu()when all cpus are brought up, but confusing the numa slab in the process.Here is a quick fix for this. The right fix obviously is to have cpu_to_node[bsp] setup early for
numa_init_array(). The following patch will fix the problem now, and the code can stay on even when cpu_to_node{BP] gets fixed early correctly.Thanks to Petr for access to his box.
Signed off by: Ravikiran Thirumalai <[email protected]>
Signed-off-by Alok N Kataria <[email protected]>Index:
slab-x86_64-fix-2.6.14-rc2/arch/x86_64/mm/numa.c
===================================================================arch/x86_64/mm/numa.c +1
-1@@ -167,15 +167,14 @@mapping. To avoid this fill in the mapping for all possible CPUs, as the number of CPUs is not known yet. We round robin the existing nodes. */ - rr =0 ; + rr =first_node (node_online_map);for (i =0 ; i <NR_CPUS ; i++) {if (cpu_to_node[i] !=NUMA_NO_NODE )
arch/x86_64/mm/numa.c line 180
Quoted 10 lines
arch/x86_64/mm/numa.c +1
-2continue ; + cpu_to_node[i] = rr; rr =next_node (rr, node_online_map);if (rr ==MAX_NUMNODES ) rr =first_node (node_online_map); - cpu_to_node[i] = rr; - rr++; } }
Is this fix independent from ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.14-rc2/2.6.14-rc2-mm2/broken-out/x86_64-fix-the-bp-node_to_cpumask.patch ?
Quoted 5 lines · Andrew Morton
Ravikiran G Thirumalai <[email protected]> wrote:
Is this fix independent from ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.14-rc2/2.6.14-rc2-mm2/broken-out/x86_64-fix-the-bp-node_to_cpumask.patch ?
Yes.
Quoted 1 line · Ravikiran G Thirumalai
Yes.
Kiran, your patch works for me, too. I can boot 2.6.14-rc2 with your patch, but not without it.
Thanks for your help.
<b
Quoted 2 lines · Bryan O'Sullivan
Kiran, your patch works for me, too. I can boot 2.6.14-rc2 with your patch, but not without it.
The patch is not in rc2-mm2 right? I can now reproduce it on a AMD64 single processor with numa emulation (numa=fake=2). So all x86_64 NUMA systems will throw these same stacktraces for rc2-mm2?
Bootdata ok (command line is root=/dev/sda2 console=tty0 console=ttyS0,38400n8 numa=fake=2)
Linux version 2.6.14-rc2-mm2 ([email protected]) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #2 SMP Fri Sep 30 15:49:50 UTC 2005 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003fff0000 (usable)
BIOS-e820: 000000003fff0000 - 000000003fff3000 (ACPI NVS)
BIOS-e820: 000000003fff3000 - 0000000040000000 (ACPI data)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
BIOS-e820: 00000000fefffc00 - 00000000ff000000 (reserved)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
Faking node 0 at 0000000000000000-000000000fffffff (255MB) Faking node 1 at 0000000010000000-000000003fff0000 (767MB) Bootmem setup node 0 0000000000000000-000000000fffffff Bootmem setup node 1 0000000010000000-000000003fff0000 Nvidia board detected. Ignoring ACPI timer override.
ACPI: PM-Timer IO Port: 0x4008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Processor #0 15:12 APIC version 16
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: BIOS IRQ0 pin2 override ignored.
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge) Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000) Built 2 zonelists
Initializing CPU#0
Kernel command line: root=/dev/sda2 console=tty0 console=ttyS0,38400n8 numa=fake=2
PID hash table entries: 4096 (order: 12, 131072 bytes) time.c: Using 3.579545 MHz PM timer.time.c: Detected 2210.110 MHz processor.
...
powernow-k8: Found 1 AMD Athlon 64 / Opteron processors (version 1.50.3) powernow-k8: 0 : fid 0xe (2200 MHz), vid 0x2 (1500 mV) powernow-k8: 1 : fid 0xc (2000 MHz), vid 0x6 (1400 mV) powernow-k8: 2 : fid 0xa (1800 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV)
cpu_init done, current fid 0xe, vid 0x2
Badness in cache_alloc_refill at mm/slab.c:2424
Call Trace:<ffffffff80164f8a>{cache_alloc_refill+458}
<ffffffff80165858>{kmem_cache_alloc+136}
<ffffffff8019f41e>{alloc_vfsmnt+30}
<ffffffff801893d2>{do_kern_mount+82}
<ffffffff8018fd5f>{getname+31} <ffffffff801a07ef>{do_new_mount+143}
<ffffffff801a0f5b>{do_mount+363} <ffffffff8018fd5f>{getname+31}
<ffffffff8015ee60>{__get_free_pages+16}
<ffffffff801a134b>{sys_mount+139}
<ffffffff8010b4ae>{name_to_dev_t+62}
<ffffffff80514e70>{prepare_namespace+80}
<ffffffff8010b16a>{init+250} <ffffffff8010ed2e>{child_rip+8}
<ffffffff8010b070>{init+0} <ffffffff8010ed26>{child_rip+0}Badness in cache_alloc_refill at mm/slab.c:2424
Call Trace:<ffffffff80164f8a>{cache_alloc_refill+458}
<ffffffff80165858>{kmem_cache_alloc+136}
<ffffffff8019f41e>{alloc_vfsmnt+30}
<ffffffff801893d2>{do_kern_mount+82}
<ffffffff8018fd5f>{getname+31} <ffffffff801a07ef>{do_new_mount+143}
<ffffffff801a0f5b>{do_mount+363} <ffffffff8018fd5f>{getname+31}
<ffffffff8015ee60>{__get_free_pages+16}
<ffffffff801a134b>{sys_mount+139}
<ffffffff8010b4ae>{name_to_dev_t+62}
<ffffffff80514e70>{prepare_namespace+80}
<ffffffff8010b16a>{init+250} <ffffffff8010ed2e>{child_rip+8}
<ffffffff8010b070>{init+0} <ffffffff8010ed26>{child_rip+0}Badness in cache_alloc_refill at mm/slab.c:2424
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Bryan O'Sullivan
Kiran, your patch works for me, too. I can boot 2.6.14-rc2 with your patch, but not without it.
The patch is not in rc2-mm2 right?
Correct.
Quoted 2 lines
I can now reproduce it on a AMD64
single processor with numa emulation (numa=fake=2).
That's helpful for reproducing the problem. Thanks.
Quoted 2 lines
So all
x86_64NUMA
systems will throw these same stacktraces for rc2-mm2?
I've only tried with HDAMA motherboards, but based on your report and Petr's, it seems somewhat likely.
<b
Quoted 7 lines · Bryan O'Sullivan
Quoted 1 line · Ravikiran G Thirumalai
Yes.
Kiran, your patch works for me, too. I can boot 2.6.14-rc2 with your patch, but not without it.
Thanks for your help.
It's already on its way to Linus. Thanks Kiran.
BTW for my defense: my NUMA boxes booted just fine with the original patchkit.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 7 lines · Bryan O'Sullivan
Quoted 1 line · Ravikiran G Thirumalai
Yes.
Kiran, your patch works for me, too. I can boot 2.6.14-rc2 with your patch, but not without it.
Thanks for your help.
It's already on its way to Linus. Thanks Kiran.
Thanks are also due to Alok for spending long hours trying out all combinations on Petr's box. Thanks Alok.
Kiran
Quoted 4 lines · Ravikiran G Thirumalai
Here is a quick fix for this. The right fix obviously is to have cpu_to_node[bsp] setup early for
numa_init_array(). The following patch will fix the problem now, and the code can stay on even when cpu_to_node{BP] gets fixed early correctly.
This fixes the problem that I can produce by booting with numa=fake=2
Quoted 77 lines · Andrew Morton
Quoted 59 lines · Petr Vandrovec
Quoted 12 lines · Andrew Morton
Quoted 9 lines · Petr Vandrovec
Quoted 7 lines · Andrew Morton
Quoted 3 lines · Petr Vandrovec
so now once crashes on UP system were sorted out, I tried to
put new kernel on my SMP host - and sorry to say, but it does not seem to work as advertised :-(Quoted 6 lines
Quoted 4 lines
Quoted 1 line
.config (again), please.
Any SMP with NUMA. One which I'm trying to debug now is attached. It is available at http://vana.vc.cvut.cz/config as well.
I can get 2.6.14-rc1 to crash with your .config, but current -linus is OK.
It still dies for me, with current git (tree
7513cdadc661cfe0bd1625145a4876e54df191ca, commit6c0741fbdee5bd0f8ed13ac287c4ab18e8ba7d83). Config is available at
http://platan.vc.cvut.cz/config-vana.txt. Box is dual opteron Tyan K8W, S2885....
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at mm/slab.c:1849 invalid operand: 0000 [1] SMP CPU 0 Modules linked in: Pid: 8, comm: events/0 Not tainted 2.6.14-rc1-6c07 #1 RIP: 0010:[<ffffffff8016d316>] <ffffffff8016d316>{free_block+294} RSP: 0000:ffff81007ff21d88 EFLAGS: 00010002 RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000310 RDX: 0000000000000000 RSI: ffff81007ffddd10 RDI: ffff81007ffda080 RBP: ffff81007ffde000 R08: ffff81003ffa0d50 R09: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff81007ffc9b50 R13: ffff81007ffde048 R14: ffff81007ffda080 R15: ffff81007ffda080 FS: 0000000000000000(0000) GS:ffffffff805f2800(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0 Process events/0 (pid: 8, threadinfo ffff81007ff20000, task ffff81003ff8c790) Stack: 0000000000000000 0000000000000000 0000000000000292 hda: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive 0000000200000000 ffff81007ffddd10 ffff81007ffddd10 ffff81007ffddce8 0000000000000002 0000000000000000 ffff81007ffda080 Call Trace:<ffffffff8016e8b7>{drain_array_locked+167} <ffffffff8016e9f7>{cache_reap+231} <ffffffff80131e23>{__wake_up+67} <ffffffff8016e910>{cache_reap+0} <ffffffff8014930c>{worker_thread+476} <ffffffff80131d60>{default_wake_function+0} <ffffffff80131d60>{default_wake_function+0} <ffffffff80149130>{worker_thread+0} <ffffffff8014db82>{kthread+146} <ffffffff8010ec22>{child_rip+8} <ffffffff80149130>{worker_thread+0} <ffffffff8014daf0>{kthread+0} <ffffffff8010ec1a>{child_rip+0} Code: 0f 0b 68 9d 26 3d 80 c2 39 07 48 89 ee 4c 89 ff 4c 8d 75 30Quoted 2 lines
RIP <ffffffff8016d316>{free_block+294} RSP <
ffff81007ff21d88>
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14Well. The
CPU_UP_CANCELEDlocking incpuup_callback()looks borked to me - it takes cachep->nodelists[node]->list_lock and then callsdrain_alien_cache()which appears to take the same lock. But that's not the problem here.The code in
cache_reap()recalculatesnuma_node_id()multiple times, so if the caller changes CPUs then this assertion will trigger. However it's running under keventd here, which is pinned to a single CPU. Still, it would be useful if you could try putting preempt_disable()s incache_reap(), or changecache_reap()to evaluatenuma_node_id()just the once, and cache that in a local variable.
Quoted 2 lines
I wonder why
numa_node_id()usesraw_smp_processor_id()? That's just asking for preempt non-atomicity bugs.
I've thought that this is problem, but as far as I can tell while this is problem it does not happen here. Just free_block() finds that pointer it got from caller belongs to the slab that belongs to the CPU#1/node#1 while caller obtained lock on CPU#0/node#0 structures. Which suggests that drain_array_locked() was issued with node #0 while array_cache->entry it got contains blocks which belong to node #1. Which I cannot explain.
Petr
Quoted 6 lines · Petr Vandrovec
I've thought that this is problem, but as far as I can tell while this is problem it does not happen here. Just
free_block()finds that pointer it got from caller belongs to the slab that belongs to the CPU#1/node#1 while caller obtained lock on CPU#0/node#0 structures. Which suggests thatdrain_array_locked()was issued with node #0 whilearray_cache->entryit got contains blocks which belong to node #1. Which I cannot explain.
That can happen if node 0 runs out of memory and the page_allocator falls back to take memory from node 1 for node 0 requests.
Maybe we have a problem here.