OOM-killer too aggressive?
Subsystems (4)
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
oom-killer: gfp_mask=0xd1, order=3
Call Trace: <ffffffff8104ed46>{out_of_memory+58} <ffffffff8104ff30>{__alloc_pages+534}
<ffffffff8104ffee>{__get_free_pages+48} <ffffffff8117d8e9>{dma_mem_alloc+31}
<ffffffff81183e70>{floppy_open+348} <ffffffff81072125>{do_open+172}
<ffffffff810724b4>{blkdev_open+0} <ffffffff810724dc>{blkdev_open+40}
<ffffffff81069fea>{__dentry_open+230} <ffffffff8106a10e>{nameidata_to_filp+40}
<ffffffff8106a153>{do_filp_open+51} <ffffffff8106a2cb>{get_unused_fd+116}
<ffffffff8106a477>{do_sys_open+73} <ffffffff8106a4d3>{sys_open+27}
<ffffffff8100aa3a>{system_call+126}Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:184
cpu 0 cold: high 62, batch 15 used:4
cpu 1 hot: high 186, batch 31 used:160
cpu 1 cold: high 62, batch 15 used:4
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 2843384kB (0kB HighMem)
Active:10367 inactive:38871 dirty:42 writeback:0 unstable:0 free:710846 slab:4726 mapped:2155 pagetables:147
DMA free:44kB min:32kB low:40kB high:48kB active:0kB inactive:0kB present:15728kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 3014 3014 3014
DMA32 free:2843340kB min:7008kB low:8760kB high:10512kB active:41468kB inactive:155484kB present:3086500kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44kB
DMA32: 933*4kB 573*8kB 229*16kB 74*32kB 21*64kB 5*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 690*4096kB = 2843340kB
Normal: empty
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap = 1477960kB
Total swap = 1477960kB
Free swap: 1477960kB
786416 pages of RAM
17590 reserved pages
10033 pages shared
0 pages swap cached
Out of Memory: Killed process 4886 (dbus-daemon).
Looking at floppy_open, we have:
if (!floppy_track_buffer) {
/* if opening an ED drive, reserve a big buffer,
* else reserve a small one */
if ((UDP->cmos == 6) || (UDP->cmos == 5))
try = 64; /* Only 48 actually useful */
else
try = 32; /* Only 24 actually useful */
tmp = (char *)fd_dma_mem_alloc(1024 * try);
if (!tmp && !floppy_track_buffer) {
try >>= 1; /* buffer only one side */
INFBOUND(try, 16);
tmp = (char *)fd_dma_mem_alloc(1024 * try);
}
if (!tmp && !floppy_track_buffer) {
fallback_on_nodma_alloc(&tmp, 2048 * try);
}So it will try to allocate half its first request if that fails, then fall back to non-DMA memory as a last resort, but doesn't get a chance because the OOM killer gets invoked. Maybe we need a new flag that says "fail me immediately if no memory available"?
Or should floppy.c be fixed so it doesn't ask for so much?
I found a diagnostic patch but only this part applies to 2.6.16-rc4:
Quoted 1 line
From: Jens Axboe <[email protected]>
block/ll_rw_blk.c +2 -0
@@ -637,6 +637,8 @@ void blk_queue_bounce_limit(request_queu {unsigned long bounce_pfn = dma_addr >>PAGE_SHIFT ; +printk ("q=%p, dma_addr=%llx, bounce pfn %lu\n" , q, dma_addr, bounce_pfn); +/* * set appropriate bounce gfp mask -- unfortunately we don't have a * full 4GB zone, so we have to resort to low memory for any bounces.
Signature · 2 lines
Chuck "Equations are the Devil's sentences." --Stephen Colbert
Quoted 6 lines · Chuck Ebbert
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
oom-killer: gfp_mask=0xd1, order=3
This could be related to the known GFP_DMA oom on some x86_64 machines.
Quoted 20 lines
Looking at
floppy_open, we have:if (!floppy_track_buffer) { /* if opening an ED drive, reserve a big buffer, * else reserve a small one */ if ((UDP->cmos == 6) || (UDP->cmos == 5)) try = 64; /* Only 48 actually useful */ else try = 32; /* Only 24 actually useful */ tmp = (char *)fd_dma_mem_alloc(1024 * try); if (!tmp && !floppy_track_buffer) { try >>= 1; /* buffer only one side */ INFBOUND(try, 16); tmp = (char *)fd_dma_mem_alloc(1024 * try); } if (!tmp && !floppy_track_buffer) { fallback_on_nodma_alloc(&tmp, 2048 * try); }
Quoted 4 lines
So it will try to allocate half its first request if that fails, then fall back to non-DMA memory as a last resort, but doesn't get a chance because the OOM killer gets invoked. Maybe we need a new flag that says "fail me immediately if no memory available"?
That's __GFP_NORETRY.
Quoted 2 lines
Or should
floppy.cbe fixed so it doesn't ask for so much?
The page allocator uses 32k as the threshold for when-to-try-like-crazy.
x86_64 should probably be defining its own fd_dma_mem_alloc() which doesn't use GFP_DMA.
drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +2 -1
@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_#endif #ifndef fd_dma_mem_alloc -#define fd_dma_mem_alloc (size) __get_dma_pages(GFP_KERNEL,get_order(size)) +#define fd_dma_mem_alloc (size) \ + __get_dma_pages(GFP_KERNEL|__GFP_NORETRY,get_order(size))#endif static inline void fallback_on_nodma_alloc (char **addr,size_t l)
_
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Chuck Ebbert
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
oom-killer: gfp_mask=0xd1, order=3
This could be related to the known
GFP_DMAoom on somex86_64machines.
What known GFP_DMA oom? GFP_DMA allocation should work.
-Andi
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Chuck Ebbert
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
oom-killer: gfp_mask=0xd1, order=3
This could be related to the known
GFP_DMAoom on somex86_64machines.What known
GFP_DMAoom?GFP_DMAallocation should work.
There's a problem on some x86_64 machines which confuses the BIO layer. BIO makes simple decisions about bounce pfns and some x86_64 memory layouts cause them to go wrong. Net effect: lots of GFP_DMA allocations in the BIO layer.
http://readlist.com/lists/vger.kernel.org/linux-kernel/36/182357.html
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Chuck Ebbert
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
oom-killer: gfp_mask=0xd1, order=3
This could be related to the known
GFP_DMAoom on somex86_64machines.
I'm not sure if this has any bearing on it, but the OOM Killer only does this when I compile the kernel with SMP support.
drivers/block/floppy.c static void do_fd_request(request_queue_
Quoted 10 lines
Quoted 1 line
Or should
floppy.cbe fixed so it doesn't ask for so much?The page allocator uses 32k as the threshold for when-to-try-like-crazy.
x86_64should probably be defining its ownfd_dma_mem_alloc()which doesn't useGFP_DMA.drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +0
-0@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_
Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.
--
Chris Largret <http://daga.dyndns.org>
Quoted 7 lines · Chris Largret
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Chuck Ebbert
Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
oom-killer: gfp_mask=0xd1, order=3
This could be related to the known
GFP_DMAoom on somex86_64machines.I'm not sure if this has any bearing on it, but the OOM Killer only does this when I compile the kernel with SMP support.
I doubt if that's related.
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Or should
floppy.cbe fixed so it doesn't ask for so much?The page allocator uses 32k as the threshold for when-to-try-like-crazy.
x86_64should probably be defining its ownfd_dma_mem_alloc()which doesn't useGFP_DMA.
Quoted 9 lines
drivers/block/floppy.c static void do_fd_request(request_queue_
Quoted 3 lines
drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +0
-0@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.
hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.
mm/page_alloc.c renamed from mm/page_alloc.c~a +3 -2
@@ -1003,7 +1003,8 @@ rebalance: zonelist, alloc_flags);if (page) goto got_pg; - }else if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { + }else if ((gfp_mask & __GFP_FS) && + !(gfp_mask & (__GFP_NORETRY|__GFP_DMA))) {/* * Go through the zonelist yet one more time, keep * very high watermark here, this is only to catch
@@ -1027,7 +1028,7 @@ rebalance:* <= 3, but that may not be true in other implementations. */ do_retry =0 ; -if (!(gfp_mask & __GFP_NORETRY)) { +if (!(gfp_mask & (__GFP_NORETRY|__GFP_DMA))) {if ((order <=3 ) || (gfp_mask & __GFP_REPEAT)) do_retry =1 ;if (gfp_mask & __GFP_NOFAIL)
_
mm/page_alloc.c rebalance:
Quoted 12 lines · Andrew Morton
Quoted 4 lines
Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.
hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.
mm/page_alloc.c renamed from mm/page_alloc.c~a +0
-0@@ -1003,7 +1003,8 @@ rebalance:
I reversed the previous patch before applying this one. If they were supposed to be used together, let me know.
Quoted 1 line
From the initial results it looks like the OOM Killer is not being used
now, Unfortunately I can't check with dmesg because right after login is initiated (but before I get a chance to type anything) there is a "Kernel BUG" message. This is all that is is printed when a serial console is in use. If you need the rest of the information, let me know and I'll see about typing it up.
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/vmalloc.c:352 invalid opcode: 0000 [1] SMP CPU 1 Modules linked in: snd_pcm_oss snd_mixer_oss md5 ipv6 ipt_recent
ipt_REJECT xt_state xt_tcpudp iptable_filter ip_tables x_tables nfs lockd nfs_acl sunrpc uhci_hcd r8169 ohci1394 ieee1394 emu10k1_gp gameport snd_emu10k1 snd_rawmidi snd_ac97_codec snd_ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd tda9887 tuner cx8800 cx88xx video_buf ir_common tveeprom compat_ioctl32 v4l1_compat v4l2_common btcx_risc videodev forcedeth usblp ohci_hcd i2c_nforce2 ehci_hc
--
Chris Largret <http://daga.dyndns.org>
Quoted 11 lines · Chris Largret
Quoted 9 lines · Andrew Morton
Quoted 4 lines
Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.
hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.
Quoted 6 lines
mm/page_alloc.c rebalance:
Quoted 3 lines
mm/page_alloc.c renamed from mm/page_alloc.c~a +0
-0@@ -1003,7 +1003,8 @@ rebalance:I reversed the previous patch before applying this one. If they were supposed to be used together, let me know.
No, that's right.
Quoted 12 lines
Quoted 1 line
From the initial results it looks like the OOM Killer is not being used
now, Unfortunately I can't check with dmesg because right after login is initiated (but before I get a chance to type anything) there is a "Kernel BUG" message. This is all that is is printed when a serial console is in use. If you need the rest of the information, let me know and I'll see about typing it up.
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/vmalloc.c:352 invalid opcode: 0000 [1] SMP CPU 1 Modules linked in: snd_pcm_oss snd_mixer_oss md5 ipv6 ipt_recent
Quoted 7 lines
ipt_REJECTxt_statext_tcpudpiptable_filterip_tablesx_tablesnfs lockdnfs_aclsunrpcuhci_hcdr8169 ohci1394 ieee1394emu10k1_gpgameportsnd_emu10k1snd_rawmidisnd_ac97_codecsnd_ac97_bussnd_pcmsnd_seq_devicesnd_timersnd_page_allocsnd_util_memsnd_hwdepsnd tda9887 tuner cx8800 cx88xxvideo_bufir_commontveepromcompat_ioctl32v4l1_compatv4l2_commonbtcx_riscvideodev forcedeth usblpohci_hcdi2c_nforce2ehci_hc
Sigh. The floppy driver's just a jpke. Looks like the failed allocation fell back to vmalloc then screwed it up.
I rather doubt whether x86_64 needs to be constraining itself to the ISA DMA region anyway - something for Andi to look at please?
You could try this one instead, although I guess I'll need to fire up the test box for this bug.
include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1 -1
@@ -40,7 +40,7 @@#define fd_disable_irq () disable_irq(FLOPPY_IRQ)#define fd_free_irq () free_irq(FLOPPY_IRQ, NULL)#define fd_get_dma_residue () SW._get_dma_residue(FLOPPY_DMA) -#define fd_dma_mem_alloc (size) SW._dma_mem_alloc(size) +#define fd_dma_mem_alloc (size) __alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))#define fd_dma_setup (addr, size, mode, io) SW._dma_setup(addr, size, mode, io)#define FLOPPY_CAN_FALLBACK_ON_NODMA
_
Quoted 9 lines · Andrew Morton
Sigh. The floppy driver's just a jpke. Looks like the failed allocation fell back to vmalloc then screwed it up.
You could try this one instead, although I guess I'll need to fire up the test box for this bug.
include/asm-x86_64/floppy.h line 46
Quoted 8 lines
include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1
-1@@ -40,7 +40,7 @@#define fd_disable_irq () disable_irq(FLOPPY_IRQ)#define fd_free_irq () free_irq(FLOPPY_IRQ, NULL)#define fd_get_dma_residue () SW._get_dma_residue(FLOPPY_DMA) -#define fd_dma_mem_alloc (size) SW._dma_mem_alloc(size) +#define fd_dma_mem_alloc (size) __alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))#define fd_dma_setup (addr, size, mode, io) SW._dma_setup(addr, size, mode, io)#define FLOPPY_CAN_FALLBACK_ON_NODMA
CC drivers/block/floppy.odrivers/block/floppy.c: In function `raw_cmd_copyin':drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'drivers/block/floppy.c: In function `floppy_open':drivers/block/floppy.c:3738: error: too few arguments to function `__alloc_pages'drivers/block/floppy.c:3742: error: too few arguments to function `__alloc_pages'
make[2]: *** [drivers/block/floppy.o] Error 1
make[1]: *** [drivers/block] Error 2
make: *** [drivers] Error 2
I'm sorry, but I'm not sure where to start for looking up the definition for __alloc_pages().
--
Chris Largret <http://daga.dyndns.org>
Quoted 3 lines · Chris Largret
drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'
doh.
include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1 -1
@@ -40,7 +40,7 @@#define fd_disable_irq () disable_irq(FLOPPY_IRQ)#define fd_free_irq () free_irq(FLOPPY_IRQ, NULL)#define fd_get_dma_residue () SW._get_dma_residue(FLOPPY_DMA) -#define fd_dma_mem_alloc (size) SW._dma_mem_alloc(size) +#define fd_dma_mem_alloc (size) alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))#define fd_dma_setup (addr, size, mode, io) SW._dma_setup(addr, size, mode, io)#define FLOPPY_CAN_FALLBACK_ON_NODMA
_
include/asm-x86_64/floppy.h
Quoted 9 lines · Andrew Morton
Quoted 3 lines · Chris Largret
drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'doh.
include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +0
-0
Earlier I said that there was a "Kernel BUG" and all processing stopped right after the login prompt was displayed (but before I could type anything). Now the kernel continues to work, but the messages are a little disconcerting. Here is the version with a backtrace (from dmesg):
Bad page state in process 'swapper'
page:ffff810001539168 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:0
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff8100015391a0 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff8100015391d8 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff810001539210 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff810001539248 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff810001539280 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff8100015392b8 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}Bad page state in process 'swapper'
page:ffff8100015392f0 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:
Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
<ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
<ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
<ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
<ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
<ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
<ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
<ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
<ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
<ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
<ffffffff814390a1>{start_secondary+1189}--
Chris Largret <http://daga.dyndns.org>
Quoted 3 lines
hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.
Better would be to fix the block layer. I think something like that would be better: (only lightly tested - it booted on a 6GB x86-64 box)
It is over pessimistic on systems with real IOMMU that can even remap to DMA addresses < 4GB - in the future those might want to define some ARCH_HAS macro so it can be checked here.
Does that patch fix the problem?
That said adding GFP_NORETRY to the floppy allocation is probably still a good idea. I will do that change here.
-Andi
Disable block layer bouncing for most memory on 64bit systems
The low level PCI DMA mapping functions should handle it in most cases.
This should fix problems with depleting the DMA zone early. The old code used precious GFP_DMA memory in many cases where it was not needed.
block/ll_rw_blk.c +18 -12
@@ -625,26 +625,32 @@ static inline int ordered_bio_endio(stru* Different hardware can have different requirements as to what pages * it can do I/O directly to. A low level driver can call * blk_queue_bounce_limit to have lower memory pages allocated as bounce -* buffers for doing I/O to pages residing above @page. By default -* the block layer sets this to the highest numbered "low" memory page. +* buffers for doing I/O to pages residing above @page. **/ void blk_queue_bounce_limit (request_queue_t *q,u64 dma_addr) {unsigned long bounce_pfn = dma_addr >>PAGE_SHIFT ; +int dma =0 ; -/* -* set appropriate bounce gfp mask -- unfortunately we don't have a -* full 4GB zone, so we have to resort to low memory for any bounces. -* ISA has its own < 16MB zone. -*/ -if (bounce_pfn < blk_max_low_pfn) { + q->bounce_gfp =GFP_NOIO ; +#if BITS_PER_LONG ==64 +/* Assume anything >= 4GB can be handled by IOMMU. +Actually some IOMMUs can handle everything, but I don't +know of a way to test this here. */ +if (bounce_pfn < (0xffffffff >>PAGE_SHIFT )) + dma =1 ; + q->bounce_pfn = max_low_pfn; +#else +if (bounce_pfn < blk_max_low_pfn) + dma =1 ; + q->bounce_pfn = bounce_pfn; +#endif +if (dma) {BUG_ON (dma_addr <BLK_BOUNCE_ISA );init_emergency_isa_pool (); q->bounce_gfp =GFP_NOIO |GFP_DMA ; - }else - q->bounce_gfp =GFP_NOIO ; - - q->bounce_pfn = bounce_pfn; + q->bounce_pfn = bounce_pfn; + } }EXPORT_SYMBOL (blk_queue_bounce_limit);
Thinking about this more I think we need a __GFP_NOOOM for other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.
-Andi
Quoted 3 lines · Andi Kleen
Thinking about this more I think we need a
__GFP_NOOOMfor other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.
Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andi Kleen
Thinking about this more I think we need a
__GFP_NOOOMfor other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.
No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.
Maybe __GFP_NORETRY is actually good enough for this purpose. Opinions?
-Andi
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andi Kleen
Thinking about this more I think we need a
__GFP_NOOOMfor other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.
Quoted 8 lines
Quoted 2 lines
Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.
No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.
Maybe
__GFP_NORETRYis actually good enough for this purpose. Opinions?
I was thinking that your __GFP_NOOOM was a thinko. How would it differ from __GFP_NORETRY?
Quoted 6 lines · Andrew Morton
Quoted 6 lines · Christoph Lameter
Quoted 2 lines
Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.
No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.
Right but it specifies in its flags that not all system memory can satisfy this particular memory request. That fact may be detected by the out_of_memory() function. We could do something special there instead of OOMing.
Quoted 2 lines
I was thinking that your
__GFP_NOOOMwas a thinko. How would it differ from__GFP_NORETRY?
__GFP_NORETRY seems to skip at least one retry pass as far as I can see. __GFP_NOOOM wouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 3 lines
I was thinking that your
__GFP_NOOOMwas a thinko. How would it differ from__GFP_NORETRY?
__GFP_NORETRYseems to skip at least one retry pass as far as I can see.__GFP_NOOOMwouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.
Oh, OK. That final get_page_from_freelist() is allegedly to see if a parallel oom-killing freed some pages - we already know that try_to_free_pages() didn't work.
I rather doubt that it'll make any difference.
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Andi Kleen
Quoted 3 lines
I was thinking that your
__GFP_NOOOMwas a thinko. How would it differ from__GFP_NORETRY?
__GFP_NORETRYseems to skip at least one retry pass as far as I can see.__GFP_NOOOMwouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.
Quoted 5 lines
Oh, OK. That final
get_page_from_freelist()is allegedly to see if a parallel oom-killing freed some pages - we already know thattry_to_free_pages()didn't work.I rather doubt that it'll make any difference.
I switched over the x86-64 IOMMU code and floppy code to use __GFP_NORETRY now.
But perhaps it would be better to rename it to __GFP_NOOOM because I think that would express its meaning better.
-Andi