Skip to contentSkip to conversation

OOM-killer too aggressive?

PATCH Chuck Ebbert 5 participants 20 messages Feb 26, 2006 – Feb 28, 2006

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

oom-killer: gfp_mask=0xd1, order=3

Call Trace: <ffffffff8104ed46>{out_of_memory+58} <ffffffff8104ff30>{__alloc_pages+534}
       <ffffffff8104ffee>{__get_free_pages+48} <ffffffff8117d8e9>{dma_mem_alloc+31}
       <ffffffff81183e70>{floppy_open+348} <ffffffff81072125>{do_open+172}
       <ffffffff810724b4>{blkdev_open+0} <ffffffff810724dc>{blkdev_open+40}
       <ffffffff81069fea>{__dentry_open+230} <ffffffff8106a10e>{nameidata_to_filp+40}
       <ffffffff8106a153>{do_filp_open+51} <ffffffff8106a2cb>{get_unused_fd+116}
       <ffffffff8106a477>{do_sys_open+73} <ffffffff8106a4d3>{sys_open+27}
       <ffffffff8100aa3a>{system_call+126}

Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:184
cpu 0 cold: high 62, batch 15 used:4
cpu 1 hot: high 186, batch 31 used:160
cpu 1 cold: high 62, batch 15 used:4
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 2843384kB (0kB HighMem)
Active:10367 inactive:38871 dirty:42 writeback:0 unstable:0 free:710846 slab:4726 mapped:2155 pagetables:147
DMA free:44kB min:32kB low:40kB high:48kB active:0kB inactive:0kB present:15728kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 3014 3014 3014
DMA32 free:2843340kB min:7008kB low:8760kB high:10512kB active:41468kB inactive:155484kB present:3086500kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44kB
DMA32: 933*4kB 573*8kB 229*16kB 74*32kB 21*64kB 5*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 690*4096kB = 2843340kB
Normal: empty
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap = 1477960kB
Total swap = 1477960kB
Free swap: 1477960kB
786416 pages of RAM
17590 reserved pages
10033 pages shared
0 pages swap cached
Out of Memory: Killed process 4886 (dbus-daemon).

Looking at floppy_open, we have:

        if (!floppy_track_buffer) {
                /* if opening an ED drive, reserve a big buffer,
                 * else reserve a small one */
                if ((UDP->cmos == 6) || (UDP->cmos == 5))
                        try = 64;       /* Only 48 actually useful */
                else
                        try = 32;       /* Only 24 actually useful */

                tmp = (char *)fd_dma_mem_alloc(1024 * try);
                if (!tmp && !floppy_track_buffer) {
                        try >>= 1;      /* buffer only one side */
                        INFBOUND(try, 16);
                        tmp = (char *)fd_dma_mem_alloc(1024 * try);
                }
                if (!tmp && !floppy_track_buffer) {
                        fallback_on_nodma_alloc(&tmp, 2048 * try);
                }

So it will try to allocate half its first request if that fails, then fall back to non-DMA memory as a last resort, but doesn't get a chance because the OOM killer gets invoked. Maybe we need a new flag that says "fail me immediately if no memory available"?

Or should floppy.c be fixed so it doesn't ask for so much?

I found a diagnostic patch but only this part applies to 2.6.16-rc4:

Quoted 1 line

From: Jens Axboe <[email protected]>

block/ll_rw_blk.c +2 -0
@@ -637,6 +637,8 @@ void blk_queue_bounce_limit(request_queu
 {
 	unsigned long bounce_pfn = dma_addr >> PAGE_SHIFT;
 
+	printk("q=%p, dma_addr=%llx, bounce pfn %lu\n", q, dma_addr, bounce_pfn);
+
 	/*
 	 * set appropriate bounce gfp mask -- unfortunately we don't have a
 	 * full 4GB zone, so we have to resort to low memory for any bounces.
Signature · 2 lines
Chuck
"Equations are the Devil's sentences."  --Stephen Colbert
#1 lore raw
Quoted 6 lines · Chuck Ebbert

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

oom-killer: gfp_mask=0xd1, order=3

This could be related to the known GFP_DMA oom on some x86_64 machines.

Quoted 20 lines

Looking at floppy_open, we have:

        if (!floppy_track_buffer) {
                /* if opening an ED drive, reserve a big buffer,
                 * else reserve a small one */
                if ((UDP->cmos == 6) || (UDP->cmos == 5))
                        try = 64;       /* Only 48 actually useful */
                else
                        try = 32;       /* Only 24 actually useful */

                tmp = (char *)fd_dma_mem_alloc(1024 * try);
                if (!tmp && !floppy_track_buffer) {
                        try >>= 1;      /* buffer only one side */
                        INFBOUND(try, 16);
                        tmp = (char *)fd_dma_mem_alloc(1024 * try);
                }
                if (!tmp && !floppy_track_buffer) {
                        fallback_on_nodma_alloc(&tmp, 2048 * try);
                }
Quoted 4 lines

So it will try to allocate half its first request if that fails, then fall back to non-DMA memory as a last resort, but doesn't get a chance because the OOM killer gets invoked. Maybe we need a new flag that says "fail me immediately if no memory available"?

That's __GFP_NORETRY.

Quoted 2 lines

Or should floppy.c be fixed so it doesn't ask for so much?

The page allocator uses 32k as the threshold for when-to-try-like-crazy.

x86_64 should probably be defining its own fd_dma_mem_alloc() which doesn't use GFP_DMA.

drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +2 -1
@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_
 #endif
 
 #ifndef fd_dma_mem_alloc
-#define fd_dma_mem_alloc(size) __get_dma_pages(GFP_KERNEL,get_order(size))
+#define fd_dma_mem_alloc(size)	\
+		__get_dma_pages(GFP_KERNEL|__GFP_NORETRY,get_order(size))
 #endif
 
 static inline void fallback_on_nodma_alloc(char **addr, size_t l)

_

#2 lore raw
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Chuck Ebbert

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

oom-killer: gfp_mask=0xd1, order=3

This could be related to the known GFP_DMA oom on some x86_64 machines.

What known GFP_DMA oom? GFP_DMA allocation should work.

-Andi

#3 lore raw
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Chuck Ebbert

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

oom-killer: gfp_mask=0xd1, order=3

This could be related to the known GFP_DMA oom on some x86_64 machines.

What known GFP_DMA oom? GFP_DMA allocation should work.

There's a problem on some x86_64 machines which confuses the BIO layer. BIO makes simple decisions about bounce pfns and some x86_64 memory layouts cause them to go wrong. Net effect: lots of GFP_DMA allocations in the BIO layer.

http://readlist.com/lists/vger.kernel.org/linux-kernel/36/182357.html
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173

#4 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Chuck Ebbert

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

oom-killer: gfp_mask=0xd1, order=3

This could be related to the known GFP_DMA oom on some x86_64 machines.

I'm not sure if this has any bearing on it, but the OOM Killer only does this when I compile the kernel with SMP support.

drivers/block/floppy.c static void do_fd_request(request_queue_

Quoted 10 lines
Quoted 1 line

Or should floppy.c be fixed so it doesn't ask for so much?

The page allocator uses 32k as the threshold for when-to-try-like-crazy.

x86_64 should probably be defining its own fd_dma_mem_alloc() which doesn't use GFP_DMA.

drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +0 -0
@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_

Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.

--
Chris Largret <http://daga.dyndns.org>

#5 lore raw
Quoted 7 lines · Chris Largret
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Chuck Ebbert

Chris Largret is getting repeated OOM kills because of DMA memory exhaustion:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

oom-killer: gfp_mask=0xd1, order=3

This could be related to the known GFP_DMA oom on some x86_64 machines.

I'm not sure if this has any bearing on it, but the OOM Killer only does this when I compile the kernel with SMP support.

I doubt if that's related.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Or should floppy.c be fixed so it doesn't ask for so much?

The page allocator uses 32k as the threshold for when-to-try-like-crazy.

x86_64 should probably be defining its own fd_dma_mem_alloc() which doesn't use GFP_DMA.

Quoted 9 lines

drivers/block/floppy.c static void do_fd_request(request_queue_

Quoted 3 lines
drivers/block/floppy.c renamed from drivers/block/floppy.c~floppy-false-oom-fix +0 -0
@@ -278,7 +278,8 @@ static void do_fd_request(request_queue_

Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.

hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.

mm/page_alloc.c renamed from mm/page_alloc.c~a +3 -2
@@ -1003,7 +1003,8 @@ rebalance:
 						zonelist, alloc_flags);
 		if (page)
 			goto got_pg;
-	} else if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) {
+	} else if ((gfp_mask & __GFP_FS) &&
+			!(gfp_mask & (__GFP_NORETRY|__GFP_DMA))) {
 		/*
 		 * Go through the zonelist yet one more time, keep
 		 * very high watermark here, this is only to catch
@@ -1027,7 +1028,7 @@ rebalance:
 	 * <= 3, but that may not be true in other implementations.
 	 */
 	do_retry = 0;
-	if (!(gfp_mask & __GFP_NORETRY)) {
+	if (!(gfp_mask & (__GFP_NORETRY|__GFP_DMA))) {
 		if ((order <= 3) || (gfp_mask & __GFP_REPEAT))
 			do_retry = 1;
 		if (gfp_mask & __GFP_NOFAIL)

_

#6 lore raw

mm/page_alloc.c rebalance:

Quoted 12 lines · Andrew Morton
Quoted 4 lines

Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.

hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.

mm/page_alloc.c renamed from mm/page_alloc.c~a +0 -0
@@ -1003,7 +1003,8 @@ rebalance:

I reversed the previous patch before applying this one. If they were supposed to be used together, let me know.

Quoted 1 line

From the initial results it looks like the OOM Killer is not being used

now, Unfortunately I can't check with dmesg because right after login is initiated (but before I get a chance to type anything) there is a "Kernel BUG" message. This is all that is is printed when a serial console is in use. If you need the rest of the information, let me know and I'll see about typing it up.

----------- [cut here ] --------- [please bite here ] ---------

Kernel BUG at mm/vmalloc.c:352
invalid opcode: 0000 [1] SMP 
CPU 1 
Modules linked in: snd_pcm_oss snd_mixer_oss md5 ipv6 ipt_recent

ipt_REJECT xt_state xt_tcpudp iptable_filter ip_tables x_tables nfs lockd nfs_acl sunrpc uhci_hcd r8169 ohci1394 ieee1394 emu10k1_gp gameport snd_emu10k1 snd_rawmidi snd_ac97_codec snd_ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd tda9887 tuner cx8800 cx88xx video_buf ir_common tveeprom compat_ioctl32 v4l1_compat v4l2_common btcx_risc videodev forcedeth usblp ohci_hcd i2c_nforce2 ehci_hc

--
Chris Largret <http://daga.dyndns.org>

#7 lore raw
Quoted 11 lines · Chris Largret
Quoted 9 lines · Andrew Morton
Quoted 4 lines

Sorry, this didn't help on my machine. I am running that latest kernel pre-patch (2.6.16-rc4) for testing right now and had to modify the offsets a little. If there's any output that would help, please let me know.

hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.

Quoted 6 lines

mm/page_alloc.c rebalance:

Quoted 3 lines
mm/page_alloc.c renamed from mm/page_alloc.c~a +0 -0
@@ -1003,7 +1003,8 @@ rebalance:

I reversed the previous patch before applying this one. If they were supposed to be used together, let me know.

No, that's right.

Quoted 12 lines
Quoted 1 line

From the initial results it looks like the OOM Killer is not being used

now, Unfortunately I can't check with dmesg because right after login is initiated (but before I get a chance to type anything) there is a "Kernel BUG" message. This is all that is is printed when a serial console is in use. If you need the rest of the information, let me know and I'll see about typing it up.

----------- [cut here ] --------- [please bite here ] ---------

Kernel BUG at mm/vmalloc.c:352
invalid opcode: 0000 [1] SMP 
CPU 1 
Modules linked in: snd_pcm_oss snd_mixer_oss md5 ipv6 ipt_recent
Quoted 7 lines

ipt_REJECT xt_state xt_tcpudp iptable_filter ip_tables x_tables nfs lockd nfs_acl sunrpc uhci_hcd r8169 ohci1394 ieee1394 emu10k1_gp gameport snd_emu10k1 snd_rawmidi snd_ac97_codec snd_ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd tda9887 tuner cx8800 cx88xx video_buf ir_common tveeprom compat_ioctl32 v4l1_compat v4l2_common btcx_risc videodev forcedeth usblp ohci_hcd i2c_nforce2 ehci_hc

Sigh. The floppy driver's just a jpke. Looks like the failed allocation fell back to vmalloc then screwed it up.

I rather doubt whether x86_64 needs to be constraining itself to the ISA DMA region anyway - something for Andi to look at please?

You could try this one instead, although I guess I'll need to fire up the test box for this bug.

include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1 -1
@@ -40,7 +40,7 @@
 #define fd_disable_irq()        disable_irq(FLOPPY_IRQ)
 #define fd_free_irq()		free_irq(FLOPPY_IRQ, NULL)
 #define fd_get_dma_residue()    SW._get_dma_residue(FLOPPY_DMA)
-#define fd_dma_mem_alloc(size)	SW._dma_mem_alloc(size)
+#define fd_dma_mem_alloc(size)	__alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))
 #define fd_dma_setup(addr, size, mode, io) SW._dma_setup(addr, size, mode, io)
 
 #define FLOPPY_CAN_FALLBACK_ON_NODMA

_

#8 lore raw
Quoted 9 lines · Andrew Morton

Sigh. The floppy driver's just a jpke. Looks like the failed allocation fell back to vmalloc then screwed it up.

You could try this one instead, although I guess I'll need to fire up the test box for this bug.

include/asm-x86_64/floppy.h line 46

Quoted 8 lines
include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1 -1
@@ -40,7 +40,7 @@
 #define fd_disable_irq()        disable_irq(FLOPPY_IRQ)
 #define fd_free_irq()		free_irq(FLOPPY_IRQ, NULL)
 #define fd_get_dma_residue()    SW._get_dma_residue(FLOPPY_DMA)
-#define fd_dma_mem_alloc(size)	SW._dma_mem_alloc(size)
+#define fd_dma_mem_alloc(size)	__alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))
 #define fd_dma_setup(addr, size, mode, io) SW._dma_setup(addr, size, mode, io)
 
 #define FLOPPY_CAN_FALLBACK_ON_NODMA

CC drivers/block/floppy.o
drivers/block/floppy.c: In function `raw_cmd_copyin':
drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'
drivers/block/floppy.c: In function `floppy_open':
drivers/block/floppy.c:3738: error: too few arguments to function `__alloc_pages'
drivers/block/floppy.c:3742: error: too few arguments to function `__alloc_pages'
make[2]: *** [drivers/block/floppy.o] Error 1
make[1]: *** [drivers/block] Error 2
make: *** [drivers] Error 2

I'm sorry, but I'm not sure where to start for looking up the definition for __alloc_pages().

--
Chris Largret <http://daga.dyndns.org>

#9 lore raw
Quoted 3 lines · Chris Largret

drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'

doh.

include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +1 -1
@@ -40,7 +40,7 @@
 #define fd_disable_irq()        disable_irq(FLOPPY_IRQ)
 #define fd_free_irq()		free_irq(FLOPPY_IRQ, NULL)
 #define fd_get_dma_residue()    SW._get_dma_residue(FLOPPY_DMA)
-#define fd_dma_mem_alloc(size)	SW._dma_mem_alloc(size)
+#define fd_dma_mem_alloc(size)	alloc_pages(GFP_KERNEL|__GFP_DMA32, get_order(size))
 #define fd_dma_setup(addr, size, mode, io) SW._dma_setup(addr, size, mode, io)
 
 #define FLOPPY_CAN_FALLBACK_ON_NODMA

_

#10 lore raw

include/asm-x86_64/floppy.h

Quoted 9 lines · Andrew Morton
Quoted 3 lines · Chris Largret

drivers/block/floppy.c:3245: error: too few arguments to function `__alloc_pages'

doh.

include/asm-x86_64/floppy.h renamed from include/asm-x86_64/floppy.h~b +0 -0

Earlier I said that there was a "Kernel BUG" and all processing stopped right after the login prompt was displayed (but before I could type anything). Now the kernel continues to work, but the messages are a little disconcerting. Here is the version with a backtrace (from dmesg):

Bad page state in process 'swapper'
page:ffff810001539168 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:0
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff8100015391a0 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff8100015391d8 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff810001539210 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff810001539248 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff810001539280 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff8100015392b8 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

Bad page state in process 'swapper'
page:ffff8100015392f0 flags:0x4000000000000400 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed

Backtrace:

Call Trace: <IRQ> <ffffffff8104f0cd>{bad_page+85}
<ffffffff8104f419>{__free_pages_ok+179}
       <ffffffff810500da>{__free_pages+49} <ffffffff8105015f>{free_pages
+131}
       <ffffffff8117db99>{_fd_dma_mem_free+43}
<ffffffff81182ca0>{floppy_release_irq_and_dma+243}
       <ffffffff811809fc>{set_dor+323}
<ffffffff81183d50>{motor_off_callback+0}
       <ffffffff81183d74>{motor_off_callback+36}
<ffffffff81034e97>{run_timer_softirq+366}
       <ffffffff810314e5>{__do_softirq+85}
<ffffffff8100bec6>{call_softirq+30}
       <ffffffff8100d92e>{do_softirq+51} <ffffffff8103162a>{irq_exit+72}
       <ffffffff810160b8>{smp_apic_timer_interrupt+75}
<ffffffff81008d0f>{default_idle+0}
       <ffffffff8100b820>{apic_timer_interrupt+132} <EOI>
<ffffffff81008d0f>{default_idle+0}
       <ffffffff81008d3e>{default_idle+47} <ffffffff81008f44>{cpu_idle
+103}
       <ffffffff814390a1>{start_secondary+1189}

--
Chris Largret <http://daga.dyndns.org>

#11 lore raw
In reply to Andrew Morton
Quoted 3 lines

hm, OK. I suppose we can hit it with the big hammer, but I'd be reluctant to merge this patch because it has the potential to hide problems, such as the as-yet-unfixed bio-uses-ZONE_DMA one.

Better would be to fix the block layer. I think something like that would be better: (only lightly tested - it booted on a 6GB x86-64 box)

It is over pessimistic on systems with real IOMMU that can even remap to DMA addresses < 4GB - in the future those might want to define some ARCH_HAS macro so it can be checked here.

Does that patch fix the problem?

That said adding GFP_NORETRY to the floppy allocation is probably still a good idea. I will do that change here.

-Andi

Disable block layer bouncing for most memory on 64bit systems

The low level PCI DMA mapping functions should handle it in most cases.

This should fix problems with depleting the DMA zone early. The old code used precious GFP_DMA memory in many cases where it was not needed.

block/ll_rw_blk.c +18 -12
Index: linux/block/ll_rw_blk.c
===================================================================
@@ -625,26 +625,32 @@ static inline int ordered_bio_endio(stru
  *    Different hardware can have different requirements as to what pages
  *    it can do I/O directly to. A low level driver can call
  *    blk_queue_bounce_limit to have lower memory pages allocated as bounce
- *    buffers for doing I/O to pages residing above @page. By default
- *    the block layer sets this to the highest numbered "low" memory page.
+ *    buffers for doing I/O to pages residing above @page. 
  **/
 void blk_queue_bounce_limit(request_queue_t *q, u64 dma_addr)
 {
 	unsigned long bounce_pfn = dma_addr >> PAGE_SHIFT;
+	int dma = 0;
 
-	/*
-	 * set appropriate bounce gfp mask -- unfortunately we don't have a
-	 * full 4GB zone, so we have to resort to low memory for any bounces.
-	 * ISA has its own < 16MB zone.
-	 */
-	if (bounce_pfn < blk_max_low_pfn) {
+	q->bounce_gfp = GFP_NOIO;
+#if BITS_PER_LONG == 64
+	/* Assume anything >= 4GB can be handled by IOMMU. 
+	   Actually some IOMMUs can handle everything, but I don't
+	   know of a way to test this here. */
+	if (bounce_pfn < (0xffffffff>>PAGE_SHIFT))
+		dma = 1;
+	q->bounce_pfn = max_low_pfn;
+#else
+	if (bounce_pfn < blk_max_low_pfn)
+		dma = 1;
+	q->bounce_pfn = bounce_pfn;
+#endif
+	if (dma) {	
 		BUG_ON(dma_addr < BLK_BOUNCE_ISA);
 		init_emergency_isa_pool();
 		q->bounce_gfp = GFP_NOIO | GFP_DMA;
-	} else
-		q->bounce_gfp = GFP_NOIO;
-
-	q->bounce_pfn = bounce_pfn;
+		q->bounce_pfn = bounce_pfn;
+	}
 }
 
 EXPORT_SYMBOL(blk_queue_bounce_limit);
#12 lore raw
In reply to Andrew Morton

Thinking about this more I think we need a __GFP_NOOOM for other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.

-Andi

#13 lore raw
Quoted 3 lines · Andi Kleen

Thinking about this more I think we need a __GFP_NOOOM for other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.

Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.

#14 lore raw
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andi Kleen

Thinking about this more I think we need a __GFP_NOOOM for other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.

Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.

No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.

Maybe __GFP_NORETRY is actually good enough for this purpose. Opinions?

-Andi

#15 lore raw
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andi Kleen

Thinking about this more I think we need a __GFP_NOOOM for other purposes too. e.g. the x86-64 IOMMU code tries to do similar fallbacks and I suspect it will be hit by the OOM killer too.

Quoted 8 lines
Quoted 2 lines

Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.

No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.

Maybe __GFP_NORETRY is actually good enough for this purpose. Opinions?

I was thinking that your __GFP_NOOOM was a thinko. How would it differ from __GFP_NORETRY?

#16 lore raw
Quoted 6 lines · Andrew Morton
Quoted 6 lines · Christoph Lameter
Quoted 2 lines

Isnt this also a constrained allocation? We could expand the check to also catch these types of restrictions and fail.

No, it uses the full fallback zone list of the target node, not a custom one. Would be hard to detect without a flag.

Right but it specifies in its flags that not all system memory can satisfy this particular memory request. That fact may be detected by the out_of_memory() function. We could do something special there instead of OOMing.

#17 lore raw
In reply to Andrew Morton
Quoted 2 lines

I was thinking that your __GFP_NOOOM was a thinko. How would it differ from __GFP_NORETRY?

__GFP_NORETRY seems to skip at least one retry pass as far as I can see. __GFP_NOOOM wouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.

-Andi

#18 lore raw
Quoted 8 lines · Andi Kleen
Quoted 3 lines

I was thinking that your __GFP_NOOOM was a thinko. How would it differ from __GFP_NORETRY?

__GFP_NORETRY seems to skip at least one retry pass as far as I can see. __GFP_NOOOM wouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.

Oh, OK. That final get_page_from_freelist() is allegedly to see if a parallel oom-killing freed some pages - we already know that try_to_free_pages() didn't work.

I rather doubt that it'll make any difference.

#19 lore raw
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Andi Kleen
Quoted 3 lines

I was thinking that your __GFP_NOOOM was a thinko. How would it differ from __GFP_NORETRY?

__GFP_NORETRY seems to skip at least one retry pass as far as I can see. __GFP_NOOOM wouldn't. But perhaps the additional pass only makes sense with oom killing? I'm not sure - that is why i was asking.

Quoted 5 lines

Oh, OK. That final get_page_from_freelist() is allegedly to see if a parallel oom-killing freed some pages - we already know that try_to_free_pages() didn't work.

I rather doubt that it'll make any difference.

I switched over the x86-64 IOMMU code and floppy code to use __GFP_NORETRY now.

But perhaps it would be better to rename it to __GFP_NOOOM because I think that would express its meaning better.

-Andi