Skip to contentSkip to conversation

RFC for 2.6: avoid OOM at bounce buffer storm

PATCH Martin Wilck 2 participants 8 messages Jun 3, 2005 – Jun 14, 2005

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello,

I have recently seen massive problems with the bounce buffer allocation implementation in 2.6. My system got OOM with an ata_piix controller (4GB bounce limit), 2 P4 CPUs and 8GB of RAM with a very simple copy-compare test: About 800MB worth of data are copied with cp(1) in 7 synchronous jobs. Each job does a sync(1) after the copy.

All available ZONE_NORMAL memory is filled with bounce buffers in fractions of a second after the sync(1) calls are issued. The system goes OOM (becomes unusable, OOM killer strikes several times against innocent processes).

The attached quick-and-dirty patch gives you an idea of what I figured could be a useful workaround for that problem. It follows 2 main ideas:

1 When bounce buffer allocations fail, it is wrong to trigger the page
reclaim mechanism which is likely to genetate even more bounce buffer requests. Thus there should be no wakeup_bdflush() call for the page_pool.

2 bounce buffer allocations should not simply use alloc_page() because
there is no limit for allocations that way. Rather, bounce buffer allocations should stop if ZONE_NORMAL is full to a certain degree (my patch stops at 51% - chosen arbitrarily).

For the case I described, the system behavior is clearly improved with the patch. The system remains usable, and the OOM killer isn't triggered.

(The patch is against an RH enterprise kernel, but it applies against almost all recent 2.6.kernels except 2.6.12-rcX).

Idea 1) is already implemented in a recent patch by Nick Piggin which went into 2.6.12-rc5. I haven't tested yet if that patch alone fixes my problem because I couldn't get 2.6.12-rc5 to run on my system. I doubt so, though, because I started with idea 1) alone and it didn't help much.

Here is another idea how the bounce buffer behavior could be enhanced:
Introduce a new memory zone ZONE_4G with memory from 896M-4096M
and allocate bounce buffers preferably in that zone. Advantage: Bounce buffers would't interfere with valuable ZONE_NORMAL memory. But that may be too much effort just for systems which have too much memory for their IO controllers...

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
arch/i386/mm/pgtable.c +2 -0
diff -rupX /root/diff-exludes linux-2.6.9-6.37.EL-orig/arch/i386/mm/pgtable.c linux-2.6.9-6.37.EL/arch/i386/mm/pgtable.c
@@ -24,6 +24,7 @@
 #include <asm/tlbflush.h>
 #include <asm/atomic_kmap.h>
 
+extern atomic_t n_bounce_pages;
 void show_mem(void)
 {
 	int total = 0, reserved = 0;
@@ -55,6 +56,7 @@ void show_mem(void)
 	printk("%d reserved pages\n",reserved);
 	printk("%d pages shared\n",shared);
 	printk("%d pages swap cached\n",cached);
+	printk("%d bounce buffers used\n",atomic_read(&n_bounce_pages));
 }
 
 EXPORT_SYMBOL_GPL(show_mem);
mm/highmem.c +39 -6

88 lines not on this page · raw message

mm/mempool.c +5 -2
diff -rupX /root/diff-exludes linux-2.6.9-6.37.EL-orig/mm/mempool.c linux-2.6.9-6.37.EL/mm/mempool.c
@@ -15,6 +15,8 @@
 #include <linux/blkdev.h>
 #include <linux/writeback.h>
 
+extern mempool_t *page_pool, *isa_page_pool;
+
 static void add_element(mempool_t *pool, void *element)
 {
 	BUG_ON(pool->curr_nr >= pool->min_nr);
@@ -211,11 +213,12 @@ repeat_alloc:
 		if (likely(element != NULL))
 			return element;
 	}
-
+	
 	/*
 	 * Kick the VM at this point.
 	 */
-	wakeup_bdflush(0);
+	if (pool != page_pool && pool != isa_page_pool)
+		wakeup_bdflush(0);
 
 	spin_lock_irqsave(&pool->lock, flags);
 	if (likely(pool->curr_nr)) {
#1 lore raw
Quoted 8 lines · Martin Wilck

Hello,

I have recently seen massive problems with the bounce buffer allocation implementation in 2.6. My system got OOM with an ata_piix controller (4GB bounce limit), 2 P4 CPUs and 8GB of RAM with a very simple copy-compare test: About 800MB worth of data are copied with cp(1) in 7 synchronous jobs. Each job does a sync(1) after the copy.

Fair enough.

Quoted 10 lines

...

The attached quick-and-dirty patch gives you an idea of what I figured could be a useful workaround for that problem. It follows 2 main ideas:

1 When bounce buffer allocations fail, it is wrong to trigger the page
reclaim mechanism which is likely to genetate even more bounce buffer requests. Thus there should be no wakeup_bdflush() call for the page_pool.

Quoted 4 lines

2 bounce buffer allocations should not simply use alloc_page() because
there is no limit for allocations that way. Rather, bounce buffer allocations should stop if ZONE_NORMAL is full to a certain degree (my patch stops at 51% - chosen arbitrarily).

It might be neater to do this at the mempool level: that way we're adding general-purpose infrastructure and then just using it, rather than special-casing the bounce code.

See below a (n untested) patch against the latest devel tree. It won't be stunningly scalable on big SMP, but the overhead of bouncing will probably hide that.

Quoted 1 line

I couldn't get 2.6.12-rc5 to run on my system.

Ow. Could you please investigate further? Any boot messages for us to see? it's quite possibly some missing config option..

 include/linux/mempool.h |   17 +++++++++++++++--
 mm/highmem.c            |    4 +++-
 mm/mempool.c            |   17 +++++++++++++----
 3 files changed, 31 insertions(+), 7 deletions(-)
include/linux/mempool.h renamed from include/linux/mempool.h~mempool-bounce-buffer-restriction +15 -2
diff -puN include/linux/mempool.h~mempool-bounce-buffer-restriction include/linux/mempool.h
@@ -5,6 +5,7 @@
 #define _LINUX_MEMPOOL_H
 
 #include <linux/wait.h>
+#include <asm/semaphore.h>
 
 typedef void * (mempool_alloc_t)(unsigned int __nocast gfp_mask, void *pool_data);
 typedef void (mempool_free_t)(void *element, void *pool_data);
@@ -14,14 +15,26 @@ typedef struct mempool_s {
 	int min_nr;		/* nr of elements at *elements */
 	int curr_nr;		/* Current nr of elements at *elements */
 	void **elements;
+	int limit;
 
 	void *pool_data;
 	mempool_alloc_t *alloc;
 	mempool_free_t *free;
 	wait_queue_head_t wait;
+	struct semaphore limit_sem;
 } mempool_t;
-extern mempool_t * mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
-				 mempool_free_t *free_fn, void *pool_data);
+
+extern mempool_t *mempool_create_restricted(int min_nr,
+	mempool_alloc_t *alloc_fn, mempool_free_t *free_fn,
+	void *pool_data, int limit);
+
+static inline mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
+			mempool_free_t *free_fn, void *pool_data)
+{
+	return mempool_create_restricted(min_nr, alloc_fn, free_fn,
+					pool_data, 0);
+}
+
 extern int mempool_resize(mempool_t *pool, int new_min_nr, unsigned int __nocast gfp_mask);
 extern void mempool_destroy(mempool_t *pool);
 extern void * mempool_alloc(mempool_t *pool, unsigned int __nocast gfp_mask);
mm/highmem.c renamed from mm/highmem.c~mempool-bounce-buffer-restriction +3 -1
diff -puN mm/highmem.c~mempool-bounce-buffer-restriction mm/highmem.c
@@ -224,7 +224,9 @@ static __init int init_emergency_pool(vo
 	if (!i.totalhigh)
 		return 0;
 
-	page_pool = mempool_create(POOL_SIZE, page_pool_alloc, page_pool_free, NULL);
+	page_pool = mempool_create_restricted(POOL_SIZE, page_pool_alloc,
+				page_pool_free, NULL,
+				200 * 1024 * 1024 / PAGE_SIZE);
 	if (!page_pool)
 		BUG();
 	printk("highmem bounce pool size: %d pages\n", POOL_SIZE);
mm/mempool.c renamed from mm/mempool.c~mempool-bounce-buffer-restriction +13 -4
diff -puN mm/mempool.c~mempool-bounce-buffer-restriction mm/mempool.c
@@ -38,12 +38,13 @@ static void free_pool(mempool_t *pool)
 }
 
 /**
- * mempool_create - create a memory pool
+ * mempool_create_restricted - create a memory pool
  * @min_nr:    the minimum number of elements guaranteed to be
  *             allocated for this pool.
  * @alloc_fn:  user-defined element-allocation function.
  * @free_fn:   user-defined element-freeing function.
  * @pool_data: optional private data available to the user-defined functions.
+ * @limit:     maximum number of in-flight objects
  *
  * this function creates and allocates a guaranteed size, preallocated
  * memory pool. The pool can be used from the mempool_alloc and mempool_free
@@ -51,8 +52,8 @@ static void free_pool(mempool_t *pool)
  * functions might sleep - as long as the mempool_alloc function is not called
  * from IRQ contexts.
  */
-mempool_t * mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
-				mempool_free_t *free_fn, void *pool_data)
+mempool_t *mempool_create_restricted(int min_nr, mempool_alloc_t *alloc_fn,
+			mempool_free_t *free_fn, void *pool_data, int limit)
 {
 	mempool_t *pool;
 
@@ -71,6 +72,8 @@ mempool_t * mempool_create(int min_nr, m
 	init_waitqueue_head(&pool->wait);
 	pool->alloc = alloc_fn;
 	pool->free = free_fn;
+	pool->limit = limit;
+	sema_init(&pool->limit_sem, limit);
 
 	/*
 	 * First pre-allocate the guaranteed number of buffers.
@@ -87,7 +90,7 @@ mempool_t * mempool_create(int min_nr, m
 	}
 	return pool;
 }
-EXPORT_SYMBOL(mempool_create);
+EXPORT_SYMBOL(mempool_create_restricted);
 
 /**
  * mempool_resize - resize an existing memory pool
@@ -208,6 +211,9 @@ void * mempool_alloc(mempool_t *pool, un
 
 	gfp_temp = gfp_mask & ~(__GFP_WAIT|__GFP_IO);
 
+	if (unlikely(pool->limit))
+		down(&pool->limit_sem);
+
 repeat_alloc:
 
 	element = pool->alloc(gfp_temp, pool->pool_data);
@@ -250,6 +256,9 @@ void mempool_free(void *element, mempool
 {
 	unsigned long flags;
 
+	if (unlikely(pool->limit))
+		up(&pool->limit_sem);
+
 	smp_mb();
 	if (pool->curr_nr < pool->min_nr) {
 		spin_lock_irqsave(&pool->lock, flags);

_

#2 lore raw
Quoted 7 lines · Andrew Morton

It might be neater to do this at the mempool level: that way we're adding general-purpose infrastructure and then just using it, rather than special-casing the bounce code.

See below a (n untested) patch against the latest devel tree. It won't be stunningly scalable on big SMP, but the overhead of bouncing will probably hide that.

I don't quite understand your patch. You introduce a "limit" field but you never actually use it. You also don't count the allocated pages. Are you using the semaphore for slowing things down on purpose?

(Note that the problem is not in the mempool allocation itself but in the "normal" allocation path (page_pool_alloc() -> alloc_page()))

Anyway, I think could figure out your patch but with 2.6.12-rc5-mm2 I couldn't reproduce the problem any more. It appears to run much more smoothly now, perhaps because wakeup_bdflush() isn't called any more. Are you still interested in more data?

Quoted 4 lines
Quoted 1 line

I couldn't get 2.6.12-rc5 to run on my system.

Ow. Could you please investigate further? Any boot messages for us to see? it's quite possibly some missing config option..

It turned out to be a problem with Red Hat's nash that didn't check the returned pid in it's wait4() call and thus ended up insmod'ing mutliple modules simultaneously, leading to "Unkown symbol" errors. Yuck, it took me a day figure that out.

That bug is fixed in redhat's "mkinitrd" package 4.2.0.3-1 and later, but that package is currently only in Fedora's "Development" tree.

Thanks,
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#3 lore raw
Quoted 5 lines · Martin Wilck
Quoted 5 lines

It might be neater to do this at the mempool level: that way we're adding general-purpose infrastructure and then just using it, rather than special-casing the bounce code.

Quoted 7 lines
Quoted 3 lines

See below a (n untested) patch against the latest devel tree. It won't be stunningly scalable on big SMP, but the overhead of bouncing will probably hide that.

I don't quite understand your patch. You introduce a "limit" field but you never actually use it. You also don't count the allocated pages. Are you using the semaphore for slowing things down on purpose?

The semaphore is initialised with the limit level, so once it has been down()ed more than `limit' times, processes will block until someone does up().

Quoted 2 lines

(Note that the problem is not in the mempool allocation itself but in the "normal" allocation path (page_pool_alloc() -> alloc_page()))

yup. The semaphore will prevent more than `limit' pages being allocated at any point in time.

Quoted 2 lines

Anyway, I think could figure out your patch but with 2.6.12-rc5-mm2 I couldn't reproduce the problem any more.

Oh bugger.

Quoted 3 lines

It appears to run much more
smoothly now, perhaps because wakeup_bdflush() isn't called any more. Are you still interested in more data?

Perhaps the newer kernel has writeback thresholding fixes so it's not possible to dirty as much memory with write().

You can probably trigger the same problem if the memory is instead dirtied with mmap(MAP_SHARED).

#4 lore raw

Hello Andrew,

Quoted 3 lines

The semaphore is initialised with the limit level, so once it has been down()ed more than `limit' times, processes will block until someone does up().

Oh - of course. Neat.

Quoted 4 lines
Quoted 1 line

It appears to run much more smoothly now, perhaps because wakeup_bdflush() isn't called any more. Are you still interested in more data?

Perhaps the newer kernel has writeback thresholding fixes so it's not possible to dirty as much memory with write().

I have collected more data and the behavior with 2.6.12-rc5-mm2 is flawless, there is a continuous writeback flow close to the maximum rate possible, and the bounce buffer usage never gets anywhere near the limit where it'd become dangerous. At least not in my test setup. The latest fedora kernel 2.6.11-1.27 also behaves ok, although it doesn't adapt to changing io load as smoothly as 2.6.12-rc5-mm2 does, and the writeback rate is oscillating more strongly.

The kernels where I observe the problem are 2.6.9 kernels from RedHat EL4. I have posted this here because I saw that the highmem bounce buffer/memory pool implementation was identical between the 2.6.9 kernel and all but the very latest development kernels, and I concluded prematurely that the behavior under my scenario must also be the same -- which it wasn't. I apologize for not having looked more closely.

Many thanks for looking into this anyway. From a theoretical point of view, I still think I had a valid point :-/.

Your patch sure looks good to me.

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#5 lore raw
Quoted 31 lines · Martin Wilck

Hello Andrew,

Quoted 3 lines

The semaphore is initialised with the limit level, so once it has been down()ed more than `limit' times, processes will block until someone does up().

Oh - of course. Neat.

Quoted 6 lines
Quoted 3 lines

It appears to run much more
smoothly now, perhaps because wakeup_bdflush() isn't called any more. Are you still interested in more data?

Perhaps the newer kernel has writeback thresholding fixes so it's not possible to dirty as much memory with write().

I have collected more data and the behavior with 2.6.12-rc5-mm2 is flawless, there is a continuous writeback flow close to the maximum rate possible, and the bounce buffer usage never gets anywhere near the limit where it'd become dangerous. At least not in my test setup. The latest fedora kernel 2.6.11-1.27 also behaves ok, although it doesn't adapt to changing io load as smoothly as 2.6.12-rc5-mm2 does, and the writeback rate is oscillating more strongly.

The kernels where I observe the problem are 2.6.9 kernels from RedHat EL4. I have posted this here because I saw that the highmem bounce buffer/memory pool implementation was identical between the 2.6.9 kernel and all but the very latest development kernels, and I concluded prematurely that the behavior under my scenario must also be the same -- which it wasn't. I apologize for not having looked more closely.

Quoted 4 lines

Many thanks for looking into this anyway. From a theoretical point of view, I still think I had a valid point :-/.

Your patch sure looks good to me.

Well. As I said, I think what you're seeing here is recent changes to mm/page-writeback.c which reduce the amount of memory which we'll permit to be dirtied due to write() calls. You'll probably find that the bounce buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in 2.6.9, for the same reasons.

What concerns me is that there are other ways of dirtying lots of memory apart from write(): namely mmap(MAP_SHARED). If someone dirties 90% of all memory via mmap() then we might again get into bounce buffer starvation.

#6 lore raw

Hi Andrew,

Quoted 9 lines

Well. As I said, I think what you're seeing here is recent changes to mm/page-writeback.c which reduce the amount of memory which we'll permit to be dirtied due to write() calls. You'll probably find that the bounce buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in 2.6.9, for the same reasons.

What concerns me is that there are other ways of dirtying lots of memory apart from write(): namely mmap(MAP_SHARED). If someone dirties 90% of all memory via mmap() then we might again get into bounce buffer starvation.

I have tried the mmap(MAP_SHARED) method now extensively. I haven't been able to come anywhere near the catastrophic situations I saw with the 2.6.9 kernel, even by dirtying the full 8GB in fractions of a second.

There was another strangeness there though: Even with the high memory pressure applied, The ZONE_NORMAL free memory would never go below ~300MB. When the mem pressure got too high, the kernel would rather free almost slabs and start swapping than use those remaining 300M. It seems to me that the new logic is a bit too conservative with ZONE_NORMAL allocations.

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#7 lore raw
Quoted 9 lines · Martin Wilck

Hi Andrew,

Quoted 6 lines

Well. As I said, I think what you're seeing here is recent changes to mm/page-writeback.c which reduce the amount of memory which we'll permit to be dirtied due to write() calls. You'll probably find that the bounce buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in 2.6.9, for the same reasons.

Quoted 7 lines
Quoted 3 lines

What concerns me is that there are other ways of dirtying lots of memory apart from write(): namely mmap(MAP_SHARED). If someone dirties 90% of all memory via mmap() then we might again get into bounce buffer starvation.

I have tried the mmap(MAP_SHARED) method now extensively. I haven't been able to come anywhere near the catastrophic situations I saw with the 2.6.9 kernel, even by dirtying the full 8GB in fractions of a second.

OK, thanks.

Quoted 6 lines

There was another strangeness there though: Even with the high memory pressure applied, The ZONE_NORMAL free memory would never go below ~300MB. When the mem pressure got too high, the kernel would rather free almost slabs and start swapping than use those remaining 300M. It seems to me that the new logic is a bit too conservative with ZONE_NORMAL allocations.

Yes, we reserve lots of ZONE_NORMAL memory when performing GFP_HIGHMEM allocations to avoid a weird corner case in which all of the lowmem memory is pinned down because some application mlocked a lot of memory.

You can do

echo 1000 > /proc/sys/vm/lowmem_reserve_ratio

to get the old behaviour back (I do this all the time).