Kernel 2.6.8.1: swap storm of death - nr_requests > 1024 on swap partition

Quoted 6 lines · Andrew Morton

Quoted 4 lines · Karl Vogel

Further testing shows that all the schedulers exhibit this exact same problem when run with a nr_requests size of 8192 on the drive hosting the swap partition.

Quoted 7 lines

Quoted 3 lines

I tried noop, deadline, as and CFQ with:

	echo 8192 >/sys/block/hda/queue/nr_requests

That allows up to 2GB of memory to be under writeout at the same time. The VM cannot touch any of that memory.

Well I used that value as it is the default for CFQ.. and it was with CFQ that I had the problems. The patch Jens offered to track down the problem, commented out this 'q->nr_requests = 8192' in CFQ and it helped. Therefor I tried the other schedulers with this value to see if it made a difference.

So if I understand you correctly, CFQ shouldn't be using 8192 on 512Mb systems?!

With overcommit_memory set to 1, the program can be run again after the OOM kill.. but the OOM killing remains.

With overcommit_memory set to 0 a second run fails. I 'think' it's because somehow SwapCache is 500Kb after the OOM, so in effect my system doesn't have 1Gb to spare anymore. Doing swapoff/swapon frees this and then I can do the calloc(1Gb) again.

Another way to free the SwapCached is to generate lots of I/O doing 'dd if=/dev/hda of=/dev/null' ... after a while SwapCached is < 1Mb again.

Quoted 8 lines

With overcommit_memory set to 1, the program can be run again after the OOM kill.. but the OOM killing remains.
With overcommit_memory set to 0 a second run fails. I 'think' it's because somehow SwapCache is 500Kb after the OOM, so in effect my system doesn't have 1Gb to spare anymore. Doing swapoff/swapon frees this and then I can do the calloc(1Gb) again.
Another way to free the SwapCached is to generate lots of I/O doing 'dd if=/dev/hda of=/dev/null' ... after a while SwapCached is < 1Mb again.

urgh. It sounds like the overcommit logic forgot to account swapcache as reclaimable. It's been a ton of trouble, that code.

Quoted 8 lines · Karl Vogel

Quoted 8 lines

With overcommit_memory set to 1, the program can be run again after the OOM kill.. but the OOM killing remains.
With overcommit_memory set to 0 a second run fails. I 'think' it's because somehow SwapCache is 500Kb after the OOM, so in effect my system doesn't have 1Gb to spare anymore. Doing swapoff/swapon frees this and then I can do the calloc(1Gb) again.
Another way to free the SwapCached is to generate lots of I/O doing 'dd if=/dev/hda of=/dev/null' ... after a while SwapCached is < 1Mb again.

Quoted 2 lines · Andrew Morton

urgh. It sounds like the overcommit logic forgot to account swapcache as reclaimable. It's been a ton of trouble, that code.

For overcommit purposes, swapcache still counts as committed AS; it requires swap as backing store to evict. So AFAICT there isn't an issue there.

I was under the impression this had something to do with IO schedulers.

Quoted 3 lines

For overcommit purposes, swapcache still counts as committed AS; it requires swap as backing store to evict. So AFAICT there isn't an issue there.

But that backing store is allocated?

Quoted 2 lines

I was under the impression this had something to do with IO schedulers.

Separate issue.

Quoted 1 line · Andrew Morton

Separate issue.

It certainly appears to be the deciding factor from the thread.

Quoted 6 lines · William Lee Irwin III

Quoted 1 line · Andrew Morton

Separate issue.

It certainly appears to be the deciding factor from the thread.

It's all very bizarre.

If you do a big `usemem -m 250' on a 256MB box, you end up with all memory in swapcache _after_ usemem exits. That's wrong: all the memory which usemem allocated should now be free.

But all that swapcache is reclaimable under memory pressure. It seems to be floating about on the LRU still.

It only happens with the CFQ elevator, and this backout patch makes it go away.

The main effect of this patch is to increase the elevator's nr_requests from 128 to 8192. Something to do with that, I guess.

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~a +5 -6

@@ -1534,6 +1534,9 @@ request_queue_t *blk_init_queue(request_
 		printk("Using %s io scheduler\n", chosen_elevator->elevator_name);
 	}
 
+	if (elevator_init(q, chosen_elevator))
+		goto out_elv;
+
 	q->request_fn		= rfn;
 	q->back_merge_fn       	= ll_back_merge_fn;
 	q->front_merge_fn      	= ll_front_merge_fn;

@@ -1551,12 +1554,8 @@ request_queue_t *blk_init_queue(request_
 	blk_queue_max_hw_segments(q, MAX_HW_SEGMENTS);
 	blk_queue_max_phys_segments(q, MAX_PHYS_SEGMENTS);
 
-	/*
-	 * all done
-	 */
-	if (!elevator_init(q, chosen_elevator))
-		return q;
-
+	return q;
+out_elv:
 	blk_cleanup_queue(q);
 out_init:
 	kmem_cache_free(requestq_cachep, q);

Quoted 9 lines

It's all very bizarre.

If you do a big `usemem -m 250' on a 256MB box, you end up with all memory in swapcache _after_ usemem exits. That's wrong: all the memory which usemem allocated should now be free.

But all that swapcache is reclaimable under memory pressure. It seems to be floating about on the LRU still.

Quoted 2 lines

It only happens with the CFQ elevator, and this backout patch makes it go away.

It's not bizarre, if you backout that fix (it is a fix!), ->nr_requests isn't initialized when cfq gets there. So it'll throttle incorrectly in may_queue, not a good idea.

Oh, and I think the main issue is the vm. It should cope correctly no matter how much pending memory can be in progress on the queue, else it should not write out so much. CFQ is just exposing this bug because it defaults to bigger nr_requests.

Quoted 4 lines

Quoted 3 lines

It only happens with the CFQ elevator, and this backout patch makes it go away.

Quoted 3 lines

It's not bizarre, if you backout that fix (it is a fix!), ->nr_requests isn't initialized when cfq gets there. So it'll throttle incorrectly in may_queue, not a good idea.

Oh, and I think the main issue is the vm. It should cope correctly no matter how much pending memory can be in progress on the queue, else it should not write out so much. CFQ is just exposing this bug because it defaults to bigger nr_requests.

Quoted 8 lines · Jens Axboe

Quoted 7 lines

Quoted 3 lines

It only happens with the CFQ elevator, and this backout patch makes it go away.

It's not bizarre, if you backout that fix (it is a fix!), ->nr_requests isn't initialized when cfq gets there. So it'll throttle incorrectly in may_queue, not a good idea.

Quoted 4 lines

Oh, and I think the main issue is the vm. It should cope correctly no matter how much pending memory can be in progress on the queue, else it should not write out so much. CFQ is just exposing this bug because it defaults to bigger nr_requests.

That was my point.

Quoted 2 lines

That was my point.

I didn't understand your message at all, maybe that wasn't clear enough in my email :-). You state that the main effect of that particular patch is to bump nr_requests to 8192, which is definitely not true. The main effect of the patch is to make sure that ->nr_requests was valid, so that cfqd->max_queued is valid. ->nr_requests was always overwritten with 8192 for quite some time, irregardless of that patch. So this particular change has nothing to do with that, and other io schedulers will experience exactly this very problem with 8192 requests.

Why you do see a difference is that when ->max_queued isn't valid, you end up block a lot more in get_request_wait() because cfq_may_queue will disallow you to queue a lot more than with the patch. Since other io schedulers don't have these sort of checks, they behave like CFQ does with the bug in blk_init_queue() fixed.

Quoted 5 lines

Why you do see a difference is that when ->max_queued isn't valid, you end up block a lot more in get_request_wait() because cfq_may_queue will disallow you to queue a lot more than with the patch. Since other io schedulers don't have these sort of checks, they behave like CFQ does with the bug in blk_init_queue() fixed.

The changlog wasn't that detailed ;)
But yes, it's the large nr_requests which is tripping up swapout. I'm assuming that when a process exits with its anonymous memory still under swap I/O we're forgetting to actually free the pages when the I/O completes. So we end up with a ton of zero-ref swapcache pages on the LRU.
I assume. Something odd's happening, that's for sure.

Quoted 6 lines · Andrew Morton

The changlog wasn't that detailed ;)
But yes, it's the large nr_requests which is tripping up swapout. I'm assuming that when a process exits with its anonymous memory still under swap I/O we're forgetting to actually free the pages when the I/O completes. So we end up with a ton of zero-ref swapcache pages on the LRU.
I assume. Something odd's happening, that's for sure.

Maybe we need to be checking for this in end_swap_bio_write() or rotate_reclaimable_page()?

Quoted 8 lines · William Lee Irwin III

Quoted 6 lines · Andrew Morton

The changlog wasn't that detailed ;)
But yes, it's the large nr_requests which is tripping up swapout. I'm assuming that when a process exits with its anonymous memory still under swap I/O we're forgetting to actually free the pages when the I/O completes. So we end up with a ton of zero-ref swapcache pages on the LRU.
I assume. Something odd's happening, that's for sure.

Quoted 2 lines

Maybe we need to be checking for this in end_swap_bio_write() or rotate_reclaimable_page()?

Maybe. I thought a get_page() in swap_writepage() and a put_page() in end_swap_bio_write() would cause the page to be freed. But not. It needs some actual real work done on it.

Quoted 12 lines · Jens Axboe

Quoted 2 lines

That was my point.

I didn't understand your message at all, maybe that wasn't clear enough in my email :-). You state that the main effect of that particular patch is to bump nr_requests to 8192, which is definitely not true. The main effect of the patch is to make sure that ->nr_requests was valid, so that cfqd->max_queued is valid. ->nr_requests was always overwritten with 8192 for quite some time, irregardless of that patch. So this particular change has nothing to do with that, and other io schedulers will experience exactly this very problem with 8192 requests.

Quoted 5 lines

Why you do see a difference is that when ->max_queued isn't valid, you end up block a lot more in get_request_wait() because cfq_may_queue will disallow you to queue a lot more than with the patch. Since other io schedulers don't have these sort of checks, they behave like CFQ does with the bug in blk_init_queue() fixed.

The changlog wasn't that detailed ;)

But yes, it's the large nr_requests which is tripping up swapout. I'm assuming that when a process exits with its anonymous memory still under swap I/O we're forgetting to actually free the pages when the I/O completes. So we end up with a ton of zero-ref swapcache pages on the LRU.

I assume. Something odd's happening, that's for sure.

What is the problem Karl is seeing again? There seem to be several, lets separate them

- OOM killer triggering (if there's swap space available and
"enough" anonymous memory to be swapped out this should not happen). One of his complaint on the initial report (about the OOM killer).

- Swap cache not freed after test app exists. Should not be a problem because such memory will be freed as soon as theres pressure, I think.

How can you reproduce that?

Quoted 6 lines · Marcelo Tosatti

What is the problem Karl is seeing again? There seem to be several, lets separate them

- OOM killer triggering (if there's swap space available and "enough" anonymous memory to be swapped out this should not happen). One of his complaint on the initial report (about the OOM killer).

Correct. On my 512Mb RAM system with 1Gb swap partition, running a calloc(1Gb) causes the process to get OOM killed when using CFQ. The problem is not CFQ as such.. the problem is when nr_requests is too large (8192 being the default for CFQ).

The same will happen with the default nr_request of 128 which AS uses, if you use a low memory system. e.g. I booted with mem=128M and then a calloc(128Mb) can trigger the OOM.

Quoted 3 lines

- Swap cache not freed after test app exists. Should not be a problem because such memory will be freed as soon as theres pressure, I think.

After the OOM killer killed the calloc() task, the SwapCache still contains a large chunk of the original allocation. This get's cleared if there is alot of I/O (example: dd if=/dev/hdX of=/dev/null).

However, without the I/O's it doesn't seem to get freed.. this also causes a second run of calloc(1Gb) to fail as the SwapCache still accounts for used memory.

Quoted 1 line

How can you reproduce that?

It should be reproducable as follows:
- boot with mem=512M
- have a 1Gb swap partition / swapfile (the size doesn't really matter)
- use CFQ or set nr_requests to 8192 on the drive _hosting the swap_
- run 'expunge 1024' (might work the 1st time, if so, run it again)

--- expunge.c program source ---
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])

{
    char *p= calloc(1, atol(argv[1])*1024L*1024L);
    if (!p) {
        perror("calloc");
        exit(1);
    }
    return 0;
}

--- expunge.c program source ---

Another thing that you can try:
- boot with mem=128M
- have enough swap
- execute: while true; do expunge 128; done

This will trigger an OOM even with AS (nr_requests = 128)

After the OOM, SwapCache still holds part of the allocation.

Karl,

Please apply the attached patch and rerun your tests. With it applied, the OOM killer output will print the number of available swap pages at the time of killing.

Quoted 1 line

Quoted 1 line

lots of these cut from mail

Quoted 6 lines · Marcelo Tosatti

What is the problem Karl is seeing again? There seem to be several, lets separate them

- OOM killer triggering (if there's swap space available and "enough" anonymous memory to be swapped out this should not happen). One of his complaint on the initial report (about the OOM killer).

Correct. On my 512Mb RAM system with 1Gb swap partition, running a calloc(1Gb) causes the process to get OOM killed when using CFQ. The problem is not CFQ as such.. the problem is when nr_requests is too large (8192 being the default for CFQ).

The same will happen with the default nr_request of 128 which AS uses, if you use a low memory system. e.g. I booted with mem=128M and then a calloc(128Mb) can trigger the OOM.

Review

Quoted 9 lines · Marcelo Tosatti

static int may_write_to_queue(struct backing_dev_info *bdi) {

(quoted diff) +6 -0

+	int nr_writeback = read_page_state(nr_writeback);
+
+	if (nr_writeback > (totalram_pages * 25 / 100)) { 
+		blk_congestion_wait(WRITE, HZ/5);
+		return 0;
+	}

That's probably a good way of special-casing this special-place problem.

For a final patch I'd be inclined to take into account /proc/sys/vm/dirty_ratio and to avoid running the expensive read_page_state() once per writepage.

What you think of this, which tries to address your comments

Quoted 2 lines · Marcelo Tosatti

What you think of this, which tries to address your comments

Suggest you pass the scan_control structure down into pageout(), stick `inflight' into struct scan_control and use some flag in scan_control to

ensure that we only throttle once per try_to_free_pages()/blaance_pgdat() pass.

See, page reclaim is now, as much as possible, "batched". Think of it as operating in units of 32 pages at a time. We should only examine the dirty memory thresholds and throttle once per "batch", not once per page.

Quoted 6 lines · Andrew Morton

Quoted 2 lines · Marcelo Tosatti

What you think of this, which tries to address your comments

Suggest you pass the scan_control structure down into pageout(), stick `inflight' into struct scan_control and use some flag in scan_control to

Done the scan_control modifications.

Quoted 2 lines

ensure that we only throttle once per try_to_free_pages()/blaance_pgdat() pass.

Throttling once is enough

I added a

vmscan.c renamed from vmscan.c.orig +4 -0

+		 if (sc->throttled < 5) {
+                       blk_congestion_wait(WRITE, HZ/5);
+                       sc->throttled++;
+               }

To loop five times max per try_to_free_pages()/balance_pgdat().

Because only one blk_congestion_wait(WRITE, HZ/5)
makes my 64MB boot testcase with 8192 nr_requests fail. The OOM killer triggers prematurely.

Quoted 3 lines

See, page reclaim is now, as much as possible, "batched". Think of it as operating in units of 32 pages at a time. We should only examine the dirty memory thresholds and throttle once per "batch", not once per page.

That should do it

vmscan.c renamed from vmscan.c.orig +31 -4

@@ -73,6 +73,10 @@
 	unsigned int gfp_mask;
 
 	int may_writepage;
+
+	int inflight;
+
+	int throttled; /* how many times have we throttled on VM inflight IO limit */
 };
 
 /*

@@ -245,8 +249,30 @@
 	return page_count(page) - !!PagePrivate(page) == 2;
 }
 
-static int may_write_to_queue(struct backing_dev_info *bdi)
+/*
+ * This function calculates the maximum pinned-for-IO memory
+ * the page eviction threads can generate. If we hit the max,
+ * we throttle taking a nap.
+ *
+ * Returns true if we cant writeout.
+ */
+int vm_eviction_limits(struct scan_control *sc)
+{
+        if (sc->inflight > (totalram_pages * vm_dirty_ratio) / 100)  {
+		if (sc->throttled < 5) {
+			blk_congestion_wait(WRITE, HZ/5);
+			sc->throttled++;
+		}
+                return 1;
+        }
+        return 0;
+}
+
+static int may_write_to_queue(struct backing_dev_info *bdi, struct scan_control *sc)
 {
+	if (vm_eviction_limits(sc)) /* Check VM writeout limit */
+		return 0;
+
 	if (current_is_kswapd())
 		return 1;
 	if (current_is_pdflush())	/* This is unlikely, but why not... */

@@ -286,7 +312,7 @@
 /*
  * pageout is called by shrink_list() for each dirty page. Calls ->writepage().
  */
-static pageout_t pageout(struct page *page, struct address_space *mapping)
+static pageout_t pageout(struct page *page, struct address_space *mapping, struct scan_control *sc)
 {
 	/*
 	 * If the page is dirty, only perform writeback if that write

@@ -311,7 +337,7 @@
 		return PAGE_KEEP;
 	if (mapping->a_ops->writepage == NULL)
 		return PAGE_ACTIVATE;
-	if (!may_write_to_queue(mapping->backing_dev_info))
+	if (!may_write_to_queue(mapping->backing_dev_info, sc))
 		return PAGE_KEEP;
 
 	if (clear_page_dirty_for_io(page)) {

@@ -421,7 +447,7 @@
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */
-			switch(pageout(page, mapping)) {
+			switch(pageout(page, mapping, sc)) {
 			case PAGE_KEEP:
 				goto keep_locked;
 			case PAGE_ACTIVATE:

@@ -807,6 +833,7 @@
 		nr_inactive = 0;
 
 	sc->nr_to_reclaim = SWAP_CLUSTER_MAX;
+	sc->throttled = 0;
 
 	while (nr_active || nr_inactive) {
 		if (nr_active) {

Quoted 5 lines · Andrew Morton

Quoted 1 line · Marcelo Tosatti

What you think of this, which tries to address your comments

Suggest you pass the scan_control structure down into pageout(), stick `inflight' into struct scan_control and use some flag in scan_control to

Done the scan_control modifications.

Quoted 8 lines · Marcelo Tosatti

Quoted 5 lines · Andrew Morton

Quoted 1 line · Marcelo Tosatti

What you think of this, which tries to address your comments

Suggest you pass the scan_control structure down into pageout(), stick `inflight' into struct scan_control and use some flag in scan_control to

Done the scan_control modifications.

Took the patch for a spin.. it seems to behave ok here! No more OOMs.

Quick question: is it to be expected that when I run a calloc(500Mb) on my system, when X is up and amarok is streaming live audio, that everything (apps) freezes for a few seconds until the calloc task exits?! The apps probably get pushed out to swap, but I would think that since these applications are running, that their pages are kept on the active list?! Setting swappiness to 0 doesn't make a difference.

Is there a concept of a minimum working set size of an application? (kind of the reverse of an RSS limit)

Quoted 2 lines

Is there a concept of a minimum working set size of an application? (kind of the reverse of an RSS limit)

Not really. A hungry memory app can starve the rest of the system.

One thing: what kernel version are you using?

I've seen extreme decreases in performance (interactivity) with hungry memory apps with Rik's swap token code.

Quoted 4 lines · Marcelo Tosatti

Quoted 2 lines

Is there a concept of a minimum working set size of an application? (kind of the reverse of an RSS limit)

Not really. A hungry memory app can starve the rest of the system.

I noticed that a few times on our spamassassin box :-)

Quoted 1 line

One thing: what kernel version are you using?

2.6.9-rc1-bk3

Quoted 2 lines

I've seen extreme decreases in performance (interactivity) with hungry memory apps with Rik's swap token code.

Decrease?!

Can you try the same tests with 2.6.8.1 and check the difference, pretty please?

Quoted 5 lines · Karl Vogel

Quoted 2 lines · Marcelo Tosatti

I've seen extreme decreases in performance (interactivity) with hungry memory apps with Rik's swap token code.

Decrease?!

Yep, its odd. Rik knows the exact reason.

Can you please try the following - it limits the number of in-flight writeback pages to 25% of total RAM at the VM level.

Does wonders for me with 8192 nr_requests. The hogs finish _much_ faster and and interactivity feels much better.

With nr_requests=128, this limit is not reached (probably never), but with 8192, it certainly does.

mm/vmscan.c +0 -0

@@ -247,6 +247,12 @@
 
 static int may_write_to_queue(struct backing_dev_info *bdi)

Quoted 2 lines

Quoted 2 lines

I was under the impression this had something to do with IO schedulers.

Quoted 1 line · Andrew Morton

Separate issue.

It certainly appears to be the deciding factor from the thread.

Quoted 1 line

It certainly appears to be the deciding factor from the thread.

Has nothing to do with the io scheduler itself, apart from the fact that CFQ exposes the problem by setting a larger q->nr_requests. And that is the very deciding factor, not the io scheduler.

Quoted 1 line

So if I understand you correctly, CFQ shouldn't be using 8192 on 512Mb systems?!

Yup. It's asking for trouble to allow that much memory to be unreclaimably pinned.

Of course, you could have the same problem with just 128 requests per queue, and lots of queues. I solved all these problems in the dirty memory writeback paths. But I forgot about swapout!

Quoted 6 lines

With overcommit_memory set to 1, the program can be run again after the OOM kill.. but the OOM killing remains.

With overcommit_memory set to 0 a second run fails. I 'think' it's because somehow SwapCache is 500Kb after the OOM, so in effect my system doesn't have 1Gb to spare anymore. Doing swapoff/swapon frees this and then I can do the calloc(1Gb) again.

Another way to free the SwapCached is to generate lots of I/O doing 'dd if=/dev/hda of=/dev/null' ... after a while SwapCached is < 1Mb again.

urgh. It sounds like the overcommit logic forgot to account swapcache as reclaimable. It's been a ton of trouble, that code.

(Added linux-mm)

Quoted 22 lines · Karl Vogel

Quoted 6 lines · Andrew Morton

Quoted 4 lines · Karl Vogel

Further testing shows that all the schedulers exhibit this exact same problem when run with a nr_requests size of 8192 on the drive hosting the swap partition.

Quoted 7 lines

Quoted 3 lines

I tried noop, deadline, as and CFQ with:

	echo 8192 >/sys/block/hda/queue/nr_requests

That allows up to 2GB of memory to be under writeout at the same time. The VM cannot touch any of that memory.

Well I used that value as it is the default for CFQ.. and it was with CFQ that I had the problems. The patch Jens offered to track down the problem, commented out this 'q->nr_requests = 8192' in CFQ and it helped. Therefor I tried the other schedulers with this value to see if it made a difference.

Quoted 2 lines

So if I understand you correctly, CFQ shouldn't be using 8192 on 512Mb systems?!

Yup. It's asking for trouble to allow that much memory to be unreclaimably pinned.

Of course, you could have the same problem with just 128 requests per queue, and lots of queues. I solved all these problems in the dirty memory writeback paths. But I forgot about swapout!