Skip to contentSkip to conversation

new OOM heuristic failure (was: Re: VM: qsbench)

DISCUSSION Lorenzo Allegrucci 3 participants 7 messages Nov 1, 2001 – Nov 2, 2001 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 35 lines · you
Quoted 2 lines
Quoted 1 line · Stephan von Krawczynski

On Wed, 31 Oct 2001 22:31:40 +0100 Lorenzo Allegrucci

<[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines · wrote:

Linus,

your patch seems to help one case out of three. (even though I have not any meaningful statistical data)

Hm, I will not say that I expected that :-), he knows by far more

than me.

Quoted 1 line
Quoted 1 line

But can you try my patch below in addition or comparison to linus'

?

Quoted 12 lines
Quoted 1 line

Give me a hint what happens.

Well, your patch works but it hurts performance :(

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.500u 1.790s 2:29.18 49.1% 0+0k 0+0io 18498pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.460u 1.990s 2:26.87 50.0% 0+0k 0+0io 18257pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.220u 2.200s 2:26.82 50.0% 0+0k 0+0io 18326pf+0w
0:55 kswapd

Quoted 8 lines

Linux-2.4.14-pre5:
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.340u 3.450s 2:13.62 55.2% 0+0k 0+0io 16829pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.590u 2.940s 2:15.48 54.2% 0+0k 0+0io 17182pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.140u 3.480s 2:14.66 54.6% 0+0k 0+0io 17122pf+0w
0:01 kswapd

Quoted 8 lines

Hello Lorenzo,

to be honest: I expected that. The patch according to my knowledge fixes a "definition hole" in the shrink_cache algorithm. I tend to say it is the right thing to do it this way, but I am sure it is not as fast as immediate exit to swap. It would be interesting to know if it does hurt performance in not-near-oom environment. I'd say Andrea or Linus might know that, or you can try, of course :-)

400M of swap now (from 200M), Linux-2.4.14-pre6 + your vmscan-patch:

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.320u 2.260s 2:28.92 49.4% 0+0k 0+0io 18755pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.330u 2.120s 2:28.40 49.4% 0+0k 0+0io 18838pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.880u 2.100s 2:28.31 49.8% 0+0k 0+0io 18646pf+0w
0:56 kswapd

qsbench vsize is just 343M, definitely not-near-oom environment :)

Quoted 7 lines

Anyway may I beg you to post my patch and your answer to the list, because I currently cannot do it (I am not in office right now, but on a web-terminal somewhere in the outbacks ;-). I have neither patch at hand nor am I able to attach it with this mailer...

Thanks,
Stephan

your vmscan-patch:

mm/vmscan.c +11 -9
@@ -469,16 +469,10 @@
 			spin_unlock(&pagecache_lock);
 			UnlockPage(page);
 page_mapped:
-			if (--max_mapped >= 0)
-				continue;
+			if (max_mapped > 0)
+				max_mapped--;
+			continue;
 
-			/*
-			 * Alert! We've found too many mapped pages on the
-			 * inactive list, so we start swapping out now!
-			 */
-			spin_unlock(&pagemap_lru_lock);
-			swap_out(priority, gfp_mask, classzone);
-			return nr_pages;
 		}
 
 		/*
@@ -514,6 +508,14 @@
 		break;
 	}
 	spin_unlock(&pagemap_lru_lock);
+
+	/*
+	 * Alert! We've found too many mapped pages on the
+	 * inactive list, so we start swapping out - delayed!
+	 * -skraw
+	 */
+	if (max_mapped==0)
+		swap_out(priority, gfp_mask, classzone);
 
 	return nr_pages;
 }
Signature · 1 line
Lorenzo
#2 lore raw
Quoted 38 lines
Quoted 36 lines · you
Quoted 34 lines

Well, your patch works but it hurts performance :(

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

71.500u 1.790s 2:29.18 49.1% 0+0k 0+0io 18498pf+0w

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

71.460u 1.990s 2:26.87 50.0% 0+0k 0+0io 18257pf+0w

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

71.220u 2.200s 2:26.82 50.0% 0+0k 0+0io 18326pf+0w

0:55 kswapd

Linux-2.4.14-pre5:

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

70.340u 3.450s 2:13.62 55.2% 0+0k 0+0io 16829pf+0w

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

70.590u 2.940s 2:15.48 54.2% 0+0k 0+0io 17182pf+0w

lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100

70.140u 3.480s 2:14.66 54.6% 0+0k 0+0io 17122pf+0w

0:01 kswapd

Quoted 7 lines
Quoted 7 lines

Hello Lorenzo,

to be honest: I expected that. The patch according to my knowledge

fixes a "definition hole" in the shrink_cache algorithm. I tend to

say

Quoted 3 lines
Quoted 3 lines

it is the right thing to do it this way, but I am sure it is not as

fast as immediate exit to swap. It would be interesting to know if

it

Quoted 1 line
Quoted 1 line

does hurt performance in not-near-oom environment. I'd say Andrea

or

Quoted 1 line
Quoted 1 line

Linus might know that, or you can try, of course :-)

To clarify this one a bit:
shrink_cache is thought to do what it says, it is given a number of pages it should somehow manage to free by shrinking the cache. What my patch does is go after the _whole_ list to fulfill that. One cannot really say that this is the wrong thing to do, I guess. If it takes time to _find_ free pages with shrink_cache, then probably the idea to use it was wrong in the first place (which is not the fault of the function itself). Or the number of free-pages to find is to high, or (as a last but guess unrealistic approach) the swap_out eats the time and shouldn't be called when nr_pages (return value) is equal to zero. This last one could be checked (hint hint Lorenzo ;-) by simply modifiying

if (max_swapped==0)

to

if (max_swapped==0 && nr_pages>0)

at the end of shrink_cache.
Thinking again about this it really sounds like the right choice, because there is no need to swap when we fulfilled the requested number of free-pages.

You should try.

Thank you for your patience Lorenzo

Regards,
Stephan

PS: just fishing for lobster, Linus ;-)

#3 lore raw
Quoted 5 lines · Stephan von Krawczynski

To clarify this one a bit:
shrink_cache is thought to do what it says, it is given a number of pages it should somehow manage to free by shrinking the cache. What my patch does is go after the _whole_ list to fulfill that.

I would suggest a slight modification: make "max_mapped" grow as the priority goes up.

Right now max_mapped is fixed at "nr_pages*10".

You could have something like

	max_mapped = nr_pages * 60 / priority;

instead, which might also alleviate the problem with not even bothering to scan much of the inactive list simply because 99% of all pages are mapped.

That way you don't waste time on looking at the rest of the inactive list until you _need_ to.

Linus

#4 lore raw
Quoted 5 lines · Stephan von Krawczynski
Quoted 3 lines

To clarify this one a bit:
shrink_cache is thought to do what it says, it is given a number

of

Quoted 1 line
Quoted 1 line

pages it should somehow manage to free by shrinking the cache.

What my

Quoted 5 lines
Quoted 1 line

patch does is go after the _whole_ list to fulfill that.

I would suggest a slight modification: make "max_mapped" grow as the priority goes up.

Quoted 7 lines

Right now max_mapped is fixed at "nr_pages*10".

You could have something like

	max_mapped = nr_pages * 60 / priority;                             

instead, which might also alleviate the problem with not even

bothering to

Quoted 1 line

scan much of the inactive list simply because 99% of all pages are

mapped.

Quoted 2 lines

That way you don't waste time on looking at the rest of the inactive

list

Quoted 1 line

until you _need_ to.

Wait a minute: there is something illogical in this approach:
Basically you say by making max_mapped bigger that the "early exit" from shrink_cache shouldn't be that early. But if you _know_ that merely all pages are mapped, then why don't you just go to swap_out right away without even walking through the list, because in the end, you will go to swap_out anyway (simply because of the high percentage of mapped pages). That makes scanning somehow superfluous. Making it priority-dependant sounds like you want to swap_out earlier the _lower_ memory pressure is. In the end it sounds just like a hack to hold up the early exit against every logic (but not against some benchmark of course).
It doesn't sound like the right thing.
Is the inactive list somehow sorted currently? If not, could it be implicitly sorted to match this criteria (not mapped versa mapped), so that shrink_cache finds the not-mapped first (with a chance to fulfill nr_pages-request). If it isn't fulfilled and hits the first mapped page, it can go to swap_out right away, because more scanning doesn't make sense and can only end in swap_out anyways.

I am no fan of complete list scanning, but if you are looking for something you have to scan until you find it.

Regards,
Stephan

PS: I am still no pro in this area, so I try to go after the global picture and find the right direction...

#5 lore raw
Quoted 8 lines · Stephan von Krawczynski

Wait a minute: there is something illogical in this approach:
Basically you say by making max_mapped bigger that the "early exit" from shrink_cache shouldn't be that early. But if you _know_ that merely all pages are mapped, then why don't you just go to swap_out right away without even walking through the list, because in the end, you will go to swap_out anyway (simply because of the high percentage of mapped pages). That makes scanning somehow superfluous.

Well, no.

There's two things: sure, we know we have tons of mapped pages, and we obviously will have done the "swap_out()" for th efirst iteration (and probably the second and third ones too).

But at some point you have to say "Ok, _this_ process has done its due work to clean up the VM pressure, and now this process needs to get on with its life and stop caring about other peoples bad memory usage".

Remember: everybody who calls "swap_out()" will free several pages from the pag tables. And everybody starts off with a low priority (ie 6). So if we're truly 99% mapped, then every single allocator will start off doing swap_out(), but at some point they obviously need to do other things too (ie they need to get to the point int he inactive queue where those swapped out pages are now, and try to write them out to disk).

Imagine a inactive queue that is a million entries. That's 4GB worth of RAM, sure, but there are lots of machines like that. If we only allow shrink_cache() to look at 320 pages at a time, we'll never get a life of our own.

(Yeah, sure, if you have all that 4GB on the inactive list, and it's all mapped, you're going to spend some time cleaning it up _regardless_ of what you do. That's life.)

Linus

#6 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Stephan von Krawczynski
Quoted 3 lines

To clarify this one a bit:
shrink_cache is thought to do what it says, it is given a number

of

Quoted 1 line
Quoted 1 line

pages it should somehow manage to free by shrinking the cache.

What my

Quoted 5 lines
Quoted 1 line

patch does is go after the _whole_ list to fulfill that.

I would suggest a slight modification: make "max_mapped" grow as the priority goes up.

Quoted 7 lines

Right now max_mapped is fixed at "nr_pages*10".

You could have something like

	max_mapped = nr_pages * 60 / priority;                             

instead, which might also alleviate the problem with not even

bothering to

Quoted 1 line

scan much of the inactive list simply because 99% of all pages are

mapped.

Quoted 2 lines

That way you don't waste time on looking at the rest of the inactive

list

Quoted 1 line

until you _need_ to.

Ok. I re-checked the code and found out this approach cannot stand. the list scan _is_ already exited early when priority is low:

        int max_scan = nr_inactive_pages / priority;                  
                                                                      
        while (--max_scan >= 0 && (entry = inactive_list.prev) !=     

&inactive_list) {

It will not make big sense to do it again in max_mapped.

On the other hand I am also very sure, that refining:

        if (max_mapped==0)                                            
                swap_out(priority, gfp_mask, classzone);              
                                                                      
        return nr_pages;                                              

in the end to:

        if (max_mapped==0 && nr_pages>0)                              
                swap_out(priority, gfp_mask, classzone);              
                                                                      
        return nr_pages;                                              

is a good thing. We don't need swap_out if we gained all the pages requested, no matter if we _could_ do it or not.

Is there some performance difference in this approach, Lorenzo? I guess it should.

Regards,
Stephan

#7 lore raw
Quoted 3 lines

Ok. I re-checked the code and found out this approach cannot stand.

the list scan _is_ already exited early when priority is low:

Sorry for followup on my own mail, but there is another thing that comes to my mind:

swap_out is currently in no way priority-dependant. But it could be (the parameter is there). How about swapping more pages in tighter memory situation? The basic idea is that if there is a rising need for mem it cannot be wrong to do a bit more than under normal circumstances. One could achieve this simply by:

        int counter, nr_pages = SWAP_CLUSTER_MAX;                     

to

int counter, nr_pages = SWAP_CLUSTER_MAX * DEF_PRIORITY /
priority;

in swap_out.
The idea behind is to reduce the overhead in finding out if swapping is needed by simply swapping more everytime we already gone "the long way to knowing".

Regards,
Stephan