new OOM heuristic failure (was: Re: VM: qsbench)
Subsystems (3)
Quoted 35 lines · you
Quoted 2 lines
Quoted 1 line · Stephan von Krawczynski
On Wed, 31 Oct 2001 22:31:40 +0100 Lorenzo Allegrucci
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines · wrote:
Linus,
your patch seems to help one case out of three. (even though I have not any meaningful statistical data)
Hm, I will not say that I expected that :-), he knows by far more
than me.
Quoted 1 line
Quoted 1 line
But can you try my patch below in addition or comparison to linus'
?
Quoted 12 lines
Quoted 1 line
Give me a hint what happens.
Well, your patch works but it hurts performance :(
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.500u 1.790s 2:29.18 49.1% 0+0k 0+0io 18498pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.460u 1.990s 2:26.87 50.0% 0+0k 0+0io 18257pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.220u 2.200s 2:26.82 50.0% 0+0k 0+0io 18326pf+0w
0:55 kswapdQuoted 8 lines
Linux-2.4.14-pre5:
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.340u 3.450s 2:13.62 55.2% 0+0k 0+0io 16829pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.590u 2.940s 2:15.48 54.2% 0+0k 0+0io 17182pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.140u 3.480s 2:14.66 54.6% 0+0k 0+0io 17122pf+0w
0:01 kswapd
Quoted 8 lines
Hello Lorenzo,
to be honest: I expected that. The patch according to my knowledge fixes a "definition hole" in the
shrink_cachealgorithm. I tend to say it is the right thing to do it this way, but I am sure it is not as fast as immediate exit to swap. It would be interesting to know if it does hurt performance in not-near-oom environment. I'd say Andrea or Linus might know that, or you can try, of course :-)
400M of swap now (from 200M), Linux-2.4.14-pre6 + your vmscan-patch:
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.320u 2.260s 2:28.92 49.4% 0+0k 0+0io 18755pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.330u 2.120s 2:28.40 49.4% 0+0k 0+0io 18838pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.880u 2.100s 2:28.31 49.8% 0+0k 0+0io 18646pf+0w
0:56 kswapd
qsbench vsize is just 343M, definitely not-near-oom environment :)
Quoted 7 lines
Anyway may I beg you to post my patch and your answer to the list, because I currently cannot do it (I am not in office right now, but on a web-terminal somewhere in the outbacks ;-). I have neither patch at hand nor am I able to attach it with this mailer...
Thanks,
Stephan
your vmscan-patch:
mm/vmscan.c +11 -9
@@ -469,16 +469,10 @@spin_unlock (&pagecache_lock);UnlockPage (page); page_mapped: -if (--max_mapped >=0 ) -continue ; +if (max_mapped >0 ) + max_mapped--; +continue ; -/* -* Alert! We've found too many mapped pages on the -* inactive list, so we start swapping out now! -*/ -spin_unlock (&pagemap_lru_lock); -swap_out (priority, gfp_mask, classzone); -return nr_pages; }/*
@@ -514,6 +508,14 @@break ; }spin_unlock (&pagemap_lru_lock); + +/* +* Alert! We've found too many mapped pages on the +* inactive list, so we start swapping out - delayed! +* -skraw +*/ +if (max_mapped==0 ) +swap_out (priority, gfp_mask, classzone);return nr_pages; }
Signature · 1 line
Lorenzo
Quoted 38 lines
Quoted 36 lines · you
Quoted 34 lines
Well, your patch works but it hurts performance :(
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.500u 1.790s 2:29.18 49.1% 0+0k 0+0io 18498pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.460u 1.990s 2:26.87 50.0% 0+0k 0+0io 18257pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
71.220u 2.200s 2:26.82 50.0% 0+0k 0+0io 18326pf+0w
0:55 kswapd
Linux-2.4.14-pre5:
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.340u 3.450s 2:13.62 55.2% 0+0k 0+0io 16829pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.590u 2.940s 2:15.48 54.2% 0+0k 0+0io 17182pf+0w
lenstra:~/src/qsort> time ./qsbench -n 90000000 -p 1 -s 140175100
70.140u 3.480s 2:14.66 54.6% 0+0k 0+0io 17122pf+0w
0:01 kswapd
Quoted 7 lines
Quoted 7 lines
Hello Lorenzo,
to be honest: I expected that. The patch according to my knowledge
fixes a "definition hole" in the
shrink_cachealgorithm. I tend to
say
Quoted 3 lines
Quoted 3 lines
it is the right thing to do it this way, but I am sure it is not as
fast as immediate exit to swap. It would be interesting to know if
it
Quoted 1 line
Quoted 1 line
does hurt performance in not-near-oom environment. I'd say Andrea
or
Quoted 1 line
Quoted 1 line
Linus might know that, or you can try, of course :-)
To clarify this one a bit:shrink_cache is thought to do what it says, it is given a number of pages it should somehow manage to free by shrinking the cache. What my patch does is go after the _whole_ list to fulfill that. One cannot really say that this is the wrong thing to do, I guess. If it takes time to _find_ free pages with shrink_cache, then probably the idea to use it was wrong in the first place (which is not the fault of the function itself). Or the number of free-pages to find is to high, or (as a last but guess unrealistic approach) the swap_out eats the time and shouldn't be called when nr_pages (return value) is equal to zero. This last one could be checked (hint hint Lorenzo ;-) by simply modifiying
if (max_swapped==0)
to
if (max_swapped==0 && nr_pages>0)
at the end of shrink_cache.
Thinking again about this it really sounds like the right choice, because there is no need to swap when we fulfilled the requested number of free-pages.
You should try.
Thank you for your patience Lorenzo
Regards,
Stephan
PS: just fishing for lobster, Linus ;-)
Quoted 5 lines · Stephan von Krawczynski
To clarify this one a bit:
shrink_cacheis thought to do what it says, it is given a number of pages it should somehow manage to free by shrinking the cache. What my patch does is go after the _whole_ list to fulfill that.
I would suggest a slight modification: make "max_mapped" grow as the priority goes up.
Right now max_mapped is fixed at "nr_pages*10".
You could have something like
max_mapped = nr_pages * 60 / priority;
instead, which might also alleviate the problem with not even bothering to scan much of the inactive list simply because 99% of all pages are mapped.
That way you don't waste time on looking at the rest of the inactive list until you _need_ to.
Linus
Quoted 5 lines · Stephan von Krawczynski
Quoted 3 lines
To clarify this one a bit:
shrink_cacheis thought to do what it says, it is given a number
of
Quoted 1 line
Quoted 1 line
pages it should somehow manage to free by shrinking the cache.
What my
Quoted 5 lines
Quoted 1 line
patch does is go after the _whole_ list to fulfill that.
I would suggest a slight modification: make "
max_mapped" grow as the priority goes up.
Quoted 7 lines
Right now
max_mappedis fixed at "nr_pages*10".You could have something like
max_mapped = nr_pages * 60 / priority;instead, which might also alleviate the problem with not even
bothering to
Quoted 1 line
scan much of the inactive list simply because 99% of all pages are
mapped.
Quoted 2 lines
That way you don't waste time on looking at the rest of the inactive
list
Quoted 1 line
until you _need_ to.
Wait a minute: there is something illogical in this approach:
Basically you say by making max_mapped bigger that the "early exit" from shrink_cache shouldn't be that early. But if you _know_ that merely all pages are mapped, then why don't you just go to swap_out right away without even walking through the list, because in the end, you will go to swap_out anyway (simply because of the high percentage of mapped pages). That makes scanning somehow superfluous. Making it priority-dependant sounds like you want to swap_out earlier the _lower_ memory pressure is. In the end it sounds just like a hack to hold up the early exit against every logic (but not against some benchmark of course).
It doesn't sound like the right thing.
Is the inactive list somehow sorted currently? If not, could it be implicitly sorted to match this criteria (not mapped versa mapped), so that shrink_cache finds the not-mapped first (with a chance to fulfill nr_pages-request). If it isn't fulfilled and hits the first mapped page, it can go to swap_out right away, because more scanning doesn't make sense and can only end in swap_out anyways.
I am no fan of complete list scanning, but if you are looking for something you have to scan until you find it.
Regards,
Stephan
PS: I am still no pro in this area, so I try to go after the global picture and find the right direction...
Quoted 8 lines · Stephan von Krawczynski
Wait a minute: there is something illogical in this approach:
Basically you say by makingmax_mappedbigger that the "early exit" fromshrink_cacheshouldn't be that early. But if you _know_ that merely all pages are mapped, then why don't you just go toswap_outright away without even walking through the list, because in the end, you will go toswap_outanyway (simply because of the high percentage of mapped pages). That makes scanning somehow superfluous.
Well, no.
There's two things: sure, we know we have tons of mapped pages, and we obviously will have done the "swap_out()" for th efirst iteration (and probably the second and third ones too).
But at some point you have to say "Ok, _this_ process has done its due work to clean up the VM pressure, and now this process needs to get on with its life and stop caring about other peoples bad memory usage".
Remember: everybody who calls "swap_out()" will free several pages from the pag tables. And everybody starts off with a low priority (ie 6). So if we're truly 99% mapped, then every single allocator will start off doing swap_out(), but at some point they obviously need to do other things too (ie they need to get to the point int he inactive queue where those swapped out pages are now, and try to write them out to disk).
Imagine a inactive queue that is a million entries. That's 4GB worth of RAM, sure, but there are lots of machines like that. If we only allow shrink_cache() to look at 320 pages at a time, we'll never get a life of our own.
(Yeah, sure, if you have all that 4GB on the inactive list, and it's all mapped, you're going to spend some time cleaning it up _regardless_ of what you do. That's life.)
Linus
Quoted 5 lines · Stephan von Krawczynski
Quoted 3 lines
To clarify this one a bit:
shrink_cacheis thought to do what it says, it is given a number
of
Quoted 1 line
Quoted 1 line
pages it should somehow manage to free by shrinking the cache.
What my
Quoted 5 lines
Quoted 1 line
patch does is go after the _whole_ list to fulfill that.
I would suggest a slight modification: make "
max_mapped" grow as the priority goes up.
Quoted 7 lines
Right now
max_mappedis fixed at "nr_pages*10".You could have something like
max_mapped = nr_pages * 60 / priority;instead, which might also alleviate the problem with not even
bothering to
Quoted 1 line
scan much of the inactive list simply because 99% of all pages are
mapped.
Quoted 2 lines
That way you don't waste time on looking at the rest of the inactive
list
Quoted 1 line
until you _need_ to.
Ok. I re-checked the code and found out this approach cannot stand. the list scan _is_ already exited early when priority is low:
int max_scan = nr_inactive_pages / priority;
while (--max_scan >= 0 && (entry = inactive_list.prev) != &inactive_list) {
It will not make big sense to do it again in max_mapped.
On the other hand I am also very sure, that refining:
if (max_mapped==0)
swap_out(priority, gfp_mask, classzone);
return nr_pages; in the end to:
if (max_mapped==0 && nr_pages>0)
swap_out(priority, gfp_mask, classzone);
return nr_pages; is a good thing. We don't need swap_out if we gained all the pages requested, no matter if we _could_ do it or not.
Is there some performance difference in this approach, Lorenzo? I guess it should.
Regards,
Stephan
Quoted 3 lines
Ok. I re-checked the code and found out this approach cannot stand.
the list scan _is_ already exited early when priority is low:
Sorry for followup on my own mail, but there is another thing that comes to my mind:
swap_out is currently in no way priority-dependant. But it could be (the parameter is there). How about swapping more pages in tighter memory situation? The basic idea is that if there is a rising need for mem it cannot be wrong to do a bit more than under normal circumstances. One could achieve this simply by:
int counter, nr_pages = SWAP_CLUSTER_MAX;
to
int counter, nr_pages = SWAP_CLUSTER_MAX * DEF_PRIORITY /
priority;
in swap_out.
The idea behind is to reduce the overhead in finding out if swapping is needed by simply swapping more everytime we already gone "the long way to knowing".
Regards,
Stephan