VM Requirement Document - v0.0
Quoted 4 lines
Quoted 4 lines
* If we're getting low cache hit rates, don't flush processes to swap.
* If we're getting good cache hit rates, flush old, idle processes to swap.
Rik> ... but I fail to see this one. If we get a low cache hit rate, Rik> couldn't that just mean we allocated too little memory for the Rik> cache ?
maybe more specific: If the hit-rate is low and the cache is already
70+% of the systems memory, the chances maybe slim that more cache is going to improve the hit-rate.
I do not care much whether the cache is using 99% of the systems memory
or 50%. As long as there is free memory, using it for cache is great. I care a lot if the cache takes down interactivity, because it pushes out processes that it thinks idle, but that I need in 5 seconds. The caches pressure against processes should decrease with the (relative) size of the cache. Especially in low hit-rate situations.
OT: I asked the question before somewhere else. Are there interfaces to
the VM that expose the various cache sizes and, more important, hit-rates to userland? I would love to see (or maybe help writing in my free time) a tool to just visualize/analyze the efficiency of the VM system.
Martin
Signature · 5 lines
------------------------------------------------------------------ Martin Knoblauch | email: [email protected] TeraPort GmbH | Phone: +49-89-510857-309 C+ITS | Fax: +49-89-510857-111 http://www.teraport.de | Mobile: +49-170-4904759
Quoted 5 lines · Martin Knoblauch
I do not care much whether the cache is using 99% of the systems memory
or 50%. As long as there is free memory, using it for cache is great. I care a lot if the cache takes down interactivity, because it pushes out processes that it thinks idle, but that I need in 5 seconds. The caches pressure against processes
Too bad that processes are in general cached INSIDE the cache.
You'll have to write a new balancing story now ;)
regards,
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/
Quoted 9 lines · Rik van Riel
Quoted 5 lines · Martin Knoblauch
I do not care much whether the cache is using 99% of the systems memory
or 50%. As long as there is free memory, using it for cache is great. I care a lot if the cache takes down interactivity, because it pushes out processes that it thinks idle, but that I need in 5 seconds. The caches pressure against processes
Quoted 4 lines
Too bad that processes are in general cached INSIDE the cache.
You'll have to write a new balancing story now ;)
maybe that is part of "the answer" :-)
Martin
Signature · 5 lines
------------------------------------------------------------------ Martin Knoblauch | email: [email protected] TeraPort GmbH | Phone: +49-89-510857-309 C+ITS | Fax: +49-89-510857-111 http://www.teraport.de | Mobile: +49-170-4904759
Quoted 5 lines · Martin Knoblauch
maybe more specific: If the hit-rate is low and the cache is already
70+% of the systems memory, the chances maybe slim that more cache is going to improve the hit-rate.
Oh, but this is posible. You can get into situations where the (file cache) working set needs 80% or so of memory to get a near-perfect hitrate, and where
using 70% of memory will trash madly due to the file access pattern. And this won't be a problem either, if the working set of "other" (non-file)
stuff is below 20% of memory. The total size of non-file stuff may be above 20% though, so something goes into swap.
I definitely want the machine to work under such circumstances, so an arbitrary limit of 70% won't work.
Preventing swap-trashing at all cost doesn't help if the machine loose to io-trashing instead. Performance will be just as much down, although perhaps more satisfying because people aren't that surprised if explicit file operations take a long time. They hate it when moving the mouse or something cause a disk access even if their apps runs faster. :-(
Helge Hafting
Quoted 7 lines · Helge Hafting
Preventing swap-trashing at all cost doesn't help if the machine loose to io-trashing instead. Performance will be just as much down, although perhaps more satisfying because people aren't that surprised if explicit file operations take a long time. They hate it when moving the mouse or something cause a disk access even if their apps runs faster. :-(
Exactly. I still want the ability to tune the system according to my taste. I've been thinking about this for some time, and I've specifically tried to come up with nice tunables, completely ignoring if it is possible now or not.
If individual pages could be classified as code (text segments), data, file cache, and so on, I would specify costs to the paging of such pages in or out. This way I can make the system perfer to drop a file cache page that has not been accessed for five minutes, over a program text page that has not been acccessed for one hour (or much more).
This would be very useful, I think. Would it be very hard to classify pages like this (text/data/cache/...)?
Any reason why this is a bad idea?
/Tobias
Quoted 2 lines · Tobias Ringstrom
This would be very useful, I think. Would it be very hard to classify pages like this (text/data/cache/...)?
How would you classify a page of perl code ?
Xav
Quoted 6 lines · Xavier Bestel
Quoted 2 lines · Tobias Ringstrom
This would be very useful, I think. Would it be very hard to classify pages like this (text/data/cache/...)?
How would you classify a page of perl code ?
I do know how the Perl interpreter works, but I think it byte-compiles the code and puts it in the data segment, which also would have a high paging cost.
The perl source code would be paged in/out before running binaries such as shells and the window system, but the same thing would happen to binaries with short life-span, I suppose. Perhaps cached executables and cached data files can be classified differently as well.
What I meant to ask with the question above was if it would be hard to implement the classification in the kernel.
/Tobias
Quoted 4 lines · Helge Hafting
Quoted 1 line · Martin Knoblauch
Quoted 8 lines
Quoted 4 lines
maybe more specific: If the hit-rate is low and the cache is already
70+% of the systems memory, the chances maybe slim that more cache is going to improve the hit-rate.Oh, but this is posible. You can get into situations where the (file cache) working set needs 80% or so of memory to get a near-perfect hitrate, and where
using 70% of memory will trash madly due to the file access
thats why I said "maybe" :-) Sure, another 5% of cache may improve
things, but they also may kill the interactive performance. Thats why there should be probably more than one VM strategy to accomodate Servers and Workstations/Lpatops.
Quoted 6 lines
pattern. And this won't be a problem either, if the working set of "other" (non-file)
stuff is below 20% of memory. The total size of non-file stuff may be above 20% though, so something goes into swap.
And that is the problem. To much seems to go into swap. At least for
interactive work. Unfortunatelly, with 128MB of memory I cannot entirely turn of swap. I will see how things are going once I have 256 or 512 MB (hopefully soon :-)
Quoted 3 lines
I definitely want the machine to work under such circumstances, so an arbitrary limit of 70% won't work.
Do not take the 70% as an arbitrary limit. I never said that. The 70%
is just my situation. The problems may arise at 60% cache or at 97.38% cache.
Quoted 1 line
Preventing swap-trashing at all cost doesn't help if the
Never said at all cost.
Quoted 7 lines
machine loose to io-trashing instead. Performance will be just as much down, although perhaps more satisfying because people aren't that surprised if explicit file operations take a long time. They hate it when moving the mouse or something cause a disk access even if their apps runs faster. :-(
Absolutely true. And if the main purpose of the machine is interactive
work (we do want to be Linux a success on the desktop, don't we?), it should not be hampered by by an IO improvement that may be only of secondary importance to the user (that the final "customer" for all the work that is done to the kernel :-). On big servers a litle paging now and then may be absolutely OK, as long as the IO is going strong.
I am observing the the discussions of VM behaviour in 2.4.x for some
time. They are mostly very entertaining and revealing. But they also show that one solution does not seem to benefit all possible scenarios. Therfore either more than one VM strategy is necessary, or better means of tuning the cache behaviour, or both. Definitely better ways of measuring the VM efficiency seem to be needed.
While implementing VM strategies is probably out of question for a lot
of the people that complain, I hope that at least my complaints are kind of useful.
Martin
Signature · 5 lines
------------------------------------------------------------------ Martin Knoblauch | email: [email protected] TeraPort GmbH | Phone: +49-89-510857-309 C+ITS | Fax: +49-89-510857-111 http://www.teraport.de | Mobile: +49-170-4904759