Possible Idea with filesystem buffering.
Quoted 11 lines
Re: Possible Idea with filesystem buffering.
From: Richard B. Johnson ([email protected])
Date: Tue Jan 22 2002 - 17:10:27 ESTWe need a free-RAM target, possibly based upon a percentage of available RAM. The lack of such a target is what causes the out-of-RAM condition we have been experiencing. Somebody thought that "free RAM is wasted RAM" and the VM has been based upon that theory. That theory has been proven incorrect. You need
Now, I think the theory itself is OK. The problem is that the stuff in
buffer/caches is to sticky. It does not go away when "more important" uses for memory come up. Or at least it does not go away fast enough.
Quoted 4 lines
free RAM, just like you need "excess horsepower" to make automobiles drivable. That free RAM is the needed "rubber-band" to absorb the dynamics of real-world systems.
Correct. The free target would help to avoid the panic/frenzy that
breaks out when we run out of free memory.
Question: what about just setting a maximum limit to the cache/buffer
size. Either absolute, or as a fraction of total available memory? Sure, it maybe a waste of memory in most situations, but sometimes the administrator/user of a system simply "knows better" than the FVM (F == Fine ? :-)
While being there, one could also add a "guaranteed minimum" limit for
the cache/buffer size. This way preventing a complete meltdown of IO performance. True64 has such limits. They are usually at 100% (max) and I think 20% (min), giving the cache access to all memory. But there were situations where a max of 10% was just the rigth thing to do.
I know, the tuning-knob approach is frowned upon. But sometimes there
are workloads where even the best VM may not know how to react correctly.
Martin
Signature · 5 lines
------------------------------------------------------------------ Martin Knoblauch | email: [email protected] TeraPort GmbH | Phone: +49-89-510857-309 C+ITS | Fax: +49-89-510857-111 http://www.teraport.de | Mobile: +49-170-4904759
Quoted 7 lines · Martin Knoblauch
Quoted 7 lines
Re: Possible Idea with filesystem buffering.
From: Richard B. Johnson ([email protected])
Date: Tue Jan 22 2002 - 17:10:27 EST
Quoted 6 lines
Quoted 5 lines
We need a free-RAM target, possibly based upon a percentage of available RAM. The lack of such a target is what causes the out-of-RAM condition we have been experiencing. Somebody thought that "free RAM is wasted RAM" and the VM has been based upon that theory. That theory has been proven incorrect. You need
As far as I know, there is a free target. The kernel will try to get rid of old pages (swapout program memory, toss cache pages) when there's too little free memory around. This keeps memory around so future allocations and IO request may start immediately. Maybe the current target is too small, but it is there. Without it, _every_ allocation or file operation would block waiting for a swapout/cache flush in order to get free pages. Linux isn't nearly _that_ bad.
Quoted 4 lines
Now, I think the theory itself is OK. The problem is that the stuff in
buffer/caches is to sticky. It does not go away when "more important" uses for memory come up. Or at least it does not go away fast enough.
Then we need a larger free target to cope with the slow cache freeing.
Quoted 9 lines
Quoted 3 lines
free RAM, just like you need "excess horsepower" to make automobiles drivable. That free RAM is the needed "rubber-band" to absorb the dynamics of real-world systems.
Question: what about just setting a maximum limit to the cache/buffer
size. Either absolute, or as a fraction of total available memory? Sure, it maybe a waste of memory in most situations, but sometimes the administrator/user of a system simply "knows better" than the FVM (F == Fine ? :-)
[...]
Quoted 3 lines
I know, the tuning-knob approach is frowned upon. But sometimes there
are workloads where even the best VM may not know how to react correctly.
Wasting memory "in most situations" isn't really an option. But I see nothing wrong with "knobs" as long as they are automatic by default. Those who want to optimize for a corner case can go and turn off the autopilot.
Helge Hafting
On Wed, 23 Jan 2002, Helge Hafting wrote:
[free memory is wasted memory]
Quoted 5 lines
Quoted 3 lines
Now, I think the theory itself is OK. The problem is that the stuff in
buffer/caches is to sticky. It does not go away when "more important" uses for memory come up. Or at least it does not go away fast enough.Then we need a larger free target to cope with the slow cache freeing.
Or we make the cache freeing faster. ;)
If you have the time, you might want to try -rmap some day and see about the cache freeing...
regards,
Rik
Signature · 4 lines
"Linux holds advantages over the single-vendor commercial OS"
-- Microsoft's "Competing with Linux" document
http://www.surriel.com/ http://distro.conectiva.com/Quoted 15 lines · Helge Hafting
Quoted 7 lines · Martin Knoblauch
Quoted 7 lines
Re: Possible Idea with filesystem buffering.
From: Richard B. Johnson ([email protected])
Date: Tue Jan 22 2002 - 17:10:27 ESTQuoted 6 lines
Quoted 5 lines
We need a free-RAM target, possibly based upon a percentage of available RAM. The lack of such a target is what causes the out-of-RAM condition we have been experiencing. Somebody thought that "free RAM is wasted RAM" and the VM has been based upon that theory. That theory has been proven incorrect. You need
Quoted 9 lines
As far as I know, there is a free target. The kernel will try to get rid of old pages (swapout program memory, toss cache pages) when there's too little free memory around. This keeps memory around so future allocations and IO request may start immediately. Maybe the current target is too small, but it is there. Without it, _every_ allocation or file operation would block waiting for a swapout/cache flush in order to get free pages. Linux isn't nearly _that_ bad.
Nobody said it is _that_ bad. There are just some [maybe rare]
situations where it falls over and does not recover gracefully.
Quoted 6 lines
Quoted 4 lines
Now, I think the theory itself is OK. The problem is that the stuff in
buffer/caches is to sticky. It does not go away when "more important" uses for memory come up. Or at least it does not go away fast enough.Then we need a larger free target to cope with the slow cache freeing.
And as Rik said, we need to make freeing cache faster. All of this will
help the 98+% cases that the VM can be optimized for. But I doubt that you can make it 100% and keep it simple at the same time.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
free RAM, just like you need "excess horsepower" to make automobiles drivable. That free RAM is the needed "rubber-band" to absorb the dynamics of real-world systems.
Question: what about just setting a maximum limit to the cache/buffer
size. Either absolute, or as a fraction of total available memory? Sure, it maybe a waste of memory in most situations, but sometimes the administrator/user of a system simply "knows better" than the FVM (F == Fine ? :-)[...]
Quoted 9 lines
Quoted 3 lines
I know, the tuning-knob approach is frowned upon. But sometimes there
are workloads where even the best VM may not know how to react correctly.Wasting memory "in most situations" isn't really an option. But I see nothing wrong with "knobs" as long as they are automatic by default. Those who want to optimize for a corner case can go and turn off the autopilot.
Definitely. The defaults need to be set for the general case.
Martin
Signature · 5 lines
------------------------------------------------------------------ Martin Knoblauch | email: [email protected] TeraPort GmbH | Phone: +49-89-510857-309 C+ITS | Fax: +49-89-510857-111 http://www.teraport.de | Mobile: +49-170-4904759