large files unnecessary trashing filesystem cache?
(please note, i'am not subscribed to the list, please CC me on reply)
Story:
Once in while we have a discussion at the vdr (video disk recorder) mailing list about very large files trashing the filesystems memory cache leading to unnecessary delays accessing directory contents no longer cached.
This program and certainly all applications that deal with very large files only read once (much larger than usual memory) - it happens that all other cached blocks of the filessystem are removed from memory solely to keep as much as possible of that file in memory, which seems to be a bad strategy in most situations.
Of course one could always implement f_advise-calls in all applications, but i suggest a discussion if a maximum (configurable) in-memory-cache on a per-file base should be implemented in linux/mm or where this belongs.
My guess was, it has something to do with mm/readahead.c, a test limiting the result of the function "max_sane_readahead(...) to 8 MBytes as a quick and dirty test did not solve the issue, but i might have done something wrong.
I've searched the archive but could not find a previous discussion - is this a new idea?
It would be interesting to discuss if and when this proposed feature could lead to better performance or has any unwanted side effects.
Thanks for ideas on that issue.
Quoted 21 lines · Guido Fiala
(please note, i'am not subscribed to the list, please CC me on reply)
Story:
Once in while we have a discussion at the vdr (video disk recorder) mailing list about very large files trashing the filesystems memory cache leading to unnecessary delays accessing directory contents no longer cached.This program and certainly all applications that deal with very large files only read once (much larger than usual memory) - it happens that all other cached blocks of the filessystem are removed from memory solely to keep as much as possible of that file in memory, which seems to be a bad strategy in most situations.
Of course one could always implement f_advise-calls in all applications, but i suggest a discussion if a maximum (configurable) in-memory-cache on a per-file base should be implemented in linux/mm or where this belongs.
My guess was, it has something to do with
mm/readahead.c, a test limiting the result of the function "max_sane_readahead(...)to 8 MBytes as a quick and dirty test did not solve the issue, but i might have done something wrong.
Quoted 7 lines
I've searched the archive but could not find a previous discussion - is this a new idea?
It would be interesting to discuss if and when this proposed feature could lead to better performance or has any unwanted side effects.
Thanks for ideas on that issue.
Is there a reason why those applications couldn't use O_DIRECT ?
Thanks,
Badari
Quoted 11 lines · Badari Pulavarty
Quoted 6 lines · Guido Fiala
Story:
Once in while we have a discussion at the vdr (video disk recorder) mailing list about very large files trashing the filesystems memory cache leading to unnecessary delays accessing directory contents no longer cached.
[...]Is there a reason why those applications couldn't use
O_DIRECT?Thanks,
Badari
I asked a vdr-expert on this and here is the reason why O_DIRECT is not suitable:
O_DIRECT would be great if it were a simple option for opening files. But as a matter of facts O_DIRECT completely changes the semantics of file access. You have to read blocks of a defines size to memory that is aligned to defined block borders. Memory provided by normal malloc() or new() is not usable and results in IO errors. So the result is you have to have a complete rewrite of the whole IO subsystem of the affected program. Most maintainers of non-trivial applications are completely resistant against such changes - for good reasons.
If there would be an O_DIRECT_EX32++ (or O_STREAMING) that doesn't have this change in semantic it would be much easier to apply the necessary changes.
BTW: In the case of the VDR program not even a per process limit in used buffer caches would help: the same program reads huge files _and_ huge directory trees with a lot of small files that should be cached. A heuristic for this case has to work on per file base. It needs to detect that some files are only used in a streaming manner - with very seldom jumps in random directions (skipping commercials, review a scene). I don't know if such a heuristic is possible and if it would not break other things.
PS: using f_advise helps a bit. One can keep IO semantics but you have to add a virtualisation layer for all streaming IO. And you can't combine posix_fadvise(POSIX_FADV_DONTNEED) with posix_fadvise(POSIX_FADV_WILLNEED) when you possibly have jumps in your access pattern because you can't cancel (at least to my knowledge) the POSIX_FADV_WILLNEED call when you see the read ahead is not needed any more. It would be an interesting add on if POSIX_FADV_DONTNEED would cancel the read of a region that has been requested by POSIX_FADV_WILLNEED before.
Ralf (forwarded by me on his request)
---
Hopefully i did now correctly "reply all" - sorry if i accidently caused some trouble.
Quoted 7 lines · Guido Fiala
(please note, i'am not subscribed to the list, please CC me on reply)
Story:
Once in while we have a discussion at the vdr (video disk recorder) mailing list about very large files trashing the filesystems memory cache leading to unnecessary delays accessing directory contents no longer cached.
Quoted 5 lines
This program and certainly all applications that deal with very large files only read once (much larger than usual memory) - it happens that all other cached blocks of the filessystem are removed from memory solely to keep as much as possible of that file in memory, which seems to be a bad strategy in most situations.
For this particular workload, a heuristic to detect streaming and drop
pages a few mb back from currently accessed pages would probably work well. I believe the second part is already in the kernel (activated by an f-advise
call), but the heuristic is lacking.
Quoted 10 lines
Of course one could always implement f_advise-calls in all applications, but i suggest a discussion if a maximum (configurable) in-memory-cache on a per-file base should be implemented in linux/mm or where this belongs.
My guess was, it has something to do with
mm/readahead.c, a test limiting the result of the function "max_sane_readahead(...)to 8 MBytes as a quick and dirty test did not solve the issue, but i might have done something wrong.I've searched the archive but could not find a previous discussion - is this a new idea?
I'd do searches on thrashing control and swap tokens. The problem with thrashing is similar: a process accessing large amounts of memory in a short period of time blowing away the caches. And the solution should be similar:
penalize the process doing so by preferentially reclaiming it's pages.
Quoted 2 lines
It would be interesting to discuss if and when this proposed feature could lead to better performance or has any unwanted side effects.
Sometimes you want a single file to take up most of the memory; databases spring to mind. Perhaps files/processes that take up a large proportion of memory should be penalized by preferentially reclaiming their pages, but limit the aggressiveness so that they can still take up most of the memory if sufficiently persistent (and the rest of the system isn't thrashing).
Quoted 9 lines
Thanks for ideas on that issue.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Andrew James Wade
Sometimes you want a single file to take up most of the memory; databases
spring to mind. Perhaps files/processes that take up a large proportion of memory should be penalized by preferentially reclaiming their pages, but limit the aggressiveness so that they can still take up most of the memory if sufficiently persistent (and the rest of the system isn't thrashing).
Yes. Basically any smart heuristic we apply here will have failure modes. For example, the person whose application does repeated linear reads of the first 100MB of a 4G file will get very upset.
So any such change really has to be opt-in. Yes, it can be done quite simply via repeated application of posix_fadvise(). But practically speaking, it's very hard to get upstream GPL'ed applications changed, let alone proprietary ones.
An obvious approach would be an LD_PRELOAD thingy which modifies read() and write(), perhaps controlled via an environment variable. AFAIK nobody has even attempted this.
For a kernel-based solution you could take a look at my old 2.4-based O_STREAMING patch. It works OK, but it still needs modification of each application (or an LD_PRELOAD hook into open()).
A decent kernel implementation would be to add a max_resident_pages to struct file_struct and to use that to perform drop-behind within read() and write(). That's a bit of arithmetic and a call to invalidate_mapping_pages(). The userspace interface to that could be a linux-specific extension to posix_fadvise() or to fcntl().
But that still requires that all the applications be modified.
So I'd also suggest a new resource limit which, if set, is copied into the applications's file_structs on open(). So you then write a little wrapper app which does setrlimit()+exec():
limit-cache-usage -s 1000 my-fave-backup-program <args>
Which will cause every file which my-fave-backup-program reads or writes to be limited to a maximum pagecache residency of 1000 kbytes.
This facility could trivially be used for a mini-DoS: shooting down other people's pagecache so their apps run slowly. But you can use fadvise() for that already.
Or raise a patch against glibc's read() and write() which uses some environment string to control fadvise-based invalidation. That's pretty simple.
Quoted 11 lines · Andrew Morton
Quoted 6 lines · Andrew James Wade
Sometimes you want a single file to take up most of the memory; databases
spring to mind. Perhaps files/processes that take up a large proportion of memory should be penalized by preferentially reclaiming their pages, but limit the aggressiveness so that they can still take up most of the memory if sufficiently persistent (and the rest of the system isn't thrashing).Yes. Basically any smart heuristic we apply here will have failure modes. For example, the person whose application does repeated linear reads of the first 100MB of a 4G file will get very upset.
As will any dumb heuristic for that matter; we'd need precognition[1] to avoid all of them. But we can hopefully make the failure modes rarer and more predictable. I don't know how my proposal would fare, and as I do not have the code to test the matter I think I shall drop it.
[1] Which could, on occasion, be provided by hinting.
Zitat von Andrew James Wade <[email protected]>:
Quoted 11 lines
Quoted 1 line
For example, the person whose application does repeated linear reads of
the
Quoted 1 line
first 100MB of a 4G file will get very upset.
As will any dumb heuristic for that matter; we'd need precognition[1] to avoid
all of them. But we can hopefully make the failure modes rarer and more predictable. I don't know how my proposal would fare, and as I do not have the code to test the matter I think I shall drop it.[1] Which could, on occasion, be provided by hinting.
That is why i said it should be configurable by root via proc/sys/kernel interface - if the system is intended to primarily run a database, set it to a different heuristic than to "desktop/multimedia-workstation".
Much like the recent IO-scheduler additions (do they affect this behaviour?)
Zitat von Andrew Morton <[email protected]>:
Quoted 3 lines
An obvious approach would be an
LD_PRELOADthingy which modifiesread()andwrite(), perhaps controlled via an environment variable. AFAIK nobody has even attempted this.
Sounds interesting.
Quoted 5 lines
A decent kernel implementation would be to add a
max_resident_pagesto structfile_structand to use that to perform drop-behind withinread()andwrite(). That's a bit of arithmetic and a call toinvalidate_mapping_pages(). The userspace interface to that could be a linux-specific extension toposix_fadvise()or tofcntl().
Would still like to have a way to configure a "default file policy/heuristics" for the system, just like i can choose IO-scheduler.
Quoted 11 lines
But that still requires that all the applications be modified.
So I'd also suggest a new resource limit which, if set, is copied into the applications's
file_structsonopen(). So you then write a little wrapper app which does setrlimit()+exec():limit-cache-usage -s 1000 my-fave-backup-program <args>
Which will cause every file which my-fave-backup-program reads or writes to be limited to a maximum pagecache residency of 1000 kbytes.
Or make it another 'ulimit' parameter...
Hi,
Quoted 11 lines · On Wednesday 19 October 2005 13:10, [email protected] wrote:
Zitat von Andrew Morton <[email protected]>:
Quoted 8 lines
So I'd also suggest a new resource limit which, if set, is copied into the applications's
file_structsonopen(). So you then write a little wrapper app which does setrlimit()+exec():limit-cache-usage -s 1000 my-fave-backup-program <args>
Which will cause every file which my-fave-backup-program reads or writes to be limited to a maximum pagecache residency of 1000 kbytes.
Or make it another 'ulimit' parameter...
Which is already there: There is an ulimit for "maximum RSS", which is at least a superset of "maximum pagecache residency".
This is already settable and known by many admins. But AFAIR it is not honoured by the kernel completely, right?
But per file is a much better choice, since this would allow concurrent streaming. This is needed to implement timeshifting at least[1].
So either I miss something or this is no proper solution yet.
Regards
Ingo Oeser
[1] Which is obviously done by some kind of on-disk FIFO.
Quoted 5 lines · Ingo Oeser
Hi,
Quoted 1 line · On Wednesday 19 October 2005 13:10, [email protected] wrote:
Zitat von Andrew Morton <[email protected]>:
Please don't edit Cc lines. Just do reply-to-all.
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
So I'd also suggest a new resource limit which, if set, is copied into the applications's
file_structsonopen(). So you then write a little wrapper app which does setrlimit()+exec():limit-cache-usage -s 1000 my-fave-backup-program <args>
Which will cause every file which my-fave-backup-program reads or writes to be limited to a maximum pagecache residency of 1000 kbytes.
Or make it another 'ulimit' parameter...
That's what I said. ulimit is the shell interface to resource limits.
Quoted 2 lines
Which is already there: There is an ulimit for "maximum RSS", which is at least a superset of "maximum pagecache residency".
RSS is a quite separate concept from pagecache.
Quoted 7 lines
This is already settable and known by many admins. But AFAIR it is not honoured by the kernel completely, right?
But per file is a much better choice, since this would allow concurrent streaming. This is needed to implement timeshifting at least[1].
So either I miss something or this is no proper solution yet.
I described a couple of ways in which this can be done from userspace with LD_PRELOAD.
Quoted 1 line · Andrew
Please don't edit Cc lines. Just do reply-to-all.
Don't worry, Ingo Oeser.
I think it was [email protected] who edited the Cc line in this thread, not yourself.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Hi,
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Ingo Oeser
Quoted 4 lines
Quoted 4 lines
app which does setrlimit()+exec():
limit-cache-usage -s 1000 my-fave-backup-program <args>
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Which will cause every file which my-fave-backup-program reads or writes to be limited to a maximum pagecache residency of 1000 kbytes.
Or make it another 'ulimit' parameter...
Which is already there: There is an ulimit for "maximum RSS", which is at least a superset of "maximum pagecache residency".
RSS is a quite separate concept from pagecache.
Yes I know, but the amount of pagecache which is RESIDENT for a process is not that seperate from RSS, I think.
I always thought RSS is the amount of mapped and anonymous pages of a process, which are in physical memory (aka resident).
So I consider the amount of mapped pagecache pages of a process which are in physical memory (aka resident) a subset.
Or do you care about page cache pages not mapped into the process? Is this the point I miss?
Please enlighten me :-)
Regards
Ingo Oeser
Quoted 2 lines · Guido Fiala
Of course one could always implement f_advise-calls in all applications
Um, this seems like the obvious answer. The application doing the read KNOWS it's a streaming read, while the best the kernel can do is guess.
You don't really make much of a case that fadvise can't do the job.
Lee
Quoted 8 lines · Lee Revell
Quoted 2 lines · Guido Fiala
Of course one could always implement f_advise-calls in all applications
Um, this seems like the obvious answer. The application doing the read KNOWS it's a streaming read, while the best the kernel can do is guess.
You don't really make much of a case that fadvise can't do the job.
The issue is, how will "other/random" programs/applications affect performance of my application.
Complain I hear most is from our database folks, they tune stuff and they are happy with their performance. And then, some one does a tar/cp/cpio/ftp/backup/compile on some random files on the system. Suddenly, database performance drops. They want to see a system wide/ per-filesystem tunable on how much pagecache it takes up.
Andrew, does this make sense at all ?
Thanks,
Badari
Quoted 9 lines · Lee Revell
Quoted 2 lines · Guido Fiala
Of course one could always implement f_advise-calls in all applications
Um, this seems like the obvious answer. The application doing the read KNOWS it's a streaming read, while the best the kernel can do is guess.
You don't really make much of a case that fadvise can't do the job.
Kernel could do the best to optimize default performance, applications that consider their own optimal behaviour should do so, all other files are kept under default heuristic policy (adaptable, configurable one)
Heuristic can be based on access statistic:
streaming/sequential can be guessed by getting exactly 100% cache hit rate (drop behind pages immediately),
random access/repeated reads can be guessed by >100% hit rate (keep as much in memory as possible).
Less than 100% hit rate is already handled sanely i guess by reducing readahead, precognition would gather access patterns (every n-th block is read so readahead every n-th block, unlikely scenario i guess, but might happen in databases).
How about backward-read-files? Others?
Quoted 5 lines · Guido Fiala
Kernel could do the best to optimize default performance, applications that consider their own optimal behaviour should do so, all other files are kept under default heuristic policy (adaptable, configurable one)
Heuristic can be based on access statistic:
streaming/sequential can be guessed by getting exactly 100% cache hit rate (drop behind pages immediately),
What about a grep through my kernel sources or other linear search through a large directory tree? That would get exactly 100% cache hit rate which would cause your method to drop the pages immediately, meaning that subsequent greps are equally slow. I have enough memory to hold a couple kernel trees and I want my grepping to push OO.org out of RAM for a bit while I do my kernel development.
Cheers,
Kyle Moffett
--
I lost interest in "blade servers" when I found they didn't throw knives at people who weren't supposed to be in your machine room.
-- Anthony de Boer
Quoted 11 lines · Kyle Moffett
Quoted 9 lines · Guido Fiala
Kernel could do the best to optimize default performance, applications that consider their own optimal behaviour should do so, all other files are kept under default heuristic policy (adaptable, configurable one)
Heuristic can be based on access statistic:
streaming/sequential can be guessed by getting exactly 100% cache hit rate (drop behind pages immediately),
Quoted 6 lines
What about a grep through my kernel sources or other linear search through a large directory tree? That would get exactly 100% cache hit rate which would cause your method to drop the pages immediately, meaning that subsequent greps are equally slow. I have enough memory to hold a couple kernel trees and I want my grepping to push OO.org out of RAM for a bit while I do my kernel development.
Ok, it seems this was thought to easy, needs some thinking ;-) Of course i have lots of similar work and dont like to loose the speedup either.
What other useful data for the job do we have already in the structs?