Treat disk space like memory space
The normal situation on the memory is side is that most of the memory is in use, but some pages are ready to be discarded, they're just kept around because we have nothing better to do (yet) with that page.
Is there a tool to do something similar with file systems. I have a lot of unimportant files in my file-system, typically browser caches and things like that, which can be thrown away without any risk of losing data. I just keep them around because I have nothing better to do with my disk space.
So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.
[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than /dev/null that is). ]
Stefan
Quoted 4 lines · Stefan Monnier
So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.
James Cipar, Mark D. Corner, Emery D. Berger: TFS: A Transparent File System for Contributory Storage.
"TFS provides background tasks with large amounts of unreliable storage-all of the currently available space-without impacting the performance of ordinary file access operations."
http://www.usenix.org/event/fast07/tech/cipar.html
http://prisms.cs.umass.edu/tcsm/
It's intended for P2P but I could imagine using it for caches.
Quoted 4 lines · Stefan Monnier
[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than/dev/nullthat is). ]
It would require some kernel support to reclaim the storage from the caching application rather than returning -ENOSPC when a "normal" app needs the storage. Even if you make the caching app free the space by itself you need a kernel mechanism to signal it when this happens.
Lee
Quoted 11 lines · Lee Revell
Quoted 4 lines · Stefan Monnier
[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than/dev/nullthat is). ]It would require some kernel support to reclaim the storage from the caching application rather than returning
-ENOSPCwhen a "normal" app needs the storage. Even if you make the caching app free the space by itself you need a kernel mechanism to signal it when this happens.
In particular, you need a way to hold off the "real" application until disk reclaim is done.
If you do it purely in userspace (asynchronously) then it's subject to ENOSPC while the reclaimer runs.
This, by the way, has been discussed on and off -- often in the context of undelete (which is an identical problem.) The problem usually is that performance of real storage users suffer because of locality issues. However, flash storage doesn't have locality requirements...
-hpa
Quoted 5 lines · H. Peter Anvin
This, by the way, has been discussed on and off -- often in the context of undelete (which is an identical problem.) The problem usually is that performance of real storage users suffer because of locality issues. However, flash storage doesn't have locality requirements...
It does, although significantly less so than disks. Read latency is typically between 100x and 1000x less than disk latency.
Another argument against this is that free space directly translates to speed, both for disks and flash. Disk filesystems fragment like hell if the disk is constanly near-full and flash filesystems require a lot more garbage collection overhead.
Jörn
Signature · 4 lines
To my face you have the audacity to advise me to become a thief - the worst kind of thief that is conceivable, a thief of spiritual things, a thief of ideas! It is insufferable, intolerable! -- M. Binet in Scarabouche
Hi!
Quoted 9 lines
So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.
[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than/dev/nullthat is). ]
Actually, you'll probably need fs hooks for reliable operation. And yes, someone finally implementing this would be welcome :-).
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html