Skip to contentSkip to conversation

Treat disk space like memory space

DISCUSSION Stefan Monnier 6 participants 6 messages Nov 15, 2007 – Nov 25, 2007

lore
Share & tools
lore
#0 lore raw

The normal situation on the memory is side is that most of the memory is in use, but some pages are ready to be discarded, they're just kept around because we have nothing better to do (yet) with that page.

Is there a tool to do something similar with file systems. I have a lot of unimportant files in my file-system, typically browser caches and things like that, which can be thrown away without any risk of losing data. I just keep them around because I have nothing better to do with my disk space.

So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.

[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than /dev/null that is). ]

Stefan

#1 lore raw
Quoted 4 lines · Stefan Monnier

So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.

James Cipar, Mark D. Corner, Emery D. Berger: TFS: A Transparent File System for Contributory Storage.

"TFS provides background tasks with large amounts of unreliable storage-all of the currently available space-without impacting the performance of ordinary file access operations."

http://www.usenix.org/event/fast07/tech/cipar.html
http://prisms.cs.umass.edu/tcsm/

It's intended for P2P but I could imagine using it for caches.

#2 lore raw
In reply to Stefan Monnier
Quoted 4 lines · Stefan Monnier

[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than /dev/null that is). ]

It would require some kernel support to reclaim the storage from the caching application rather than returning -ENOSPC when a "normal" app needs the storage. Even if you make the caching app free the space by itself you need a kernel mechanism to signal it when this happens.

Lee

#3 lore raw
Quoted 11 lines · Lee Revell
Quoted 4 lines · Stefan Monnier

[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than /dev/null that is). ]

It would require some kernel support to reclaim the storage from the caching application rather than returning -ENOSPC when a "normal" app needs the storage. Even if you make the caching app free the space by itself you need a kernel mechanism to signal it when this happens.

In particular, you need a way to hold off the "real" application until disk reclaim is done.

If you do it purely in userspace (asynchronously) then it's subject to ENOSPC while the reclaimer runs.

This, by the way, has been discussed on and off -- often in the context of undelete (which is an identical problem.) The problem usually is that performance of real storage users suffer because of locality issues. However, flash storage doesn't have locality requirements...

-hpa

#4 lore raw
Quoted 5 lines · H. Peter Anvin

This, by the way, has been discussed on and off -- often in the context of undelete (which is an identical problem.) The problem usually is that performance of real storage users suffer because of locality issues. However, flash storage doesn't have locality requirements...

It does, although significantly less so than disks. Read latency is typically between 100x and 1000x less than disk latency.

Another argument against this is that free space directly translates to speed, both for disks and flash. Disk filesystems fragment like hell if the disk is constanly near-full and flash filesystems require a lot more garbage collection overhead.

Jörn

Signature · 4 lines
To my face you have the audacity to advise me to become a thief - the worst
kind of thief that is conceivable, a thief of spiritual things, a thief of
ideas! It is insufferable, intolerable!
-- M. Binet in Scarabouche
#5 lore raw
In reply to Stefan Monnier

Hi!

Quoted 9 lines

So I'd like to be able to say "these areas of my file-system hold data that you can discard whenever you need space". So I can freely fill up my disk with such irrelevant data, safe in the knowledge that if I ever need this disk space it'll be automatically reclaimed.

[ I realize this is probably better implemented outside of the kernel, but
it seems like it might be of interest here. Please redirect me to a more appropriate place if you can think of one (other than /dev/null that is). ]

Actually, you'll probably need fs hooks for reliable operation. And yes, someone finally implementing this would be welcome :-).

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html