how to scale root-reserved space going forward...
5% of a 16T filesystem is getting a little crazy from the point of view of "root-reserved" - 800G!
But I think the original reason for this reserved space was actually as an allocator cushion; letting root gain access to it was just a safety-valve for that.
Now that we have a completely different allocator in ext4, and potentially much larger filesystems, I think we need to revisit how much is held back, and for what reason.
Any thoughts on a reasonable way to scale this reservation (or, just for discussion - if it's even needed at all today for ext4?)
-Eric
Quoted 6 lines · Eric Sandeen
5% of a 16T filesystem is getting a little crazy from the point of view of "root-reserved" - 800G!
But I think the original reason for this reserved space was actually as an allocator cushion; letting root gain access to it was just a safety-valve for that.
Yep, that's correct. Historically, this came from BSD Fast Filesystem, which used to use a default reserve of 10%. To quote from the FreeBSD sources, in ufs/ffs.h:
/*
* MINFREE gives the minimum acceptable percentage of filesystem
* blocks which may be free. If the freelist drops below this level
* only the superuser may continue to allocate blocks. This may
* be set to 0 if no reserve of free blocks is deemed necessary,
* however throughput drops by fifty percent if the filesystem
* is run at between 95% and 100% full; thus the minimum default
* value of fs_minfree is 5%. However, to get good clustering
* performance, 10% is a better choice. hence we use 10% as our
* default value. With 10% free space, fragmentation is not a
* problem, so we choose to optimize for time.
*/
#define MINFREE 8
The interesting thing is that FreeBSD has decided push things down to 8%. A quick survey shows that NetBSD is using a MINFREE of 5%, like Linux. (Fortunately, http://fxr.watson.org/ makes it easy to make these comparisons.)
And like Linux, it looks like the *BSD's have the same tendency not to update the comments when they update the code. :-)
Quoted 6 lines
Now that we have a completely different allocator in ext4, and potentially much larger filesystems, I think we need to revisit how much is held back, and for what reason.
Any thoughts on a reasonable way to scale this reservation (or, just for discussion - if it's even needed at all today for ext4?)
This is a reasonable question. What would be great is if we could get a benchmarking team to fill an ext4 filesystem with files. The simple thing would be if we did something fixed --- say, 50 files per directory, each file 100k, and say 10 subdirectories in each directory, to some fixed depth, and with a filesystem size of at least 8 gigabytes (which would give us at least 16 flex groups with the default flex size of 16) --- and then filled each filesystem to from 0% to 90% in increments of 10%, and from 90% to 99% in increments of 1%, and then ran some throughput benchmark like bonnie on the mostly filled filesystem.
A better filler would probably use a random file sizes with a average size of say 64k, but with outliers from 4k to 128 megs, and a similar random distribution of number of files per directory, and number of subdirectories and depth of subdirectories.
I suppose it would be good to do one set of charts with a filesystem size of 8 gigs, and another at 80 gigs and 800 gigs, and see if the shape of the filesystem curve changes at scale. Once we have that, we would be in a position to make a reasonable set of defaults.
Or we could just guess and come up with some percentage figure that sounds good. :-)
- Ted
On 03/01/2009 08:47 PM, Theodore Tso wrote:
[snip]
Quoted 3 lines
Or we could just guess and come up with some percentage figure that sounds good. :-)
5% but 100GB is the fs is .ge. 2TB?
Signature · 6 lines
Ron Johnson, Jr. Jefferson LA USA The feeling of disgust at seeing a human female in a Relationship with a chimp male is Homininphobia, and you should be ashamed of yourself.
Quoted 10 lines · Theodore Ts'o
This is a reasonable question. What would be great is if we could get a benchmarking team to fill an ext4 filesystem with files. The simple thing would be if we did something fixed --- say, 50 files per directory, each file 100k, and say 10 subdirectories in each directory, to some fixed depth, and with a filesystem size of at least 8 gigabytes (which would give us at least 16 flex groups with the default flex size of 16) --- and then filled each filesystem to from 0% to 90% in increments of 10%, and from 90% to 99% in increments of 1%, and then ran some throughput benchmark like bonnie on the mostly filled filesystem.
We've done tests like this, and it is important to take the inner vs. outer cyliners into account. It can happen that even a "perfectly" allocated filesystem will appear to show slowdowns in performance as it gets full, yet this is partitially due to physical disk layout issues.
Quoted 4 lines
A better filler would probably use a random file sizes with a average size of say 64k, but with outliers from 4k to 128 megs, and a similar random distribution of number of files per directory, and number of subdirectories and depth of subdirectories.
You describe the Reiserfs "Mongo" benchmark.
Quoted 7 lines
I suppose it would be good to do one set of charts with a filesystem size of 8 gigs, and another at 80 gigs and 800 gigs, and see if the shape of the filesystem curve changes at scale. Once we have that, we would be in a position to make a reasonable set of defaults.
Or we could just guess and come up with some percentage figure that sounds good. :-)
I suspect that at a certain filesystem size, there isn't much benefit in having more reserved space. If we keep 50GB of reserved space then this is likely to contain a decent amount of 1MB free chunks, which is what we really care about.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 6 lines · Andreas Dilger
I suspect that at a certain filesystem size, there isn't much benefit in having more reserved space. If we keep 50GB of reserved space then this is likely to contain a decent amount of 1MB free chunks, which is what we really care about.
Cheers, Andreas
As long as we don't care much about inode<->block locality, that is probably ok...
I got one off-list reply from someone who was depending on the 5% for administration reasons, but even if we change the default I assume we'll leave the tunable in place for people who really do want 5% for "root-reserved" over "allocator safety buffer" reasons.
-Eric