tuning XFS for tiny files
Hi. Is there a way to tune XFS filesystem parameters to better address the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directory
My understanding is that XFS will always be much slower than reiserfs with respect to deleting 10000s files; but that XFS might be possible to tune toward more rapid read access of 10000s of tiny files.
Regards,
-timotheus
Quoted 6 lines · timotheus
Hi. Is there a way to tune XFS filesystem parameters to better address the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directory
Quoted 6 lines
My understanding is that XFS will always be much slower than reiserfs with respect to deleting 10000s files; but that XFS might be possible to tune toward more rapid read access of 10000s of tiny files.
Regards,
-timotheus
Do you have a way to benchmark your load?
logv2 at mkfs time, and throwing in larger logbufs & logbsize as mount options might help.
-Eric
Quoted 14 lines · Eric Sandeen
Quoted 2 lines
Hi. Is there a way to tune XFS filesystem parameters to better
address
Quoted 6 lines
the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directoryMy understanding is that XFS will always be much slower than
reiserfs
Quoted 1 line
with respect to deleting 10000s files; but that XFS might be
possible to
Quoted 2 lines
tune toward more rapid read access of 10000s of tiny files.
Quoted 7 lines
Quoted 2 lines
Regards,
-timotheusDo you have a way to benchmark your load?
logv2 at mkfs time, and throwing in larger logbufs & logbsize as mount options might help.
For these kinds of workloads, you may have some joy using a mkfs run with smaller blocksize (down to 512 bytes) and also larger directory blocksize (up to 64k). (e.g. mkfs.xfs -bsize=512 -nsize=16k ...)
cheers.
--
Nathan
Quoted 9 lines · timotheus
Hi. Is there a way to tune XFS filesystem parameters to better address the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directoryMy understanding is that XFS will always be much slower than reiserfs with respect to deleting 10000s files; but that XFS might be possible to tune toward more rapid read access of 10000s of tiny files.
-d agcount=1 at mkfs time might help (unless you have a lot of CPUs)
-Andi
Quoted 8 lines · Andi Kleen
Quoted 6 lines · timotheus
Hi. Is there a way to tune XFS filesystem parameters to better address the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directory
Quoted 5 lines
Quoted 3 lines
My understanding is that XFS will always be much slower than reiserfs with respect to deleting 10000s files; but that XFS might be possible to tune toward more rapid read access of 10000s of tiny files.
-d agcount=1 at mkfs time might help (unless you have a lot of CPUs)
Yeah, might help, but it's not good for being able to repair the filesystem - repair will be unable to find a secondary superblock to compare the primary against and abort.....
-d agcount is only good for science experiments, not production systems ;)
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 13 lines · David Chinner
Quoted 12 lines · Andi Kleen
Quoted 9 lines · timotheus
Hi. Is there a way to tune XFS filesystem parameters to better address the usage pattern of 10000s of tiny files in directories such as:
maildir directory
mh mail directory
ccache directoryMy understanding is that XFS will always be much slower than reiserfs with respect to deleting 10000s files; but that XFS might be possible to tune toward more rapid read access of 10000s of tiny files.
Quoted 5 lines
Quoted 1 line
-d agcount=1 at mkfs time might help (unless you have a lot of CPUs)
Yeah, might help, but it's not good for being able to repair the filesystem - repair will be unable to find a secondary superblock to compare the primary against and abort.....
Any reason why it aborts? It could just continue with a warning, couldn't it?
Quoted 2 lines
-d agcount is only good for science experiments, not production systems ;)
XFS small file performance needs a lot of science.
-Andi