XFS for lots of small files

Hello!

I consider moving server from reiserfs to xfs. In all benchmarks I have read both file systems have had comparable results.

But I've made a test:

1. formated /dev/hda2 with reiserfs with default options and made 10.000 files
2. formated /dev/hda2 with xfs with default options and made 10.000

Reiserfs created those files in 2 (two) seconds, and xfs created them in 35 (thirty five) seconds.

Is that normal? What I am doing wrong?

My system is Debian, current stable version. Below is a log of operation.

Thanks in advance.

debian:/mnt/hdc2# mkfs.xfs /dev/hdc2 -f; mount /dev/hdc2 /mnt/hdc2; cd /mnt/hdc2; mount | egrep hdc2
meta-data=/dev/hdc2              isize=256    agcount=16, agsize=1220688 blks
        =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=19531008, imaxpct=25
        =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  log      =internal log           bsize=4096   blocks=9536, version=1
        =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0
/dev/hdc2 on /mnt/hdc2 type xfs (rw)

debian:/mnt/hdc2# time for f in `seq 9999`; do echo $f > $f; done

real 0m35.558s
user 0m0.256s
sys 0m1.080s

debian:/mnt/hdc2# time cat * | wc -l 9999

real 0m0.239s
user 0m0.020s
sys 0m0.172s

debian:/mnt/hdc2# cd /; umount /dev/hdc2; mkreiserfs /dev/hdc2 -f; mount /dev/hdc2 /mnt/hdc2; cd /mnt/hdc2; mount | egrep hdc2
mkreiserfs 3.6.19 (2003 www.namesys.com)
[...]
ReiserFS is successfully created on /dev/hdc2.
/dev/hdc2 on /mnt/hdc2 type reiserfs (rw)

debian:/mnt/hdc2# time for f in `seq 9999`; do echo $f > $f; done

real 0m1.075s
user 0m0.232s
sys 0m0.844s

debian:/mnt/hdc2# time cat * | wc -l
9999

real 0m0.257s
user 0m0.028s
sys 0m0.256s

debian:/mnt/hdc2# exit

Hello,

For metadata intensive work, you should consider a little bit of tuning.

See this usefull advice:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/21239/focus=21240

Nicolas

Am Dienstag 06 Mai 2008 schrieb Leszek Dubiel:

Hi Leszek,

[...]

I get

martin@shambala:~/Zeit/filetest -> rm *; sync ; time for ((I=1; I<=10000; I=I+1)); do echo $I > $I; done

real    0m10.642s
user    0m0.907s
sys     0m1.713s

martin@shambala:~/Zeit/filetest -> sync ; time cat * >/dev/null

real    0m0.238s
user    0m0.087s
sys     0m0.153s

martin@shambala:~/Zeit/filetest -> sync ; time cat * | wc -l 10000

real    0m0.375s
user    0m0.120s
sys     0m0.247s

martin@shambala:~/Zeit/filetest -> sync ; time rm *

real    0m7.600s
user    0m0.113s
sys     0m1.377s

for XFS with optimized settings...

shambala> xfs_info /home
meta-data=/dev/sda5 isize=256 agcount=6, agsize=4883256
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=29299536, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

agcount is two more than would be optimal cause I growed the partition once.

shambala> mount | grep home
/dev/sda5 on /home type xfs (rw,relatime,logbsize=256k,logbufs=8)

This is on a ThinkPad T42 internal laptop 160 GB harddisk drive with I think 5400rpm.

Partition I tested on was not empty at that time and is heavily used.

shambala> LANG=EN df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 112G 84G 29G 75% /home

And there is quite some fragmentation on it:

xfs_db> frag
actual 653519, ideal 587066, fragmentation factor 10.17%

I do not have free space in my playground LVM to test against ext3 and reiserfs at the moment.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

----
Not to disturb you, but are you running xfs_fsr on a
regular basis?

I have a shell script that runs in my 'cron.daily' that has two lines:
----
#!/bin/bash
ionice -c 3 /usr/sbin/xfs_fsr &
-------

I've never noticed fragmentation above 1% on my disks...but your mileage may vary ....

Hi Linda,

Am Freitag 23 Mai 2008 schrieb Linda Walsh:

No, not yet. I had those bad experiences with xfs_fsr on a ck patchset patched kernel and after this I didn't try again. Its pretty sure that my file content corruption issues was related to the ck patchset as I was not able to reproduce it with a CFS or mainline kernel. But still I have been reluctant. I think, I give it a try after I made my backups.

Lets see ;-)

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

No, there's not.

You have 653519 extents out of an "ideal" 587066.

That is 653519/587066 = 1.113 extents per file.

It is not "quite some" fragmentation, it is near perfect (although this is subjective, and also depends on the size of your files... if they are all 8k then 1.113 extents per file might be a bit high; if they average 1G then 1.113 extents on average is pretty darned good.)

-Eric

Am Sonntag 25 Mai 2008 schrieb Eric Sandeen:

OK, so there is or better was (see below) *some* fragmentation.

They vary a lot. From KMail ~/Mail directory with hundred of thousands of mails in maildir format to a picture and movie collection from various digicams with 150KB over 2-4 MB to 50-200 MB in size and a music collection and kernel sources and and and... would need to run a tool on them to gather some statistic.

Anyway, nothing that can't be optimized:

shambala> xfs_db -r /dev/sda5
xfs_db> frag
actual 683648, ideal 617593, fragmentation factor 9.66% xfs_db> quit

shambala> xfs_fsr /dev/sda5
/home start inode=0

shambala> xfs_db -r /dev/sda5
xfs_db> frag
actual 620316, ideal 617584, fragmentation factor 0.44% xfs_db> quit

xfs_fsr copied over several gigabytes and the free space of the partition temporarily more than once was 4 GB less than the 20 GB of free space it had before and after invoking xfs_fsr ;)

Not that I noticed a difference up to now however.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Sure... I'd just argue that it's diminishing returns :)

fsr needs to preallocate space to "defragment into" so this is expected, temporarily.

right, my original reply was meant to imply that fragmentation is not really a problem for you. :)

And in the larger picture, I just wanted to point out that the "fragmentation factor" can be pretty misleading. It reports as (actual
- ideal) / actual.

Imagine a filesystem full of 8GB dvd iso images, each with 4 2GB extents. The fragmentation factor would be reported as (4X - 1X) / 4X = 75%. Which looks "bad," but really isn't.

-Eric