Skip to contentSkip to conversation

Speed of rm compared to reiserfs (slow)

DISCUSSION Török Edwin 4 participants 9 messages Sep 24, 2008 – Sep 28, 2008

lore
Share & tools
lore

Török Edwin

author
#0 lore raw

Hi,

I am happily using xfs for /var, /usr and /, and I am very pleased with the read speed.
I've just recommended xfs to a friend, and he complained about the speed of rm.

I did a test on my box, and indeed the speed of rm is order of magnitude slower compared to reiserfs.
I already use lazy-count, and noatime/nodiratime. Write barriers are off because I run on raid10.

Is there anything else I can tune to get faster rm speed?

Test:
1. svn co svn://gcc.gnu.org/svn/gcc/trunk gcc
2.
# sync; echo 3 >/proc/sys/vm/drop_caches; sync
3. time rm -rf gcc

Results for XFS:
real 1m30.100s
user 0m0.188s
sys 0m11.401s

Results for Reiserfs:
real 0m10.937s
user 0m0.140s
sys 0m6.616

# xfs_info /var
meta-data=/dev/mapper/vg--all-lv--var isize=256    agcount=4,
agsize=85196800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=340787200, imaxpct=5
         =                       sunit=16     swidth=96 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
# mount | grep var
/dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)
# mount | grep reiserfs
/dev/mapper/vg--all-lv--home on /home type reiserfs (rw,noatime,notail)
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md3              9.2G  5.1G  3.7G  58% /
tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
udev                   10M  188K  9.9M   2% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vg--all-lv--usr
                      100G  5.3G   95G   6% /usr
/dev/mapper/vg--all-lv--var
                      1.3T  230G  1.1T  18% /var
/dev/mapper/vg--all-lv--home
                      600G   41G  560G   7% /home
tmpfs                 2.0G  1.9M  2.0G   1% /tmp
[   23.861497] Filesystem "dm-0": Disabling barriers, not supported by
the underlying device
[   23.863915] XFS mounting filesystem dm-0
[   24.050223] Ending clean XFS mount for filesystem: dm-0
[   24.056295] Filesystem "dm-1": Disabling barriers, not supported by
the underlying device
[   24.056295] XFS mounting filesystem dm-1
[   24.248025] Ending clean XFS mount for filesystem: dm-1
[   24.335968] ReiserFS: dm-2: found reiserfs format "3.6" with standard
journal
[   24.335968] ReiserFS: dm-2: using ordered data mode
[   24.359968] ReiserFS: dm-2: journal params: device dm-2, size 8192,
journal first block 18,
max trans len 1024, max batch 900, max commit age 30, max trans age 30
[   24.359968] ReiserFS: dm-2: checking transaction log (dm-2)
[   24.426700] ReiserFS: dm-2: Using r5 hash to sort names

Best regards,
--Edwin

#1 lore raw
Quoted 7 lines · Török Edwin

Hi,

I am happily using xfs for /var, /usr and /, and I am very pleased with the read speed.
I've just recommended xfs to a friend, and he complained about the speed of rm.

Quoted 6 lines

I did a test on my box, and indeed the speed of rm is order of magnitude slower compared to reiserfs.
I already use lazy-count, and noatime/nodiratime. Write barriers are off because I run on raid10.

Is there anything else I can tune to get faster rm speed?

mount -o logbsize=262144 <dev> <mtpt>

Quoted 2 lines
# mount | grep var
/dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)

BTW, noatime implies nodiratime - you don't ned to specify both.

Quoted 7 lines
tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
udev                   10M  188K  9.9M   2% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vg--all-lv--usr
                      100G  5.3G   95G   6% /usr
/dev/mapper/vg--all-lv--var
                      1.3T  230G  1.1T  18% /var

At 1.1T, you probably want to use inode64 for /var. The different allocation strategy of inode32 can be substantially slower than inode64.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#2 lore raw
Quoted 14 lines · Dave Chinner
Quoted 12 lines · Török Edwin

Hi,

I am happily using xfs for /var, /usr and /, and I am very pleased with the read speed.
I've just recommended xfs to a friend, and he complained about the speed of rm.

I did a test on my box, and indeed the speed of rm is order of magnitude slower compared to reiserfs.
I already use lazy-count, and noatime/nodiratime. Write barriers are off because I run on raid10.

Quoted 5 lines
Quoted 2 lines

Is there anything else I can tune to get faster rm speed?

mount -o logbsize=262144 <dev> <mtpt>

I've added it to my mount options, also tried logbufs=8 (but that didn't make much difference).

Quoted 17 lines
Quoted 4 lines
  
# mount | grep var
/dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)

BTW, noatime implies nodiratime - you don't ned to specify both.

Quoted 8 lines
tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
udev                   10M  188K  9.9M   2% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vg--all-lv--usr
                      100G  5.3G   95G   6% /usr
/dev/mapper/vg--all-lv--var
                      1.3T  230G  1.1T  18% /var
Quoted 5 lines

At 1.1T, you probably want to use inode64 for /var. The different allocation strategy of inode32 can be substantially slower than inode64.

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Best regards,
--Edwin

#3 lore raw
Quoted 9 lines · Török Edwin
Quoted 1 line
Quoted 1 line

I did a test on my box, and indeed the speed of rm

is order of magnitude

Quoted 2 lines
Quoted 2 lines

slower compared to reiserfs.
I already use lazy-count, and noatime/nodiratime.

Write barriers are off

Quoted 3 lines
Quoted 3 lines

because I run on raid10.

Is there anything else I can tune to get faster rm

speed?

Others' suggestions stand, but I have found the best way to speed up a journalled filesystem (of any kind) is with an external journal on a separate controller channel. If your XFS journal is internal, the RAID10 layer creates a longer data path when actions are recorded in the journal, then read from the journal, and finally committed to the main filesystem.

You may also investigate the speed of different elevator algorithms. XFS does best with "noop" or "deadline" in my experience. Most Linux distributions ship with "cfq" as their default elevator, which can clash badly against XFS.

#4 lore raw
In reply to Török Edwin
Quoted 5 lines · Török Edwin
Quoted 1 line · Dave Chinner

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

Quoted 7 lines

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive, small file workloads. It scales by being able to keep lots of disks busy at the same time. Those algorithms don't map to single disk configs as efficiently as a filesystem that was specifically designed for optimal performance for these workloads (like reiserfs). We're working on making it better, but that takes time....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#5 lore raw
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Török Edwin
Quoted 2 lines · Dave Chinner

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s
Quoted 7 lines
Quoted 2 lines

Is there anything else I can try to make it faster?

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive, small file workloads.

I have 6 disks, in raid10 :)

md4 : active raid10 sda3[0] sdf3[5] sdc3[4] sde3[3] sdd3[2] sdb3[1]

      2159617728 blocks 64K chunks 2 near-copies [6/6] [UUUUUU]

--- Logical volume ---

  LV Name                /dev/vg-all/lv-var
  VG Name                vg-all
  LV UUID                CQHPts-K3OE-9kWV-hg7q-328i-RP0i-Dew94c
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.27 TB
  Current LE             332800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

--- Segments ---
Logical extent 0 to 332799:

    Type                linear
    Physical volume     /dev/md4
    Physical extents    25600 to 358399
Quoted 5 lines

It scales by being able to keep lots of disks
busy at the same time. Those algorithms don't map to single disk configs as efficiently as a filesystem that was specifically designed for optimal performance for these workloads (like reiserfs). We're working on making it better, but that takes time....

I see.
Well the read performance is very good as I said in my initial email ;)

Thanks,
--Edwin

author
#6 lore raw

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

Quoted 23 lines · Török Edwin
Quoted 21 lines · Dave Chinner
Quoted 18 lines · Török Edwin
Quoted 3 lines · Dave Chinner

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Quoted 8 lines
Quoted 5 lines

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive, small file workloads.

I have 6 disks, in raid10 :)

I moved the logdevice to another disk (not part of RAID array), and now the speed of rm is down to 30 seconds!

I didn't find any documentation on how to convert an XFS fs with internal log device to use external, so I did the following:

# xfs_db -x /dev/mapper/vg--all-lv--opt
xfs_db>sb
xfs_db> write logstart 0
xfs_db> quit

The zero the logdevice, and mount using logdev=. This worked.

However if I now use xfs_repair on this filesystem it restores the internal log. Is there a way to make this change permanent? (i.e. switch permanently to using external log?)

Best regards,
--Edwin

#7 lore raw

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

Quoted 43 lines · Török Edwin
Quoted 23 lines · Török Edwin
Quoted 21 lines · Dave Chinner
Quoted 18 lines · Török Edwin
Quoted 3 lines · Dave Chinner

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Quoted 7 lines
Quoted 5 lines

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive, small file workloads.

I have 6 disks, in raid10 :)

I moved the logdevice to another disk (not part of RAID array), and now the speed of rm is down to 30 seconds!

I didn't find any documentation on how to convert an XFS fs with internal log device to use external, so I did the following:

# xfs_db -x /dev/mapper/vg--all-lv--opt
xfs_db>sb
xfs_db> write logstart 0
xfs_db> quit
Quoted 5 lines

The zero the logdevice, and mount using logdev=. This worked.

However if I now use xfs_repair on this filesystem it restores the internal log. Is there a way to make this change permanent? (i.e. switch permanently to using external log?)

Odds are you need to also set the logstart in the other superblocks:

xfs_db> sb 0
xfs_db> write logstart 0
xfs_db> sb 1
xfs_db> write logstart 0

...

you could script this with xfs_db -c

-Eric

Quoted 4 lines

Best regards,
--Edwin

author
#8 lore raw

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

Quoted 60 lines · Eric Sandeen
Quoted 49 lines · Török Edwin
Quoted 28 lines · Török Edwin
Quoted 25 lines · Dave Chinner
Quoted 21 lines · Török Edwin
Quoted 4 lines · Dave Chinner

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:

Thanks for the suggestions, the time for rm has improved a bit, but is still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Quoted 9 lines
Quoted 6 lines

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive, small file workloads.

I have 6 disks, in raid10 :)

I moved the logdevice to another disk (not part of RAID array), and now the speed of rm is down to 30 seconds!

I didn't find any documentation on how to convert an XFS fs with internal log device to use external, so I did the following:

# xfs_db -x /dev/mapper/vg--all-lv--opt
xfs_db>sb
xfs_db> write logstart 0
xfs_db> quit
Quoted 6 lines

The zero the logdevice, and mount using logdev=. This worked.

However if I now use xfs_repair on this filesystem it restores the internal log. Is there a way to make this change permanent? (i.e. switch permanently to using external log?)

Odds are you need to also set the logstart in the other superblocks:

Quoted 9 lines

xfs_db> sb 0
xfs_db> write logstart 0
xfs_db> sb 1
xfs_db> write logstart 0

...

you could script this with xfs_db -c

Thank you, xfs_repair now no longer changes the logdevice.

Best regards,
--Edwin