Skip to contentSkip to conversation

Strange delete performance using XFS

DISCUSSION Thomas Kaehn 4 participants 24 messages Apr 4, 2007 – Apr 11, 2007

lore
Share & tools
lore

Thomas Kaehn

author
#0 lore raw

Hi,

I've got a strange problem on one machine using XFS. Deleting large directories (containing about 100000 files, 20k each) using "rm -rf" lasts nearly as long as creating the the files using a bash loop.

The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel.
CPU: Dual Xeon(TM) CPU 3.20GHz
RAM: 4 GB
RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP
(Firmware Version = FE9X 3.08.00.004)

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

Has anyone realized similar effects? On a different server (Dell 6850) the directory can be deleted within seconds. What could be the reason for the huge difference in delete performance?

Please see below for "time" output.

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
| 
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s

In contrast to this the result on the Dell machine looks more reasonable:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    9m26.658s
| user    0m24.134s
| sys     3m3.623s
| # time rm -rf x
| 
| real    0m10.254s
| user    0m0.124s
| sys     0m10.105s

Ciao,
Thomas

PS: Using JFS and ext3 it is also possible to delete the above directory in a couple of seconds. Only XFS seems problematic in this regard on this system.

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#1 lore raw
Quoted 47 lines · Thomas Kaehn

Hi,

I've got a strange problem on one machine using XFS. Deleting large directories (containing about 100000 files, 20k each) using "rm -rf" lasts nearly as long as creating the the files using a bash loop.

The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel.
CPU: Dual Xeon(TM) CPU 3.20GHz
RAM: 4 GB
RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP (Firmware Version = FE9X 3.08.00.004)

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

Has anyone realized similar effects? On a different server (Dell 6850) the directory can be deleted within seconds. What could be the reason for the huge difference in delete performance?

Please see below for "time" output.

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s

In contrast to this the result on the Dell machine looks more reasonable:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
|
| real    9m26.658s
| user    0m24.134s
| sys     3m3.623s
| # time rm -rf x
|
| real    0m10.254s
| user    0m0.124s
| sys     0m10.105s
Quoted 16 lines

Ciao,
Thomas

PS: Using JFS and ext3 it is also possible to delete the above directory in a couple of seconds. Only XFS seems problematic in this regard on this system.

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

author
#2 lore raw

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#3 lore raw
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 16 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

I am running some benchmarks with SW raid and will prevent my findings shortly.

#4 lore raw
Quoted 55 lines · Justin Piszcz
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 16 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
Quoted 1 line

I am running some benchmarks with SW raid and will prevent my findings shortly.

Removal tests coming shortly, benchmarking is always interesting.

#5 lore raw
In reply to Thomas Kaehn
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 16 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

The benchmark:

$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:

real    6m24.411s
user    0m43.097s
sys     2m17.350s

2. Four Raptor 150 ADFD drives using SW RAID5:

real    3m16.962s
user    0m42.899s
sys     2m15.420s

3. Two Raptor 74GB *GD drives using SW RAID1:

real    3m19.241s
user    0m41.731s
sys     2m15.873s
#6 lore raw
Quoted 65 lines · Justin Piszcz
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 16 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

The benchmark:

$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:

real    6m24.411s
user    0m43.097s
sys     2m17.350s

2. Four Raptor 150 ADFD drives using SW RAID5:

Quoted 10 lines
real    3m16.962s
user    0m42.899s
sys     2m15.420s

3. Two Raptor 74GB *GD drives using SW RAID1:

real    3m19.241s
user    0m41.731s
sys     2m15.873s

The removals:

The benchmark:

$ time rm -rf test

1. Six 400GB SATA drives using SW RAID5:

real    0m33.996s
user    0m0.057s
sys     0m8.101s

2. Four Raptor 150 ADFD drives using SW RAID5:

real    0m43.967s
user    0m0.071s
sys     0m8.340s

3. Two Raptor 74GB *GD drives using SW RAID1:

real    0m32.965s
user    0m0.049s
sys     0m6.307s
author
#7 lore raw

Hi Justin,

Quoted 11 lines · Justin Piszcz
Quoted 9 lines · Justin Piszcz

On Wed, 4 Apr 2007, Thomas Kaehn wrote:

$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:

real    6m24.411s
user    0m43.097s
sys     2m17.350s
Quoted 8 lines

The removals:
The benchmark:

$ time rm -rf test

1. Six 400GB SATA drives using SW RAID5:

real    0m33.996s
user    0m0.057s
sys     0m8.101s

thanks for your bechmark. To my mind this clearly shows that my setup is wrong at some point. I'll try again with your mount options.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#8 lore raw
Quoted 23 lines · Thomas Kaehn

Hi Justin,

Quoted 11 lines · Justin Piszcz
Quoted 9 lines · Justin Piszcz

On Wed, 4 Apr 2007, Thomas Kaehn wrote:

$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:

real    6m24.411s
user    0m43.097s
sys     2m17.350s
Quoted 8 lines

The removals:
The benchmark:

$ time rm -rf test

1. Six 400GB SATA drives using SW RAID5:

real    0m33.996s
user    0m0.057s
sys     0m8.101s
Quoted 15 lines

thanks for your bechmark. To my mind this clearly shows that my setup is wrong at some point. I'll try again with your mount options.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

My guess is mkfs.xfs cannot optimzie for your array like it can with a SW RAID device because it cannot see what is undereath it. Have you tried making a SW RAID? I also use optimized parameters for my SW RAID1/5 as well FYI.

Justin.

author
#9 lore raw

Hi Justin,

Quoted 10 lines · Justin Piszcz
Quoted 6 lines · Justin Piszcz
Quoted 5 lines · Justin Piszcz
Quoted 4 lines · Thomas Kaehn
$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

My guess is mkfs.xfs cannot optimzie for your array like it can with a SW RAID device because it cannot see what is undereath it. Have you tried making a SW RAID? I also use optimized parameters for my SW RAID1/5 as well FYI.

I guess this might be the problem. I've already tried to alter the stripe unit to match the RAID stripe size: "-d su=64k,sw=2 -l su=64k".

Maybe the 3ware controller can't deal with the kind of read and write patterns needed by XFS. But in this case other people should have realized similar problems.

On a different system with a 3ware 9500S-4LP using 4 disks as RAID5 setup I get a better (but not really good) result for delete performance (I've taken only 50000 files in this case as the system's CPU is much slower):

| # time for i in `seq 1 50000`; do dd if=/dev/zero of=$i
| bs=1k count=20 >/dev/null 2>&1; done
| 
| real    18m21.643s
| user    0m55.727s
| sys     3m12.140s
| backup:/srv/x# cd ..
| backup:/srv# rm -rf x
| 
| # time rm -rf x
| 
| real    5m7.845s
| user    0m0.160s
| sys     0m11.369s

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#10 lore raw
Quoted 42 lines · Thomas Kaehn

Hi Justin,

Quoted 10 lines · Justin Piszcz
Quoted 6 lines · Justin Piszcz
Quoted 5 lines · Justin Piszcz
Quoted 4 lines · Thomas Kaehn
$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

My guess is mkfs.xfs cannot optimzie for your array like it can with a SW RAID device because it cannot see what is undereath it. Have you tried making a SW RAID? I also use optimized parameters for my SW RAID1/5 as well FYI.

I guess this might be the problem. I've already tried to alter the stripe unit to match the RAID stripe size: "-d su=64k,sw=2 -l su=64k".

Maybe the 3ware controller can't deal with the kind of read and write patterns needed by XFS. But in this case other people should have realized similar problems.

On a different system with a 3ware 9500S-4LP using 4 disks as RAID5 setup I get a better (but not really good) result for delete performance (I've taken only 50000 files in this case as the system's CPU is much slower):

| # time for i in `seq 1 50000`; do dd if=/dev/zero of=$i
| bs=1k count=20 >/dev/null 2>&1; done
|
| real    18m21.643s
| user    0m55.727s
| sys     3m12.140s
| backup:/srv/x# cd ..
| backup:/srv# rm -rf x
|
| # time rm -rf x
|
| real    5m7.845s
| user    0m0.160s
| sys     0m11.369s
Quoted 11 lines

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

What do you get with ext3 when using time bash -c 'rm -f file; sync'

#11 lore raw
In reply to Justin Piszcz
Quoted 65 lines · Justin Piszcz
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 16 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

The benchmark:

$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
Quoted 1 line

/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:

real    6m24.411s
user    0m43.097s
sys     2m17.350s

2. Four Raptor 150 ADFD drives using SW RAID5:

Quoted 10 lines
real    3m16.962s
user    0m42.899s
sys     2m15.420s

3. Two Raptor 74GB *GD drives using SW RAID1:

real    3m19.241s
user    0m41.731s
sys     2m15.873s

I used the DEFAULT create options for XFS as I find it highly optimizes itself (at least with SW raid) with the exception of the ROOT FS, I had that optimized awhile ago and I kept it:

/dev/md2 / xfs logbufs=8,logbsize=262144,biosize=16,noatime,nodiratime,nobarrier 0 1

For my regular RAID5s though I use defaults,noatime.

Justin.

author
#12 lore raw

Hi Justin,

Quoted 9 lines · Justin Piszcz

I used the DEFAULT create options for XFS as I find it highly optimizes itself (at least with SW raid) with the exception of the ROOT FS, I had that optimized awhile ago and I kept it:

/dev/md2 / xfs
logbufs=8,logbsize=262144,biosize=16,noatime,nodiratime,nobarrier 0 1

For my regular RAID5s though I use defaults,noatime.

I've disabled barriers, too, and performance increased dramatically.

However I am not aware of the consequences of disabling write barriers. The FAQ generally recommends using write barriers except when having a battery-backed cache (this 3ware has not).

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    3m52.182s
| user    0m30.482s
| sys     3m16.152s
| 
| # time \rm -rf y
| 
| real    0m16.327s
| user    0m0.052s
| sys     0m8.305s

So I am unsure if disabling is an option for me. I could imagine that write barriers are not properly supported by 3ware or have to be fine tuned at the kernel or SCSI level.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#13 lore raw
In reply to Thomas Kaehn
Quoted 34 lines · Thomas Kaehn

Hi Justin,

Quoted 14 lines · Justin Piszcz
Quoted 4 lines

Please see below for "time" output.

| # time for i in seq 1 100000; do dd if=/dev/zero of=$i bs=1k count=20

Quoted 1 line

/dev/null 2>&1; done

Quoted 9 lines
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 3 lines

Deletes on XFS is one area that is a little slower than other filesystems. You can increase the log size during the creation of the filesystem and also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

Quoted 15 lines

However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

For the ext3, try time bash -c 'rm -rf test; sync'

author
#14 lore raw

Hi Justin,

Quoted 1 line · Justin Piszcz

For the ext3, try time bash -c 'rm -rf test; sync'

# time bash -c 'rm -rf y; sync'
real    0m1.592s
user    0m0.032s
sys     0m1.408s

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#15 lore raw
In reply to Thomas Kaehn
Quoted 3 lines · Thomas Kaehn

I've got a strange problem on one machine using XFS. Deleting large directories (containing about 100000 files, 20k each) using "rm -rf" lasts nearly as long as creating the the files using a bash loop.

quite possible

Quoted 8 lines

RAM: 4 GB
RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP
(Firmware Version = FE9X 3.08.00.004)

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

have you tried w/o using the hw raid?

Quoted 3 lines

Has anyone realized similar effects? On a different server (Dell 6850) the directory can be deleted within seconds. What could be the reason for the huge difference in delete performance?

a lot of log updates; does the other server have a battery-backed write-cache like many cards to these days?

Quoted 5 lines
| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s

that's about the same as my quick single-spindle cheap-desktop test here

Quoted 5 lines
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s

v2 logs? what logbufs & logbsize is used?

testing with my cheap crappy desktop workstation thing with a single disk I get "1m25.004s" for the delete

author
#16 lore raw

Hi Chris,

Quoted 6 lines · Chris Wedgwood
Quoted 3 lines · Thomas Kaehn

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

have you tried w/o using the hw raid?

I'am going to test with a single disk in the same machine.

Quoted 6 lines
Quoted 3 lines

Has anyone realized similar effects? On a different server (Dell 6850) the directory can be deleted within seconds. What could be the reason for the huge difference in delete performance?

a lot of log updates; does the other server have a battery-backed write-cache like many cards to these days?

The Dell system has got a battery-backed write-cache. The 3ware system has no battery unit. However it's supposed to provide write cache, too. At least I've enabled it in the RAID's configuration. The controller has got more than 100MB memory.

Quoted 15 lines
Quoted 5 lines
| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
|
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s

that's about the same as my quick single-spindle cheap-desktop test here

Quoted 5 lines
| # time rm -rf y
|
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s
Quoted 4 lines

v2 logs? what logbufs & logbsize is used?

testing with my cheap crappy desktop workstation thing with a single disk I get "1m25.004s" for the delete

Your delete time sounds more sensible to me. The file system was created first with default options - later on I tried to match the RAID's stripe size, increased the log size and mounted with logbufs=8:

log stripe unit specified, using v2 logs

meta-data=/dev/sda1              isize=256    agcount=8, agsize=125008 blks
         =                       sectsz=512  
data     =                       bsize=4096   blocks=1000032, imaxpct=25
         =                       sunit=16     swidth=32 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=16 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
author
#17 lore raw

Hi Chris,

Quoted 9 lines · Thomas Kaehn
Quoted 6 lines · Chris Wedgwood
Quoted 3 lines · Thomas Kaehn

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

have you tried w/o using the hw raid?

I'am going to test with a single disk in the same machine.

this is what I got with a single disk (defaults for mkfs.xfs, logbufs=8 for mount) in the same machine:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    11m22.487s
| user    0m30.278s
| sys     2m33.762s
| # time \rm -rf y
| 
| real    8m20.963s
| user    0m0.056s
| sys     0m7.968s

So there is no improvement for a single disk.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#18 lore raw
Quoted 27 lines · Thomas Kaehn

Hi Chris,

Quoted 9 lines · Thomas Kaehn
Quoted 6 lines · Chris Wedgwood
Quoted 3 lines · Thomas Kaehn

The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance.

have you tried w/o using the hw raid?

I'am going to test with a single disk in the same machine.

this is what I got with a single disk (defaults for mkfs.xfs, logbufs=8 for mount) in the same machine:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
|
| real    11m22.487s
| user    0m30.278s
| sys     2m33.762s
| # time \rm -rf y
|
| real    8m20.963s
| user    0m0.056s
| sys     0m7.968s
Quoted 14 lines

So there is no improvement for a single disk.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

What kind of disks are you using? Maybe just slow disks??

author
#19 lore raw

Hi Justin,

Quoted 3 lines · Justin Piszcz
Quoted 2 lines

So there is no improvement for a single disk.

What kind of disks are you using? Maybe just slow disks??

I am using the following disks:

http://www.westerndigital.com/en/products/products.asp?DriveID=233

When disabling write barriers delete times are OK. I think that the 3ware RAID controller could have a problem with it. I'll try to contact 3ware in order to come to know if this feature is supported or not.

Additionally I am going to try out some advices presented in the 3ware knowledge base.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
#20 lore raw
Quoted 16 lines · Thomas Kaehn

Hi Justin,

Quoted 3 lines · Justin Piszcz
Quoted 2 lines

So there is no improvement for a single disk.

What kind of disks are you using? Maybe just slow disks??

I am using the following disks:

http://www.westerndigital.com/en/products/products.asp?DriveID=233

When disabling write barriers delete times are OK. I think that the 3ware RAID controller could have a problem with it. I'll try to contact 3ware in order to come to know if this feature is supported or not.

Quoted 15 lines

Additionally I am going to try out some advices presented in the 3ware knowledge base.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                             Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Ah, ok-- Keep us updated/let us know if you get any new findings/etc.

Something else you can try as well is turning off NCQ, that gave me a 10-35% performance boost depending on the benchmark.

Justin.

#21 lore raw
In reply to Thomas Kaehn
Quoted 3 lines · Thomas Kaehn

The Dell system has got a battery-backed write-cache. The 3ware system has no battery unit. However it's supposed to provide write cache, too.

That sounds like the main reason for the difference. The Dell's raid system can safely buffer outstanding writes and flsuh them, the 3ware can't so it stalls waiting fot the disks to catch up.

You could run blktrace and watch what's going on in both cases to verify this.

The numbers do seem a little low for a raid array all the same, I'd be tempted to just use the 3ware as a JBOD and use sw, but I'm arguably biased, I've had so many reliability and performance problems with hw raid over the years I will almost always use sw raid given the choice.

#22 lore raw

[ ... slowness deleting a lot of inodes ... ]

Quoted 3 lines
Quoted 3 lines

The Dell system has got a battery-backed write-cache. The 3ware system has no battery unit. However it's supposed to provide write cache, too.

Whatever, but you cannot have both metadata consistency and high speed without fully reliable hw...

Quoted 6 lines

[ ... ] The Dell's raid system can safely buffer outstanding writes and flsuh them, the 3ware can't so it stalls waiting fot the disks to catch up. [ ... ] I'd be tempted to just use the 3ware as a JBOD and use sw, but I'm arguably biased, I've had so many reliability and performance problems with hw raid over the years

Uhm, I had a friend that worked for a middling storage system vendor and he was telling me horror stories about bugs and misdesigns in their quite popular RAID products.

3ware seem to me one of the more reliable RAID brands, their cautious approach may be why they are slower above.

Quoted 1 line

I will almost always use sw raid given the choice.

Does not buy you a lot over well designed RAID host adapter. it is also a lot less convenient.

author
#23 lore raw

Hi Peter,

Quoted 7 lines · Peter Grandi
Quoted 6 lines

[ ... ] The Dell's raid system can safely buffer outstanding writes and flsuh them, the 3ware can't so it stalls waiting fot the disks to catch up. [ ... ] I'd be tempted to just use the 3ware as a JBOD and use sw, but I'm arguably biased, I've had so many reliability and performance problems with hw raid over the years

Quoted 6 lines

Uhm, I had a friend that worked for a middling storage system vendor and he was telling me horror stories about bugs and misdesigns in their quite popular RAID products.

3ware seem to me one of the more reliable RAID brands, their cautious approach may be why they are slower above.

your are probably right. 3ware didn't answer yet. However I've found an option in the controller to set the "storsave" policy.

In the default profile FUA (force unit access) commands are only acknowledged directly in case a BBU is present. Otherwise the controller waits until the data is written to disk.

When selecting the "performance" profile FUA commands are ignored and delete time lowers to a couple of seconds.

So the behaviour of the controller should be considered a feature. But I am still astonished how slow deletes were in the first place. This might be a bug or incompatibility anyhow.

Thanks to all others for your suggestions. I'll inform you in case 3ware has news for me.

Ciao,
Thomas

Signature · 7 lines
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
[email protected]                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb