XFS unlink still slow on 3.1.9 kernel ?
Hi Stan,
I was not subscribed to the list so I did not get your mail. I subscribed now, but apparently and admin has to aprove the subscription since I am not getting any confirmation email. so this will take some time.
Quoted 8 lines · Richard Ems
Quoted 1 line
The case is connected through SCSI.
Do you mean iSCSI? Does the host on which you're running your "find dir" command have a 1GbE or 10GbE connection to the InforTrend unit? More than one connection using bonding or multipath? Direct connected or through a switch(es)? What brand is the switch(es)? Switch(es) under heavy load?
NO. SCSI. Not iSCSI. No switches, it's direct attached through a LSI SCSI Controller:
lspci shows:
LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
Quoted 6 lines
If it's a single direct 1GbE connection it's possible you're running out of host pipe bandwidth which is only ~100MB/s in each direction. Check iotop/iostat while running your command to see if you're peaking the interface with either read or write bytes. If either are at or above 100MB/s then your host pipe is full, thus this is a significant part of your high run time problem.
This does not apply, or?
Quoted 5 lines
Also check the performance data in the management interface on the InforTrend unit to see if you're hitting any limits there (if it has such a feature). RAID6 is numerically intensive and that particular controller may not have the ASIC horsepower to keep up with the IOPS workload you're throwing at it.
There is no performance data in the management interface to check.
Quoted 2 lines
Lastly, please paste the exact command or script you refer to as "find dir" which is generating the workload in question.
See previous answers to the list. You will see all commands there.
Thanks,
Richard
Signature · 12 lines
Richard Ems mail: [email protected] Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Jeffrey,
Quoted 31 lines · Jeffrey Hundstad
Richard,
Someone asked if you used inode64. I didn't see a response that you did. Inode64 is a mount option. I bet this will help with your problem. It appears that all the inodes will be (by default, without the inode64 option) in the first 1TB of disk. This could cause a LOT of seeks. BTW: the option by itself will not help. You'll need to save/restore to have this help. However, I suspect over time it will help if files old files are replaced by new ones.
For example:
mount -o inode64/dev/sda1/home/Here's some documentation:
mount(8): inode64
Indicates that XFS is allowed to create inodes at any location in the filesystem, including those which will result in inode numbers occupying more than 32 bits of significance. This is provided for backwards compatibility, but causes problems for backup applications that cannot handle large inode numbers.http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F
Q: What is the inode64 mount option for?By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.
Quoted 3 lines
To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.
What about that programs using only 32-bit stat() ?
Quoted 6 lines
Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.
yes, I replied to Christoph's question stating that I am not using inode64. My reply was:
"
No, I did not use it, but I was thinking about and ran the script from
http://sandeen.net/misc/summarise_stat.pl and got as an example on /bin:
# /net/c3m/usr/local/software/XFS/summarise_stat.pl /bin/
9 6.2% are scripts (shell, perl, whatever)
65 44.8% don't use any stat() family calls at all 61 42.1% use 32-bit stat() family interfaces only
9 6.2% use 64-bit stat64() family interfaces only 1 0.7% use both 32-bit and 64-bit stat() family interfaces
So I was not sure if I should use inode64 or not.
"
Thanks, Richard
Signature · 12 lines
Richard Ems mail: [email protected] Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Richard Ems
# /net/c3m/usr/local/software/XFS/summarise_stat.pl
/bin/
9 6.2% are scripts (shell, perl, whatever)
65 44.8% don't use anystat()family calls at all 61 42.1% use 32-bitstat()family interfaces only
9 6.2% use 64-bitstat64()family interfaces only 1 0.7% use both 32-bit and 64-bitstat()family interfacesSo I was not sure if I should use inode64 or not.
Are you on a 32-bit system (userspace, kernel doesn't matter)?
If the system is 64-bit even the plain stat handles 64-bit inodes just fine.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Christoph Hellwig
Quoted 8 lines · Richard Ems
# /net/c3m/usr/local/software/XFS/summarise_stat.pl
/bin/
9 6.2% are scripts (shell, perl, whatever)
65 44.8% don't use anystat()family calls at all 61 42.1% use 32-bitstat()family interfaces only
9 6.2% use 64-bitstat64()family interfaces only 1 0.7% use both 32-bit and 64-bitstat()family interfacesSo I was not sure if I should use inode64 or not.
Are you on a 32-bit system (userspace, kernel doesn't matter)?
No, I have been running on 64-bit systems for years. That output above is on openSUSE 12.1 64-bit.
Quoted 2 lines
If the system is 64-bit even the plain stat handles 64-bit inodes just fine.
With *plain stat* you mean the *32-bit stat* listed above ? Is it then safe to switch all XFS partitions to be mounted with inode64 on 64-bit systems? Also for NFS v3 exports?
Quoted 1 line
From the XFS FAQ I read that it should work, also for NFS v3, and that I
can try it and switch back if there are problems, so I will give it a try!
Many thanks again for your time and help!
Richard
Signature · 12 lines
Richard Ems mail: [email protected] Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Richard Ems
Hi Stan,
I was not subscribed to the list so I did not get your mail. I
Ahh, my bad. I should have done a reply-all. I sometimes forget XFS is an open list.
Quoted 6 lines
subscribed now, but apparently and admin has to aprove the subscription since I am not getting any confirmation email. so this will take some time.
Quoted 3 lines · Richard Ems
Quoted 1 line
The case is connected through SCSI.
Quoted 8 lines
Quoted 5 lines
Do you mean iSCSI? Does the host on which you're running your "find dir" command have a 1GbE or 10GbE connection to the InforTrend unit? More than one connection using bonding or multipath? Direct connected or through a switch(es)? What brand is the switch(es)? Switch(es) under heavy load?
NO. SCSI. Not iSCSI. No switches, it's direct attached through a LSI SCSI Controller:
Ok so it's obviously not a host bandwidth or latency issue then. I was just trying to cover the all the bases with my hardware questions.
Also I didn't realize anyone was (still?) offering parallel SCSI host connections on their SATA arrays.
Quoted 3 lines
lspci shows:
LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
PCI-X? So this host and array have a few years on them I assume.
Quoted 7 lines
Quoted 5 lines
Also check the performance data in the management interface on the InforTrend unit to see if you're hitting any limits there (if it has such a feature). RAID6 is numerically intensive and that particular controller may not have the ASIC horsepower to keep up with the IOPS workload you're throwing at it.
There is no performance data in the management interface to check.
Ok, so this is a relatively basic, entry level array?
Signature · 7 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Stan Hoeppner
Quoted 3 lines · Richard Ems
lspci shows:
LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
PCI-X? So this host and array have a few years on them I assume.
Yes. Bought 2007.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Also check the performance data in the management interface on the InforTrend unit to see if you're hitting any limits there (if it has such a feature). RAID6 is numerically intensive and that particular controller may not have the ASIC horsepower to keep up with the IOPS workload you're throwing at it.
There is no performance data in the management interface to check.
Ok, so this is a relatively basic, entry level array?
It's an InforTrend InforTrend A24U-G2421, see
http://www.infortrend.com/main/2_product/es_a24u-g2421.asp
Richard
Signature · 12 lines
Richard Ems mail: [email protected] Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs