Verify filesystem is aligned to stripes
Hi there,
I thought there was a way to empirically check that the filesystem is correctly aligned to RAID stripes, but my attempts fail. I don't mean by looking at sunit and swidth from xfs_info, because that would not detect if there is some LVM offset problem.
I am particularly interested for parity RAIDs in MD.
I was thinking at "iostat -x 1": if writes are aligned I shouldn't see any reads from the drives in a parity RAID...
unfortunately this does not work:
- a dd streaming write test has almost no reads even when I mount with "noalign", with sufficiently large stripe_cache_size such as 1024. If it is smaller, always reads, even if xfs is aligned.
- a kernel untar will show lots of reads at any stripe_cache_size even if I'm pretty sure I aligned the stripes correctly on my 1024k x 15 data disks and the .tar.bz2 file was in cache. I tried with both xfs stripes autodetection in 2.6.37-rc2 and by specifying su and sw values by hand, which turned out to be the same; I was without LVM so I'm pretty sure alignment was correct. Why are there still lots of reads in this case?
so I'm pretty clueless. Anyone has good suggestions? Thank you
PS, OT: do you confirm it's not a good idea to have agsize multiple of stripe size like the mkfs warns you against? Today I offsetted it by +1 stripe unit (chunk) so that every AG begins on a different drive but performances didn't improve noticeably. Wouldn't that cause more unfilled stripes when writing?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Spelic
Hi there,
I thought there was a way to empirically check that the filesystem is correctly aligned to RAID stripes, but my attempts fail. I don't mean by looking at sunit and swidth fromxfs_info, because that would not detect if there is some LVM offset problem.I am particularly interested for parity RAIDs in MD.
Quoted 7 lines
I was thinking at "iostat -x 1": if writes are aligned I shouldn't see any reads from the drives in a parity RAID...
unfortunately this does not work:
- a dd streaming write test has almost no reads even when I mount with "noalign", with sufficiently largestripe_cache_sizesuch as 1024. If it is smaller, always reads, even if xfs is aligned.
IO may not be aligned, though allocation usually is. With a large stripe cache, the MD device waits long enough for sequential unaligned IO to fill a full stripe width and hence never needs to read to calculate parity.
Quoted 7 lines
- a kernel untar will show lots of reads at any
stripe_cache_sizeeven if I'm pretty sure I aligned the stripes correctly on my 1024k x 15 data disks and the .tar.bz2 file was in cache. I tried with both xfs stripes autodetection in 2.6.37-rc2 and by specifying su and sw values by hand, which turned out to be the same; I was without LVM so I'm pretty sure alignment was correct. Why are there still lots of reads in this case?
Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.
Quoted 4 lines
PS, OT: do you confirm it's not a good idea to have agsize multiple of stripe size like the mkfs warns you against? Today I offsetted it by +1 stripe unit (chunk) so that every AG begins on a different drive but performances didn't improve noticeably.
Depends on the workload and a lot of other factors. in general, putting all the AG headers on the same spindle/lun results in that spindle/lun becoming a hotspot, especially when you have a filesystem with a few hundred AGs in it...
Quoted 2 lines
Wouldn't that
cause more unfilled stripes when writing?
Not for sequential IO (for the above reason), and for small IOs it will make zero difference.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner put forth on 11/24/2010 11:46 PM:
Quoted 5 lines
Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.
Do you really mean RAID1 here Dave, or RAID10? If RAID1, please elaborate a bit. RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Stan Hoeppner
Dave Chinner put forth on 11/24/2010 11:46 PM:
Quoted 5 lines
Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.
Do you really mean RAID1 here Dave, or RAID10? If RAID1, please elaborate a bit.
RAID10 is just a convenient way of saying "striped mirrors" or "mirrored stripes". Fundamentally they are still using RAID1 for redundancy - a mirror of two devices. A device could be a single drive or a stripe of drives.
Quoted 2 lines
RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.
A good RAID1 implementation typically has the read performance of two devices (i.e. it can read from both legs simultaneously) and the write performance of a single device.
Parity based RAID is only fast for large write IOs or small IOs that are close enough together that a stripe cache can coalesce them into large writes. If this can't be acheived, parity based raid will be no faster than a _single drive_ for writes because all drives will be involved in RMW cycles. Indeed, I've seen RAID5 luns be saturated at only 50 iops because every IO required a RMW cycle, while an equivalent number of drives using RAID1 of RAID0 stripes did 1,000 iops...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner put forth on 11/25/2010 4:15 AM:
Quoted 8 lines · Stan Hoeppner
Quoted 3 lines
RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.
A good RAID1 implementation typically has the read performance of two devices (i.e. it can read from both legs simultaneously) and the write performance of a single device.
I've done no formal testing myself, but the last article I read that tested md RAID1 performance showed marginally faster read performance for a two disk mirror. IIRC, one test of 10 showed a 50% improvement. The rest showed less than 10% improvement, and some showed lower performance than a single drive. The performance would probably still be greater than the parity scenario you described with heavy RMW ops.
Quoted 8 lines
Parity based RAID is only fast for large write IOs or small IOs that are close enough together that a stripe cache can coalesce them into large writes. If this can't be acheived, parity based raid will be no faster than a
_singledrive_for writes because all drives will be involved in RMW cycles. Indeed, I've seen RAID5 luns be saturated at only 50 iops because every IO required a RMW cycle, while an equivalent number of drives using RAID1 of RAID0 stripes did 1,000 iops...
This point brings up a question I've had for some time for which I've never found a thorough technical answer (maybe for lack of looking hard enough). And I'm painfully showing my lack of knowledge of how striping actually works, so please don't beat me up too much here. :)
Lets use an IMAP mail server in our example, configured to use maildir storage format. Most email messages are less than 4KB is size, and many are less than 512B--not even a full sector. Thus, the real size of each maildir file is going to be less than 4KB or 512B.
Let's say our array, either software or hardware based, contains 14x300GB SAS drives in RAID10. Let's say we've created the array with a (7x32KB) 224KB stripe size (though most hardware controllers would probably force us to choose between 128 or 256).
Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe? Does 220KB of the stripe merely get wasted? Will XFS pack this tiny file into the same extent with other small files, and then the extent gets written into the 128KB stripe?
So, for an array+filesystem that is going to overwhelming be storing lots of tiny files (mail), what array stripe size should one use, and what XFS parameters should the filesystem be created and mounted with to yield maximum random IOPs and minimum latency? Obviously these parameters may be different depending on RAID level chosen, so let's stick with this 14 disk RAID10 for our discussion.
Signature · 7 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Thu, 25 Nov 2010 16:57:00 -0600 vous écriviez:
Quoted 4 lines
Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe?
It won't, it will simply end on one drive (actually one mirror). However because the mirrors are striped together, all drives in the array will be sollicited in my experience, that's why you need at least as many writing threads as there are stripes to reach the top IOPS. In your case, writing 56 4K files simultaneously will effectively write on all drives at once, hopefully (depends upon the filesystem allocation policy though).
Quoted 1 line
Does 220KB of the stripe merely get wasted?
It's not wasted, it just remains unallocated. What's wasted is potential IO performance.
What appears from the benchmarks I ran along the year is that anyway you turn it, whatever caching, command tag queuing and reordering your're using, a single thread can't reach maximal IOPS throughput on an array, i. e. writing on all drives simultaneously; a single thread writing to the fastest RAID 10 with 4K or 8K IOs can't do much better than with a single drive, 200 to 300 IOPS for a 15k drive.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 15 lines · Emmanuel Florac
Le Thu, 25 Nov 2010 16:57:00 -0600 vous écriviez:
Quoted 4 lines
Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe?
It won't, it will simply end on one drive (actually one mirror). However because the mirrors are striped together, all drives in the array will be sollicited in my experience, that's why you need at least as many writing threads as there are stripes to reach the top IOPS. In your case, writing 56 4K files simultaneously will effectively write on all drives at once, hopefully (depends upon the filesystem allocation policy though).
Quoted 4 lines
Quoted 1 line
Does 220KB of the stripe merely get wasted?
It's not wasted, it just remains unallocated. What's wasted is potential IO performance.
No, that's wrong. I don't have the time to explain the intricacies of how XFS packs small files together, but it does. You can observe the result by unpacking a kernel tarball and looking at the layout with xfs_bmap if you really want to...
FWIW, for workloads that do random, small IO, XFS works best when you _turn off_ aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).
Quoted 6 lines
What appears from the benchmarks I ran along the year is that anyway you turn it, whatever caching, command tag queuing and reordering your're using, a single thread can't reach maximal IOPS throughput on an array, i. e. writing on all drives simultaneously; a single thread writing to the fastest RAID 10 with 4K or 8K IOs can't do much better than with a single drive, 200 to 300 IOPS for a 15k drive.
Assuming synchronous IO. If you are doing async IO, a single CPU should be able to keep hundreds of SRDs (Spinning Rust Disks) busy...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Dave Chinner
FWIW, for workloads that do random, small IO, XFS works best when you
_turnoff_aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).
I think I also noticed this...
The thing is, for large sequential I/O it seems to me it's indifferent if XFS is aligned or not, because the resulting file will anyway be sequential, and if you have raid10 or even parity raid with a large stripe cache there won't be any reads anyway. Ok maybe with alignment you could avoid reads on the first stripe (not sure, it might read anyway if the RAID reacts fast, before enough output is sent to it), but that's the only one.
So when is alignment to be turned on?
Thanks for the info
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · On Freitag, 26. November 2010 Dave Chinner wrote:
FWIW, for workloads that do random, small IO, XFS works best when you turn off aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).
That's interesting to read.
Why would sunit/swidth be slower then? I'd thought that XFS then would know one stripe is 64k and I have 8 disks so it should try to pack 8*64=512kb in one junk on disk, and that especially for small files it would write them like that.
The man page just says inodes, log are stripe aligned, and file tails
Quoted 1 line
512k extended to full stripes on append. I thought that even the
inode/log alignment alone would help a lot.
Now what is the advantage on skipping sunit/swidth altogether? And what is the difference when it's on RAID10 to RAID6?
I'm always eager to understand performance issues ;-)
Signature · 11 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services: Protéger http://proteger.at [gesprochen: Prot-e-schee] Tel: +43 660 / 415 6531 // ****** Radiointerview zum Thema Spam ****** // http://www.it-podcast.at/archiv.html#podcast-100716 // // Haus zu verkaufen: http://zmi.at/langegg/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Fri, 26 Nov 2010 23:22:18 +1100
Dave Chinner <[email protected]> écrivait:
Quoted 3 lines
Assuming synchronous IO. If you are doing async IO, a single CPU should be able to keep hundreds of SRDs (Spinning Rust Disks) busy...
What I had in mind (typical random IO) is DB access, which is AFAIK synchronous IO, my bad :)
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs