Skip to contentSkip to conversation

Verify filesystem is aligned to stripes

DISCUSSION Spelic 5 participants 10 messages Nov 24, 2010 – Nov 26, 2010

lore
Share & tools
lore

Spelic

author
#0 lore raw

Hi there,
I thought there was a way to empirically check that the filesystem is correctly aligned to RAID stripes, but my attempts fail. I don't mean by looking at sunit and swidth from xfs_info, because that would not detect if there is some LVM offset problem.

I am particularly interested for parity RAIDs in MD.

I was thinking at "iostat -x 1": if writes are aligned I shouldn't see any reads from the drives in a parity RAID...

unfortunately this does not work:
- a dd streaming write test has almost no reads even when I mount with "noalign", with sufficiently large stripe_cache_size such as 1024. If it is smaller, always reads, even if xfs is aligned.
- a kernel untar will show lots of reads at any stripe_cache_size even if I'm pretty sure I aligned the stripes correctly on my 1024k x 15 data disks and the .tar.bz2 file was in cache. I tried with both xfs stripes autodetection in 2.6.37-rc2 and by specifying su and sw values by hand, which turned out to be the same; I was without LVM so I'm pretty sure alignment was correct. Why are there still lots of reads in this case?

so I'm pretty clueless. Anyone has good suggestions? Thank you

PS, OT: do you confirm it's not a good idea to have agsize multiple of stripe size like the mkfs warns you against? Today I offsetted it by +1 stripe unit (chunk) so that every AG begins on a different drive but performances didn't improve noticeably. Wouldn't that cause more unfilled stripes when writing?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 8 lines · Spelic

Hi there,
I thought there was a way to empirically check that the filesystem is correctly aligned to RAID stripes, but my attempts fail. I don't mean by looking at sunit and swidth from xfs_info, because that would not detect if there is some LVM offset problem.

I am particularly interested for parity RAIDs in MD.

Quoted 7 lines

I was thinking at "iostat -x 1": if writes are aligned I shouldn't see any reads from the drives in a parity RAID...

unfortunately this does not work:
- a dd streaming write test has almost no reads even when I mount with "noalign", with sufficiently large stripe_cache_size such as 1024. If it is smaller, always reads, even if xfs is aligned.

IO may not be aligned, though allocation usually is. With a large stripe cache, the MD device waits long enough for sequential unaligned IO to fill a full stripe width and hence never needs to read to calculate parity.

Quoted 7 lines

- a kernel untar will show lots of reads at any stripe_cache_size even if I'm pretty sure I aligned the stripes correctly on my 1024k x 15 data disks and the .tar.bz2 file was in cache. I tried with both xfs stripes autodetection in 2.6.37-rc2 and by specifying su and sw values by hand, which turned out to be the same; I was without LVM so I'm pretty sure alignment was correct. Why are there still lots of reads in this case?

Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.

Quoted 4 lines

PS, OT: do you confirm it's not a good idea to have agsize multiple of stripe size like the mkfs warns you against? Today I offsetted it by +1 stripe unit (chunk) so that every AG begins on a different drive but performances didn't improve noticeably.

Depends on the workload and a lot of other factors. in general, putting all the AG headers on the same spindle/lun results in that spindle/lun becoming a hotspot, especially when you have a filesystem with a few hundred AGs in it...

Quoted 2 lines

Wouldn't that
cause more unfilled stripes when writing?

Not for sequential IO (for the above reason), and for small IOs it will make zero difference.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw

Dave Chinner put forth on 11/24/2010 11:46 PM:

Quoted 5 lines

Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.

Do you really mean RAID1 here Dave, or RAID10? If RAID1, please elaborate a bit. RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.

Signature · 6 lines
Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 10 lines · Stan Hoeppner

Dave Chinner put forth on 11/24/2010 11:46 PM:

Quoted 5 lines

Because writes for workloads like this are never full stripe writes. Hence reads must be done to pullin the rest of the stripe before the new parity can be calculated. This RMW cycle for small IOs has always been the pain point for stripe based parity protection. If you are doing lots of small IOs, RAID1 is your friend.

Do you really mean RAID1 here Dave, or RAID10? If RAID1, please elaborate a bit.

RAID10 is just a convenient way of saying "striped mirrors" or "mirrored stripes". Fundamentally they are still using RAID1 for redundancy - a mirror of two devices. A device could be a single drive or a stripe of drives.

Quoted 2 lines

RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.

A good RAID1 implementation typically has the read performance of two devices (i.e. it can read from both legs simultaneously) and the write performance of a single device.

Parity based RAID is only fast for large write IOs or small IOs that are close enough together that a stripe cache can coalesce them into large writes. If this can't be acheived, parity based raid will be no faster than a _single drive_ for writes because all drives will be involved in RMW cycles. Indeed, I've seen RAID5 luns be saturated at only 50 iops because every IO required a RMW cycle, while an equivalent number of drives using RAID1 of RAID0 stripes did 1,000 iops...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw

Dave Chinner put forth on 11/25/2010 4:15 AM:

Quoted 8 lines · Stan Hoeppner
Quoted 3 lines

RAID1 traditionally has equal read performance to a single device, and half the write performance of a single device.

A good RAID1 implementation typically has the read performance of two devices (i.e. it can read from both legs simultaneously) and the write performance of a single device.

I've done no formal testing myself, but the last article I read that tested md RAID1 performance showed marginally faster read performance for a two disk mirror. IIRC, one test of 10 showed a 50% improvement. The rest showed less than 10% improvement, and some showed lower performance than a single drive. The performance would probably still be greater than the parity scenario you described with heavy RMW ops.

Quoted 8 lines

Parity based RAID is only fast for large write IOs or small IOs that are close enough together that a stripe cache can coalesce them into large writes. If this can't be acheived, parity based raid will be no faster than a _single drive_ for writes because all drives will be involved in RMW cycles. Indeed, I've seen RAID5 luns be saturated at only 50 iops because every IO required a RMW cycle, while an equivalent number of drives using RAID1 of RAID0 stripes did 1,000 iops...

This point brings up a question I've had for some time for which I've never found a thorough technical answer (maybe for lack of looking hard enough). And I'm painfully showing my lack of knowledge of how striping actually works, so please don't beat me up too much here. :)

Lets use an IMAP mail server in our example, configured to use maildir storage format. Most email messages are less than 4KB is size, and many are less than 512B--not even a full sector. Thus, the real size of each maildir file is going to be less than 4KB or 512B.

Let's say our array, either software or hardware based, contains 14x300GB SAS drives in RAID10. Let's say we've created the array with a (7x32KB) 224KB stripe size (though most hardware controllers would probably force us to choose between 128 or 256).

Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe? Does 220KB of the stripe merely get wasted? Will XFS pack this tiny file into the same extent with other small files, and then the extent gets written into the 128KB stripe?

So, for an array+filesystem that is going to overwhelming be storing lots of tiny files (mail), what array stripe size should one use, and what XFS parameters should the filesystem be created and mounted with to yield maximum random IOPs and minimum latency? Obviously these parameters may be different depending on RAID level chosen, so let's stick with this 14 disk RAID10 for our discussion.

Signature · 7 lines
Stan


_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

Le Thu, 25 Nov 2010 16:57:00 -0600 vous écriviez:

Quoted 4 lines

Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe?

It won't, it will simply end on one drive (actually one mirror). However because the mirrors are striped together, all drives in the array will be sollicited in my experience, that's why you need at least as many writing threads as there are stripes to reach the top IOPS. In your case, writing 56 4K files simultaneously will effectively write on all drives at once, hopefully (depends upon the filesystem allocation policy though).

Quoted 1 line

Does 220KB of the stripe merely get wasted?

It's not wasted, it just remains unallocated. What's wasted is potential IO performance.

What appears from the benchmarks I ran along the year is that anyway you turn it, whatever caching, command tag queuing and reordering your're using, a single thread can't reach maximal IOPS throughput on an array, i. e. writing on all drives simultaneously; a single thread writing to the fastest RAID 10 with 4K or 8K IOs can't do much better than with a single drive, 200 to 300 IOPS for a 15k drive.

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
Quoted 15 lines · Emmanuel Florac

Le Thu, 25 Nov 2010 16:57:00 -0600 vous écriviez:

Quoted 4 lines

Looking at the stripe size, which is equal to 64 sectors per array member drive (448 sectors total), how exactly is a sub 4KB mail file (8 sectors) going to be split up into equal chunks across a 224KB RAID stripe?

It won't, it will simply end on one drive (actually one mirror). However because the mirrors are striped together, all drives in the array will be sollicited in my experience, that's why you need at least as many writing threads as there are stripes to reach the top IOPS. In your case, writing 56 4K files simultaneously will effectively write on all drives at once, hopefully (depends upon the filesystem allocation policy though).

Quoted 4 lines
Quoted 1 line

Does 220KB of the stripe merely get wasted?

It's not wasted, it just remains unallocated. What's wasted is potential IO performance.

No, that's wrong. I don't have the time to explain the intricacies of how XFS packs small files together, but it does. You can observe the result by unpacking a kernel tarball and looking at the layout with xfs_bmap if you really want to...

FWIW, for workloads that do random, small IO, XFS works best when you _turn off_ aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).

Quoted 6 lines

What appears from the benchmarks I ran along the year is that anyway you turn it, whatever caching, command tag queuing and reordering your're using, a single thread can't reach maximal IOPS throughput on an array, i. e. writing on all drives simultaneously; a single thread writing to the fastest RAID 10 with 4K or 8K IOs can't do much better than with a single drive, 200 to 300 IOPS for a 15k drive.

Assuming synchronous IO. If you are doing async IO, a single CPU should be able to keep hundreds of SRDs (Spinning Rust Disks) busy...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#7 lore raw
Quoted 7 lines · Dave Chinner

FWIW, for workloads that do random, small IO, XFS works best when you _turn off_ aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).

I think I also noticed this...

The thing is, for large sequential I/O it seems to me it's indifferent if XFS is aligned or not, because the resulting file will anyway be sequential, and if you have raid10 or even parity raid with a large stripe cache there won't be any reads anyway. Ok maybe with alignment you could avoid reads on the first stripe (not sure, it might read anyway if the RAID reacts fast, before enough output is sent to it), but that's the only one.

So when is alignment to be turned on?

Thanks for the info

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
In reply to Dave Chinner
Quoted 5 lines · On Freitag, 26. November 2010 Dave Chinner wrote:

FWIW, for workloads that do random, small IO, XFS works best when you turn off aligned allocation and just let it spray the IO at the disks. This works best if you are using RAID 0/1/10. All the numbers I've been posting are with aligned allocation turned off (i.e. no sunit/swidth set).

That's interesting to read.

Why would sunit/swidth be slower then? I'd thought that XFS then would know one stripe is 64k and I have 8 disks so it should try to pack 8*64=512kb in one junk on disk, and that especially for small files it would write them like that.

The man page just says inodes, log are stripe aligned, and file tails

Quoted 1 line

512k extended to full stripes on append. I thought that even the

inode/log alignment alone would help a lot.

Now what is the advantage on skipping sunit/swidth altogether? And what is the difference when it's on RAID10 to RAID6?

I'm always eager to understand performance issues ;-)

Signature · 11 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// ****** Radiointerview zum Thema Spam ******
// http://www.it-podcast.at/archiv.html#podcast-100716
// 
// Haus zu verkaufen: http://zmi.at/langegg/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
In reply to Dave Chinner

Le Fri, 26 Nov 2010 23:22:18 +1100
Dave Chinner <[email protected]> écrivait:

Quoted 3 lines

Assuming synchronous IO. If you are doing async IO, a single CPU should be able to keep hundreds of SRDs (Spinning Rust Disks) busy...

What I had in mind (typical random IO) is DB access, which is AFAIK synchronous IO, my bad :)

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs