30 TB RAID6 + XFS slow write performance

Dear list members,

A customer of mine is currently struggling with the performance of a 30 TB RAID6 which uses XFS as the filing system. I am somewhat sure it's not XFS that's causing the performance issue but my expertise is not XFS nor RAID; I just wrote the software that after moving to the larger RAID (from a much smaller one, ~ 3TB, using ext3) suddenly seems to have a huge drop in write performance.

The software I wrote writes many small (50-150K) files in parallel (100+ processes), thousands of times per hour. Writing a file of 50-150K now and then seems to take between 30 and 90 seconds, and more rarely can take over 200 seconds (several times an hour).

When all processes are stopped and restarted again the 30-90 seconds delay start happening when about 16-20+ processes are running.

To me this sounds like something has been configured wrong. I already recommended my customer to find someone who is capable of configuring the RAID correctly; to me it sounds like a hardware/configuration issue.

Any insights are very welcome.

Hardware: card: MegaRAID SAS 9260-16i disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6 ~ 30TB

Thanks for reading, John

_______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

Le Mon, 18 Jul 2011 14:58:55 -0500 vous écriviez:

Quoted 4 lines

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

This card doesn't activate the write cache without a BBU present. Be sure you have a BBU or the performance will always be unbearably awful.

Quoted 9 lines · Emmanuel Florac

Le Mon, 18 Jul 2011 14:58:55 -0500 vous écriviez:

Quoted 4 lines

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

This card doesn't activate the write cache without a BBU present. Be sure you have a BBU or the performance will always be unbearably awful.

In addition to all the other recommendations, once the BBU is installed, disable the individual drive caches (if this isn't done automatically), and set the controller cache mode to 'write back'. The write through and direct I/O cache modes will deliver horrible RAID6 write performance.

And, BTW, RAID6 is a horrible choice for a parallel, small file, high random I/O workload such as you've described. RAID10 would be much more suitable. Actually, any striped RAID is less than optimal for such a small file workload. The default stripe size for the LSI RAID controllers, IIRC, is 64KB. With 14 spindles of stripe width you end up with 64*14 = 896KB.

XFS will try to pack as many of these 50-150K files into a single extent, but you're talking 6 to 18 files per extent,

and
this is wholly dependent on the parallel write pattern, and in which of the allocation groups XFS decides to write each file.

XFS isn't going
to be 100% efficient in this case. Thus, you will end up with many partial stripe width writes, eliminating much of the performance advantage of striping.

These are large 7200 rpm SATA drives which have poor seek performance to begin with, unlike the 'small' 300GB 15k SAS drives. You're robbing that poor seek performance further by:

1. Using double parity striped RAID
2. Writing thousands of small files in parallel

This workload is very similar to the case of a mail server using the maildir storage format.

If you read the list archives you'll see
recommendations for an optimal storage stack setup for this workload. It goes something like this:

1. Create a linear array of hardware RAID1 mirror sets.
Do this all in the controller if it can do it.
If not, use Linux RAID (mdadm) to create a '--linear' array of the multiple (7 in your case, apparently) hardware RAID1 mirror sets

2. Now let XFS handle the write parallelism. Format the resulting 7 spindle Linux RAID device with, for example:

mkfs.xfs -d agcount=14 /dev/md0

By using this configuration you eliminate the excessive head seeking associated with the partial stripe write problems of RAID6, restoring performance efficiency to the array. Using 14 allocation groups allows XFS to write write, at minimum, 14 such files in parallel.

Quoted 16 lines · Stan Hoeppner

Quoted 9 lines · Emmanuel Florac

Le Mon, 18 Jul 2011 14:58:55 -0500 vous écriviez:

Quoted 4 lines

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

This card doesn't activate the write cache without a BBU present. Be sure you have a BBU or the performance will always be unbearably awful.

In addition to all the other recommendations, once the BBU is installed, disable the individual drive caches (if this isn't done automatically), and set the controller cache mode to 'write back'. The write through and direct I/O cache modes will deliver horrible RAID6 write performance.

Quoted 6 lines

And, BTW, RAID6 is a horrible choice for a parallel, small file, high random I/O workload such as you've described. RAID10 would be much more suitable. Actually, any striped RAID is less than optimal for such a small file workload. The default stripe size for the LSI RAID controllers, IIRC, is 64KB. With 14 spindles of stripe width you end up with 64*14 = 896KB.

All good up to here.

Quoted 2 lines

XFS will try to pack as many of these 50-150K files into a single extent, but you're talking 6 to 18 files per extent,

I think you've got your terminology wrong. An extent can only belong to a single inode, but an inode can contain many extents, as can a stripe width. We do not pack data from multiple files into a single extent.

For new files on a su/sw aware filesystem, however, XFS will *not* pack multiple files into the same stripe unit. It will try to align the first extent of the file to sunit, or if you have the swalloc mount option set and the allocation is for more than a swidth of space it will align to swidth rather than sunit.

So if you have a small file workload, specifying sunit/swidth can actually -decrease- performance because it allocates the file extents sparsely. IOWs, stripe alignment is important for bandwidth intensive applications because it allows full stripe writes to occur much more frequently, but can be harmful to small file performance as the aligned allocation pattern can prevent full stripe writes from occurring.....

Quoted 3 lines

and
this is wholly dependent on the parallel write pattern, and in which of the allocation groups XFS decides to write each file.

That's pretty much irrelevant for small files as a single allocation is done for each file during writeback.

Quoted 4 lines

XFS isn't going
to be 100% efficient in this case. Thus, you will end up with many partial stripe width writes, eliminating much of the performance advantage of striping.

Yes, that's the ultimate problem, but not for the reasons you suggested. ;)

Quoted 6 lines

These are large 7200 rpm SATA drives which have poor seek performance to begin with, unlike the 'small' 300GB 15k SAS drives. You're robbing that poor seek performance further by:

1. Using double parity striped RAID
2. Writing thousands of small files in parallel

The writing in parallel is only an issue if it is direct or synchronous IO. If it's using normal buffered writes, then writeback is mostly single threaded and delayed allocation should be preventing fragmentation completely. That still doesn't guarantee that writeback avoids RAID RMW cycles (see above about allocation alignment).

Quoted 2 lines

This workload is very similar to the case of a mail server using the maildir storage format.

There's not enough detail in the workload description to make that assumption.

Quoted 12 lines

If you read the list archives you'll see
recommendations for an optimal storage stack setup for this workload. It goes something like this:

1. Create a linear array of hardware RAID1 mirror sets.
Do this all in the controller if it can do it.
If not, use Linux RAID (mdadm) to create a '--linear' array of the multiple (7 in your case, apparently) hardware RAID1 mirror sets

2. Now let XFS handle the write parallelism. Format the resulting 7 spindle Linux RAID device with, for example:

Quoted 6 lines

mkfs.xfs -d agcount=14 /dev/md0

By using this configuration you eliminate the excessive head seeking associated with the partial stripe write problems of RAID6, restoring performance efficiency to the array. Using 14 allocation groups allows XFS to write write, at minimum, 14 such files in parallel.

That's not correct. 14 AG means that if the files are laid out across all AGs then there can be 14 -allocations- in parallel at once. If Io does not require allocation, then they don't serialise at all on the AGs. IOWs, If allocation takes 1ms of work in an AG, then you could have 1,000 allocations per second per AG. With 14 AGs, that gives allocation capability of up to 14,000/s

And given that not all writes require allocation and allocation is usually only a small percentage of the total IO time. You can have many, many more write IOs in flight than you can do allocations in an AG....

Quoted 24 lines · Dave Chinner

Quoted 22 lines · Stan Hoeppner

Quoted 9 lines · Emmanuel Florac

Le Mon, 18 Jul 2011 14:58:55 -0500 vous écriviez:

Quoted 4 lines

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

This card doesn't activate the write cache without a BBU present. Be sure you have a BBU or the performance will always be unbearably awful.

In addition to all the other recommendations, once the BBU is installed, disable the individual drive caches (if this isn't done automatically), and set the controller cache mode to 'write back'. The write through and direct I/O cache modes will deliver horrible RAID6 write performance.

And, BTW, RAID6 is a horrible choice for a parallel, small file, high random I/O workload such as you've described. RAID10 would be much more suitable. Actually, any striped RAID is less than optimal for such a small file workload. The default stripe size for the LSI RAID controllers, IIRC, is 64KB. With 14 spindles of stripe width you end up with 64*14 = 896KB.

Quoted 1 line

All good up to here.

And then my lack of understanding of XFS internals begins to show. :(

Quoted 7 lines

So if you have a small file workload, specifying sunit/swidth can actually -decrease- performance because it allocates the file extents sparsely. IOWs, stripe alignment is important for bandwidth intensive applications because it allows full stripe writes to occur much more frequently, but can be harmful to small file performance as the aligned allocation pattern can prevent full stripe writes from occurring.....

I don't recall reading this before Dave. Thank you for this tidbit.

How much performance decrease are we looking at here?

An mkfs.xfs of an
mdraid striped array will by default create sunit/swidth values right? And thus this lower performance w/small files.

Quoted 19 lines

Quoted 12 lines

If you read the list archives you'll see
recommendations for an optimal storage stack setup for this workload. It goes something like this:

1. Create a linear array of hardware RAID1 mirror sets.
Do this all in the controller if it can do it.
If not, use Linux RAID (mdadm) to create a '--linear' array of the multiple (7 in your case, apparently) hardware RAID1 mirror sets

2. Now let XFS handle the write parallelism. Format the resulting 7 spindle Linux RAID device with, for example:

Quoted 6 lines

mkfs.xfs -d agcount=14 /dev/md0

By using this configuration you eliminate the excessive head seeking associated with the partial stripe write problems of RAID6, restoring performance efficiency to the array. Using 14 allocation groups allows XFS to write write, at minimum, 14 such files in parallel.

Quoted 6 lines

That's not correct. 14 AG means that if the files are laid out across all AGs then there can be 14 -allocations- in parallel at once. If Io does not require allocation, then they don't serialise at all on the AGs. IOWs, If allocation takes 1ms of work in an AG, then you could have 1,000 allocations per second per AG. With 14 AGs, that gives allocation capability of up to 14,000/s

So are you saying that we have no guarantee, nor high probability, that the small files in this case will be spread out across all AGs, thus making more efficient use of each disk's performance in the concatenated array, vs a striped array? Or, are you merely pointing out a detail I have incorrect, which I've yet to fully understand?

Quoted 4 lines

And given that not all writes require allocation and allocation is usually only a small percentage of the total IO time. You can have many, many more write IOs in flight than you can do allocations in an AG....

Ahh, I think I see your point. For the maildir case, more of the IO is likely due to things like updating message flags, etc, than actually writing new mail files into the directory.

Such operations don't
require allocation. With the workload mentioned by the OP, it's possible that all of the small file writes may indeed require allocation, unlike the maildir workload. But if this is the case, wouldn't the concatenated array still yield better overall performance than RAID6, or any other striped array?

Quoted 25 lines · Stan Hoeppner

Quoted 24 lines · Dave Chinner

Quoted 22 lines · Stan Hoeppner

Quoted 9 lines · Emmanuel Florac

Le Mon, 18 Jul 2011 14:58:55 -0500 vous écriviez:

Quoted 4 lines

card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares). RAID6
~ 30TB

This card doesn't activate the write cache without a BBU present. Be sure you have a BBU or the performance will always be unbearably awful.

In addition to all the other recommendations, once the BBU is installed, disable the individual drive caches (if this isn't done automatically), and set the controller cache mode to 'write back'. The write through and direct I/O cache modes will deliver horrible RAID6 write performance.

And, BTW, RAID6 is a horrible choice for a parallel, small file, high random I/O workload such as you've described. RAID10 would be much more suitable. Actually, any striped RAID is less than optimal for such a small file workload. The default stripe size for the LSI RAID controllers, IIRC, is 64KB. With 14 spindles of stripe width you end up with 64*14 = 896KB.

Quoted 3 lines

Quoted 1 line

All good up to here.

And then my lack of understanding of XFS internals begins to show. :(

The fact you are trying to understand them is the important bit!

....

Quoted 8 lines

Quoted 7 lines

So if you have a small file workload, specifying sunit/swidth can actually -decrease- performance because it allocates the file extents sparsely. IOWs, stripe alignment is important for bandwidth intensive applications because it allows full stripe writes to occur much more frequently, but can be harmful to small file performance as the aligned allocation pattern can prevent full stripe writes from occurring.....

Quoted 1 line

I don't recall reading this before Dave. Thank you for this tidbit.

I'm sure I've said this before, but it's possible I've said it this time in away that is obvious and understandable. Most people struggle with the concept of allocation alignment and why it might be important, let alone understand it well enough to discuss intricate details of the allocator and tuning it for different workloads...

Quoted 3 lines

An mkfs.xfs of an
mdraid striped array will by default create sunit/swidth values right? And thus this lower performance w/small files.

In general, sunit/swidth being specified provides a better tradeoff for maintaining consistent performance on files across the filesystem. it might cost a little for small files, but unaligned IO on large files cause much more noticable performace problems...

....

Quoted 26 lines

Quoted 19 lines

Quoted 12 lines

If you read the list archives you'll see
recommendations for an optimal storage stack setup for this workload. It goes something like this:

1. Create a linear array of hardware RAID1 mirror sets.
Do this all in the controller if it can do it.
If not, use Linux RAID (mdadm) to create a '--linear' array of the multiple (7 in your case, apparently) hardware RAID1 mirror sets

2. Now let XFS handle the write parallelism. Format the resulting 7 spindle Linux RAID device with, for example:

Quoted 6 lines

mkfs.xfs -d agcount=14 /dev/md0

By using this configuration you eliminate the excessive head seeking associated with the partial stripe write problems of RAID6, restoring performance efficiency to the array. Using 14 allocation groups allows XFS to write write, at minimum, 14 such files in parallel.

Quoted 6 lines

That's not correct. 14 AG means that if the files are laid out across all AGs then there can be 14 -allocations- in parallel at once. If Io does not require allocation, then they don't serialise at all on the AGs. IOWs, If allocation takes 1ms of work in an AG, then you could have 1,000 allocations per second per AG. With 14 AGs, that gives allocation capability of up to 14,000/s

Quoted 5 lines

So are you saying that we have no guarantee, nor high probability, that the small files in this case will be spread out across all AGs, thus making more efficient use of each disk's performance in the concatenated array, vs a striped array? Or, are you merely pointing out a detail I have incorrect, which I've yet to fully understand?

Yet to fully understand. It's not limited to small files, either.

XFS doesn't guarantee that specific allocations are evenly distributed across AGs, but it does try to spread the overall contents of the filesystem across all AGs. It does have concepts of locality of reference, but they change depending on the allocator in use.

Take, for example, inode32 vs inode64 which are the two most common allocation strategies and assume we have a 16TB fs with 1TB AGs. The inode32 allocator will place all inodes and most directory metadata in the first AG, below one TB. There is basically no metadata allocation parallelism in this strategy, so metadata performance is limited and will often serialise. Metadata tends to have good locality of reference - all directories and inodes will tend to be close together on disk because they are in the same AG.

Data, on the other had is rotored around AGs 2-16 on a per file basis, so there is no locality between inodes and their data, nor of data between two adjacent files in the same directory. There is, however, data allocation parallelism because files are spread across allocation groups...

Hence for inode32, metadata is closely located, but data is spread out widely. Hence metadata operations don't scale at all well on a linear concat (e.g. hit only one disk/mirror pair), but data allocations are spread effectively and hence parallelise and scale quite well. The downside to this is that data lookups involve large seeks if you have a stripe, and hence can be quite slow. Data reads on a linear concat are not guaranteed to evenly load the disks, either, simply because there's no correlation between the location of the data and the access patterns.

For inode64, locality of reference clusters around the directory structure. The inodes for files in a directory will be allocated in the same AG as the directory inode, and the data for each file will be allocated in the same AG as the file inodes. When you create a new directory, it gets placed in a different AG, and the pattern repeats. So for inode64, distributing files across all AGs is caused by distributing the directory structure.

FWIW, an example is a
kernel source tree:

~/src/kern/xfsdev$ find . -type d -exec sudo xfs_bmap -v {} \; | awk '/ 0: / { print $4 }' |sort -n |uniq -c

     76 0
     66 1
     85 2
     81 3
     82 4
     69 5
     89 6
     74 7
     90 8
     81 9
     96 10
     84 11
     85 12
     84 13
     86 14
     71 15

As you can see, there's a relatively even spread of the directories across all 16 AGs in that directory structure, and the file data will follow this pattern. Because of it's better metadata<->data locality of reference, inode64 tends to be signficantly faster on workloads that mix metadata operations with data operations (e.g. recursive grep across a kernel source tree) as the seek cost between the inode and it's data is much less than for inode32....

However, if youre workload does not spread across directories, then IO will tend to be limited to specific silos in the linear concat while other disks sit idle. If you have a stripe, then the seeks to get to the data are small, and hence much faster than inode32 on similar workloads.

This is all ignoring stripe aligned allocation - that is often lost in the noise comapred to bigger issues like seeking from AG 0 to AG 15 when reading the inode then the data or having a workload only use a single AG because it is all confined to a single directory.

IOWs, the best, most optimal filesystem layout and allocation stratgey is both workload and hardware dependent, and there's no one right answer. The defaults select the best balance for typical usage
- beyond that benchmarking the workload is the only way to really measure whether your tweaks are the right ones or not. IOWs, you need to understand the filesystem, your storage hardware and -the application IO patterns- to make the right tuning decisions.

Quoted 5 lines

Quoted 4 lines

And given that not all writes require allocation and allocation is usually only a small percentage of the total IO time. You can have many, many more write IOs in flight than you can do allocations in an AG....

Quoted 3 lines

Ahh, I think I see your point. For the maildir case, more of the IO is likely due to things like updating message flags, etc, than actually writing new mail files into the directory.

I wasn't really talking about maildir here, just pointing out that allocation is generally not the limiting factor in doing large amounts of concurrent write IO.

Quoted 6 lines

Such operations don't
require allocation. With the workload mentioned by the OP, it's possible that all of the small file writes may indeed require allocation, unlike the maildir workload. But if this is the case, wouldn't the concatenated array still yield better overall performance than RAID6, or any other striped array?

<shrug>

Quite possibly, butI can't say conclusively - I simply don't know enough about the workload or the fs configuration.

I thought this was packing multiple small files into
a single stripe write, which you just explained XFS does not do.

Quoted 2 lines · On Mittwoch, 20. Juli 2011 Stan Hoeppner wrote:

I thought this was packing multiple small files into
a single stripe write, which you just explained XFS does not do.

This is interesting, I jump in here. Does that mean that if I have a XFS volume with sw=14,su=64k (14*64=896KiB) that when I write 10 small files in the same dir with 2KB each, each file would be placed at a 896KiB boundary?

That way, all stripes of a 1GB partition would be full when there are roughly 1170 files (1170*896KiB ~ 1GB). What would happen when I create other files - is XFS "full" then, or would it start using sub- stripes? If sub-stripes, would they start at su (=64KiB) distances, or at single block (e.g. 4KiB) distances?

No, they'll get sunit aligned but default, which would be on 64k boundaries.

Quoted 5 lines

That way, all stripes of a 1GB partition would be full when there are roughly 1170 files (1170*896KiB ~ 1GB). What would happen when I create other files - is XFS "full" then, or would it start using sub- stripes? If sub-stripes, would they start at su (=64KiB) distances, or at single block (e.g. 4KiB) distances?

It starts packing files tightly into remaining free space when no free aligned extents are availble for allocation in the AG.

Quoted 2 lines · On Donnerstag, 21. Juli 2011 Dave Chinner wrote:

No, they'll get sunit aligned but default, which would be on 64k boundaries.

OK, so only when <quote Dave> "swalloc mount option set and the allocation is for more than a swidth of space it will align to swidth rather than sunit" </quote Dave>.

So even when I specify swalloc but a file is generated with only 4KB, it will very probably be sunit aligned on disk.

Quoted 6 lines

Quoted 5 lines

That way, all stripes of a 1GB partition would be full when there are roughly 1170 files (1170*896KiB ~ 1GB). What would happen when I create other files - is XFS "full" then, or would it start using sub- stripes? If sub-stripes, would they start at su (=64KiB) distances, or at single block (e.g. 4KiB) distances?

Quoted 2 lines

It starts packing files tightly into remaining free space when no free aligned extents are availble for allocation in the AG.

That means for above example, that 16384 x 2KiB files could be created, and each be sunit aligned on disk. Then all sunit start blocks are full, so additional files will be sub-sunit "packed", is it this?

That would mean fragmentation is likely to occur from that moment, if there are files that grow.

And files >64KiB are immediately fragmented
then. At this time, there are only 16384 * 2KiB = 32MiB used, which is 3,125% of the disk. I can't believe my numbers, are they true?

OK, this is a worst case scenario, and as you've said before, any filesystem can be considered full at 85% fill grade. But it's incredible how quickly you could fuck up a filesystem when using su/sw and writing small files.

If you are writing files that grow like this, then you are doing something wrong. If the app can't do it's IO differently, then this is exactly the reason we have userspace-controlled preallocation interfaces.

Filesystems cannot prevent user stupidity from screwing something up....

Quoted 3 lines

And files >64KiB are immediately fragmented
then. At this time, there are only 16384 * 2KiB = 32MiB used, which is 3,125% of the disk. I can't believe my numbers, are they true?

No, because most filesystems have a 4k block size.

Not to mention
that fragmentation is likely to be limited to the single AG the files in the directory belong to. i.e. even if we can't allocation a sunit aligned chunk in an AG, we won't switch to another AG just to do sunit aligned allocation.

Quoted 4 lines

OK, this is a worst case scenario, and as you've said before, any filesystem can be considered full at 85% fill grade. But it's incredible how quickly you could fuck up a filesystem when using su/sw and writing small files.

Well, don't use a filesystem that is optimised for storing large sizes, large files and high bandwidth for storing lots of small files, then. Indeed, the point of not packing the files is so they -don't fragemnt as they grow-. XFS is not designed to be optimal for small filesystems or small files. In most cases it will deal with them just fine, so in reality your concerns are mostly unfounded...

Yes, I just wanted to know about the corner cases, and how XFS behaves. Actually, we're changing over to using NetApps, and with their WAFL anyway I should drop all su/sw usage and just use 4KB blocks.

Quoted 3 lines · Michael Monnerie

Yes, I just wanted to know about the corner cases, and how XFS behaves. Actually, we're changing over to using NetApps, and with their WAFL anyway I should drop all su/sw usage and just use 4KB blocks.

I've never used a NetApp filer myself. However, that said, I would assume that WAFL is only in play for NFS/CIFS transactions since WAFL is itself a filesystem.

When exposing LUNs from the same filer to FC and iSCSI hosts I would assume the filer acts just as any other SAN controller would.

In this case I would think you'd probably still want to align your XFS filesystem to the underlying RAID stripe from which the LUN was carved.

Quoted 3 lines · Stan Hoeppner

I've never used a NetApp filer myself. However, that said, I would assume that WAFL is only in play for NFS/CIFS transactions since WAFL is itself a filesystem.

Netapp's website is busted, so here's a cached link:

http://webcache.googleusercontent.com/search?q=cache:9DdO2a16hdIJ:blogs.netapp.com/extensible_netapp/2008/10/what-is-wafl--3.html+netapp+san+wafl&cd=1&hl=en&ct=clnk&source=www.google.com

If you can be bothered trolling for that entire series of blog posts in the google cache, it's probably a good idea so you can get a basic understanding of what WAFL actually is.

Quoted 2 lines

When exposing LUNs from the same filer to FC and iSCSI hosts I would assume the filer acts just as any other SAN controller would.

It has it's own quirks, just like any other FC attached RAID array...

Quoted 3 lines

In this case I would think you'd probably still want to align your XFS filesystem to the underlying RAID stripe from which the LUN was carved.

Which actually matters very little when WAFL between the FS and the disk because WAFL uses copy-on-write and stages all it's writes through NVRAM and so you've got no idea what the alignment of any given address in the filesystem maps to, anyway.

Is the NetApp FC/iSCSI attachment performance still competitive for large file/streaming IO, given that one can't optimize XFS stripe alignment, and with no indication of where the file fragments are actually written on the media? Or does it lag behind something like a roughly equivalent class Infinite Storage array, or IBM DS?