ag selection
Hi all,
for streaming writes onto a raid6 the current round-robin ag selection seems does not seem to be optimal. Writing 4 files from 4 threads into a single directory we get 900 MB/s, writing 4 files in 4 different directories we only get 700 MB/s (12 disks with with hw megaraid-sas). The current round-robin scheme seems to be optimized for linear raid0? With small AGs one could also argue, that choosing AGs which are not far away from each other (in respect to the number of blocks) also adds more parallel disk access for small and medium sized files.
Any objections against a patch to improve the AG selection?
Thanks,
Bernd
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Bernd Schubert
Hi all,
for streaming writes onto a raid6 the current round-robin ag selection seems does not seem to be optimal. Writing 4 files from 4 threads into a single directory we get 900 MB/s, writing 4 files in 4 different directories we only get 700 MB/s (12 disks with with hw megaraid-sas). The current round-robin scheme seems to be optimized for linear raid0? With small AGs one could also argue, that choosing AGs which are not far away from each other (in respect to the number of blocks) also adds more parallel disk access for small and medium sized files.
Quoted 2 lines
Any objections against a patch to improve the AG selection?
I wouldn't say this it is optimized specifically for raid 0 environments but I lack some knowledge on this choice. The mainly reason for the round-robing IIRC, was to avoid lock contention in a single AG. spreading different files along the whole disk, and also making it able to allocate them contiguously along the disk.
But, I'm not sure what kind of optimization you have in mind and I believe another engineers will also need some extra information about what optimization you have in mind, what kind of tests you're doing (Direct I/O, buffered, pre-allocation), etc.. You'll also need to post filesystem configurations like FS aligment (su, sw options), etc.
For different write patterns, you might also want to take a look at the rotor_step procfs option, and some other options dedicated to streaming writes, that might help you in this case.
Just my $0.02
Quoted 8 lines
Thanks,
Bernd_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Signature · 6 lines
Carlos _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 13 lines · Carlos Maiolino
Quoted 12 lines · Bernd Schubert
Hi all,
for streaming writes onto a raid6 the current round-robin ag selection seems does not seem to be optimal. Writing 4 files from 4 threads into a single directory we get 900 MB/s, writing 4 files in 4 different directories we only get 700 MB/s (12 disks with with hw megaraid-sas). The current round-robin scheme seems to be optimized for linear raid0? With small AGs one could also argue, that choosing AGs which are not far away from each other (in respect to the number of blocks) also adds more parallel disk access for small and medium sized files.
Quoted 8 lines
Quoted 2 lines
Any objections against a patch to improve the AG selection?
I wouldn't say this it is optimized specifically for raid 0 environments but I lack some knowledge on this choice. The mainly reason for the round-robing IIRC, was to avoid lock contention in a single AG. spreading different files along the whole disk, and also making it able to allocate them contiguously along the disk.
Lock contention in inodes and blocks B-Trees for example, improving parallelism in the filesystem, but of course this might not be the optimal behavior for all environments. That's why XFS has a long list of tuning mkfs/mount options :-)
Quoted 6 lines
But, I'm not sure what kind of optimization you have in mind and I believe another engineers will also need some extra information about what optimization you have in mind, what kind of tests you're doing (Direct I/O, buffered, pre-allocation), etc.. You'll also need to post filesystem configurations like FS aligment (su, sw options), etc.
Quoted 7 lines
For different write patterns, you might also want to take a look at the
rotor_stepprocfs option, and some other options dedicated to streaming writes, that might help you in this case.Just my $0.02
Quoted 1 line
Just complementing my past comment
Signature · 6 lines
Carlos _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Carlos Maiolino
Quoted 16 lines · Carlos Maiolino
Quoted 14 lines · Bernd Schubert
Hi all,
for streaming writes onto a raid6 the current round-robin ag selection seems does not seem to be optimal. Writing 4 files from 4 threads into a single directory we get 900 MB/s, writing 4 files in 4 different directories we only get 700 MB/s (12 disks with with hw megaraid-sas). The current round-robin scheme seems to be optimized for linear raid0? With small AGs one could also argue, that choosing AGs which are not far away from each other (in respect to the number of blocks) also adds more parallel disk access for small and medium sized files.
Any objections against a patch to improve the AG selection?
Quoted 7 lines
Quoted 5 lines
I wouldn't say this it is optimized specifically for raid 0 environments but I lack some knowledge on this choice. The mainly reason for the round-robing IIRC, was to avoid lock contention in a single AG. spreading different files along the whole disk, and also making it able to allocate them contiguously along the disk.
Lock contention in inodes and blocks B-Trees for example, improving parallelism in the filesystem, but of course this might not be the optimal behavior for all
Agreed, more locks help to avoid that.
Quoted 7 lines
environments. That's why XFS has a long list of tuning mkfs/mount options :-)
Quoted 5 lines
But, I'm not sure what kind of optimization you have in mind and I believe another engineers will also need some extra information about what optimization you have in mind, what kind of tests you're doing (Direct I/O, buffered, pre-allocation), etc.. You'll also need to post filesystem configurations like FS aligment (su, sw options), etc.
One of my colleagues benchmarked this on one of our fast systems and another colleague current needs this system for other tests, so I don't have the exact parameters. However, it was for sure formated with options like these:
mkfs.xfs -d su=256k,sw=10 -l version=2,su=256k -isize=512 /dev/sdX
and mounted with these options:
mount -onoatime,nodiratime,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sdX <mountpoint>
Quoted 4 lines
Quoted 4 lines
For different write patterns, you might also want to take a look at the
rotor_stepprocfs option, and some other options dedicated to streaming writes, that might help you in this case.
Thanks, I didn't know that knob, I'm going to look into it. According to the comments its for inode32 only, but I need to read the xfs_alloc code first to see what it actually does.
Thanks,
Bernd
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Bernd Schubert
One of my colleagues benchmarked this on one of our fast systems and another colleague current needs this system for other tests, so I don't have the exact parameters. However, it was for sure formated with options like these:
mkfs.xfs -d su=256k,sw=10 -l version=2,su=256k -isize=512
/dev/sdXand mounted with these options:
mount -onoatime,nodiratime,largeio,inode64,swalloc,allocsize=131072k,nobarrier
/dev/sdX<mountpoint>
With all due respect, this is excessive knob-twiddling. Slow down. ;)
* V2 logs are default already, so -l version=2 is redundant.
* noatime implies nodiratime, so specifying both is redundant.
* "largeio" only changes the st_blksize value reported (from default page size to, in your case, the total stripe width). Does that actually affect your application behavior?
Backing up, what kernel & what userspace versions are you testing?
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hello Eric,
Quoted 8 lines · Eric Sandeen
Quoted 6 lines · Bernd Schubert
One of my colleagues benchmarked this on one of our fast systems and another colleague current needs this system for other tests, so I don't have the exact parameters. However, it was for sure formated with options like these:
mkfs.xfs -d su=256k,sw=10 -l version=2,su=256k -isize=512
/dev/sdX
Quoted 7 lines
Quoted 3 lines
and mounted with these options:
mount -onoatime,nodiratime,largeio,inode64,swalloc,allocsize=131072k,nobarrier
/dev/sdX<mountpoint>With all due respect, this is excessive knob-twiddling. Slow down. ;)
* V2 logs are default already, so -l version=2 is redundant.
Well, some of our customer are using fhgfs + xfs on old systems and we don't want to create a list
"with kernel/xfsprogs version < xyz, you need knob abc..."
So better add it by default, as long as it does not hurt.
Quoted 1 line
* noatime implies nodiratime, so specifying both is redundant.
Ok, I didn't add it to our default options, but I also didn't care about it.
Quoted 1 line
* "largeio" only changes the
st_blksizevalue reported (from default page size to, in your case, the total stripe width). Does that actually affect your application behavior?
Fhgfs does not care about it, but for some backup tools (I think rsync does/did? read it) it helps.
Quoted 2 lines
Backing up, what kernel & what userspace versions are you testing?
xfsprogs-3.1.1-10.el6 + kernel 3.11.
Cheers,
Bernd
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · Bernd Schubert
Hi all,
for streaming writes onto a raid6 the current round-robin ag selection seems does not seem to be optimal. Writing 4 files from 4 threads into a single directory we get 900 MB/s,
IOWs, writing all 4 files into the same AG, interleaving them in to the same physical location on disk.
Quoted 3 lines
writing 4 files in
4 different directories we only get 700 MB/s (12 disks with with hw megaraid-sas).
And that writes the 4 files into 4 different AGs, separating them into physically different regions of the disk. There's seeks between the streams there, and often cheap RAID controllers have problems with internal caching algorithms being unable to minimise seeks between streams effectively.
Quoted 2 lines
The current round-robin scheme seems to be optimized for linear raid0?
Not at all - sequential writes of large files are optimised to maintain high sequential *read* rates of the data that is being written. Also, RAID 0 and RAID 6 have exactly the same characteristics for this workload, so the behaviour you are seeing is more likely due to XFS is writing to slower areas of the disks when more streams are running in more AGs.
i.e. 900MB/s might be what you get at the outer edge of the disks, but you might only get 500MB/s at the inner edges. When writing into 4 AGs at once, they are not all going to the outer edge, and hence you see a much truer reflection of the speed of your storage than the single AG case.
Keep in mind the inode64 AG selection algorithm is optimised to spread the allocation load out over the entire filesystem address space via rotating the directory structure. It does this to increases allocation parallelism and reduce filesystem hotspots, to improves individual locality of disparate sets of data, and in general is significantly faster than any other AG selection algorithm that anyone has managed to come up with.
Quoted 6 lines
With small AGs one could also argue, that choosing
AGs which are not far away from each other (in respect to the number of blocks) also adds more parallel disk access for small and medium sized files.Any objections against a patch to improve the AG selection?
Define "improve". I'm interested in hearing new idea on how we might be able to make different allocation decisions, but changing algorithms is not just a matter of changing code.
At minimum, changing the way allocation is done will drastically change the aging characteristics of the filesystem, and so what might work really well for empty filesystems (like ext4's linear allocation algorithms) really hurts performance as filesystems get older and free space gets less contiguous....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs