file streams allocator behavior
I recall reading a while back something about disabling the filestreams allocator, or at least changing its behavior, but I'm unable to find that.
What I'm trying to do is use parallel dd w/O_DIRECT to write 44 files in parallel to 44 directories, thus all 44 AGs, in one test, then write 44 files to one dir, one AG, in another test. The purpose of this quick/dirty exercise is to demonstrate throughput differences due to full platter seeking in the former case and localized seeking in the latter case.
But of course the problem I'm running into in the single directory case is that the filestreams allocator starts writing all of the 44 files into the appropriate AG, but then begins allocating extents for each file in other AGs. This is of course defeating the purpose of the tests.
Quoted 5 lines
/mnt/VOL1/43# for i in
seq 0 43;doxfs_bmap-v test-$i;done test-0:EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..1535]: 92341791520..92341793055 43 (160..1695) 1536 01111 1: [1536..3071]: 92341794688..92341796223 43 (3328..4863) 1536 00011
...
Quoted 2 lines
88: [135168..136703]: 9972480..9974015 0 (9972480..9974015) 1536 00011 89: [136704..138239]: 9984768..9986303 0 (9984768..9986303) 1536 00011
...
Quoted 2 lines
146: [224256..225791]: 2158167552..2158169087 1 (10684032..10685567) 1536 147: [225792..227327]: 2158181376..2158182911 1 (10697856..10699391) 1536
...
Quoted 2 lines
160: [245760..254975]: 10744866688..10744875903 5 (7449088..7458303) 9216 00011 161: [254976..256511]: 10744877440..10744878975 5 (7459840..7461375) 1536 00011
...
...
Quoted 4 lines
test-43:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..1535]: 92341936000..92341937535 43 (144640..146175) 1536 00011 1: [1536..3071]: 92342003584..92342005119 43 (212224..213759) 1536 00011
...
Quoted 2 lines
69: [105984..107519]: 4303912064..4303913599 2 (8945024..8946559) 1536 00011 70: [107520..109055]: 4303922816..4303924351 2 (8955776..8957311) 1536 00011
...
...
Quoted 1 line
180: [276480..278015]: 8598943744..8598945279 4 (9009664..9011199) 1536 00011
...
Quoted 2 lines
181: [278016..279551]: 10744961920..10744963455 5 (7544320..7545855) 1536 00011 182: [279552..281087]: 10744968064..10744969599 5 (7550464..7551999) 1536 00011
...
...
Files being created are 1.6 GB. Filesystem is 44 TB. AGs are 1 TB. AGs are 0-43. Directories, /mnt/VOL1/0 - /mnt/VOL1/43. Device is a single RAID5 LUN.
How can I disable or change the filestreams behavior so all files go into the one AG for the single directory test?
Thanks,
Stan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Stan Hoeppner
I recall reading a while back something about disabling the filestreams allocator, or at least changing its behavior, but I'm unable to find that.
What I'm trying to do is use parallel dd w/O_DIRECT to write 44 files in parallel to 44 directories, thus all 44 AGs, in one test, then write 44 files to one dir, one AG, in another test. The purpose of this quick/dirty exercise is to demonstrate throughput differences due to full platter seeking in the former case and localized seeking in the latter case.
Quoted 4 lines
But of course the problem I'm running into in the single directory case is that the filestreams allocator starts writing all of the 44 files into the appropriate AG, but then begins allocating extents for each file in other AGs. This is of course defeating the purpose of the tests.
That's caused by allocator contention. When you try to write 44 files to the same dir in parallel, they'll all start with the same target AG, but then when one thread is allocating into AG 43 and has the AG locked, a second attempt to allocate to than AG will see the AG locked and so it will move to find the next AG that is not locked.
Remember, AGs were not originally designed for confining physical locality - they are designed to allow allocator parallelism. Hence once the file has jumped to a new AG it will try to allocate sequentially from that point onwards in that same AG, until either ENOSPC or further contention.
Hence with a workload like this, if the writes continue for long enough each file will end up finding it's own uncontended AG and hence mostly end up contiguous on disk and not getting blocked waiting for allocation on other files. When you have as many writers as there are AGs, however, such a steady state is generally not possible as there will always be files trying to write into the same AG.
As it is, filestreams is not designed for this sort of parallel workload. filestreams is designed to separate single threaded streams of IO into different locations, not handle concurrent writes into multiple files in the same directory.
As it is, the inode64 will probably demonstrate exactly the same behaviour because it will start by trying to write all the files to the same AG and hence hit allocator contention, too.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 16 lines · Dave Chinner
Quoted 10 lines · Stan Hoeppner
I recall reading a while back something about disabling the filestreams allocator, or at least changing its behavior, but I'm unable to find that.
What I'm trying to do is use parallel dd w/O_DIRECT to write 44 files in parallel to 44 directories, thus all 44 AGs, in one test, then write 44 files to one dir, one AG, in another test. The purpose of this quick/dirty exercise is to demonstrate throughput differences due to full platter seeking in the former case and localized seeking in the latter case.
Quoted 4 lines
But of course the problem I'm running into in the single directory case is that the filestreams allocator starts writing all of the 44 files into the appropriate AG, but then begins allocating extents for each file in other AGs. This is of course defeating the purpose of the tests.
Quoted 6 lines
That's caused by allocator contention. When you try to write 44 files to the same dir in parallel, they'll all start with the same target AG, but then when one thread is allocating into AG 43 and has the AG locked, a second attempt to allocate to than AG will see the AG locked and so it will move to find the next AG that is not locked.
That's what I suspected given what I was seeing.
Quoted 2 lines
Remember, AGs were not originally designed for confining physical locality - they are designed to allow allocator parallelism. Hence
Right. But they sure do come in handy when used this way with preallocated files. I suspect we'll make heavy use of this when they have me back to implement my recommendations.
Quoted 11 lines
once the file has jumped to a new AG it will try to allocate sequentially from that point onwards in that same AG, until either ENOSPC or further contention.
Hence with a workload like this, if the writes continue for long enough each file will end up finding it's own uncontended AG and hence mostly end up contiguous on disk and not getting blocked waiting for allocation on other files. When you have as many writers as there are AGs, however, such a steady state is generally not possible as there will always be files trying to write into the same AG.
Yep. That's exactly what xfs_bmap was showing. Some files had extents in 3-4 AGs, some in 8 or more AGs.
Quoted 4 lines
As it is, filestreams is not designed for this sort of parallel workload. filestreams is designed to separate single threaded streams of IO into different locations, not handle concurrent writes into multiple files in the same directory.
Right. I incorrectly refered to the filestreams allocator as I didn't know of the inode64 congestion control mechanism at that time. I simply had recalled one of your emails where you and Christoph were discussing mods to the filestreams allocator, and the file pattern to the AGs looked familiar.
Quoted 3 lines
As it is, the inode64 will probably demonstrate exactly the same behaviour because it will start by trying to write all the files to the same AG and hence hit allocator contention, too.
I was using the inode64 allocator, and yes, it did. Brian saved my bacon as I wanted to use preallocated files but didn't know exactly how.
I knew all their extents would be in the AG I wanted them in, given the
way I intended to do it.
And doing this demonstrated what I anticipated. Writing 2GB into each of 132 files with 132 parallel dd processes with O_DIRECT, 264GB total, I achieved:
AG0 only 1377 MB/s
AGs 0-43 767 MB/s
Due to the nature of the application, we should be able to distribute the files in such a manner that only 1 or 2 adjacent AGs are being accesses concurrently. This will greatly reduce head seeking, increasing throughput, as demonstrated above. All thanks to the allocation group architecture of XFS. We'd not be able to do this with any other filesystem AFAIK.
Thanks,
Stan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs