How to pre-allocate files for sequential access?
I am trying to set up a 20 TB filesystem which will contain a single directory with 10000 pre-allocated 2GB files. There will be only a small number of other directories with very little activity. Once the files are preallocated there will be almost no new file creation. The files will be written sequentially, typically with writes of about 120KB, and will not be updated until the filesystem fills, at which point the earliest files will start to be overwritten (not deleted). There will be relatively little read activity. There will be a single writer process using a single thread. The filesystem application is MongoDB. I am trying to minimize seek activity during the write process, and would also like to have contiguous file allocation since the database queries will be retrieving records from a sequentially-related set of files.
The filesystem as currently created looks like this:
meta-data=/dev/sdb1 isize=256 agcount=20, agsize=268435448
blks
= sectsz=512 attr=2
data = bsize=4096 blocks=5127012091, imaxpct=1
= sunit=8 swidth=56 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0However what I see is that the earliest created files start about 5TB into the filesystem. The files are not being created in contiguous block ranges. Here is an xfs_bmap example of three files created in sequence:
0: [0..4192255]: 24075083136..24079275391
0: [0..4192255]: 26222566720..26226758975
0: [0..4192255]: 28370050304..28374242559
Currently a process is doing continuous data inserts into the database, and is writing sequential segments within the files, filling a file in about 6 minutes, and moving on to the next. There is also a small amount of write activity to a single file containing database metadata which is located about 5TB into the filesystem. The database index files are located on a separate disk.
Using seekwatcher I've determined that the actual I/O pattern, even when a small number of files is being written to, is spread over a fairly wide range of filesystem offsets, resulting in about 250 seeks per second. I don't know how to determine how long the seeks are. (I tried to upload the seekwatcher image but apparently that's not allowed). Seekwatcher shows the I/O activity is in a range between 15 and 17 TB into the filesystem. During this time there was a set of about 4 files being actively written as far as I know.
I'm guessing that the use of multiple allocation groups may explain the non-contiguous block allocation, although I read at one point that even with multiple allocation groups, files within a single directory would use the same group. I don't believe I need multiple allocation groups for this application due to the single writer and the fact that all files will be preallocated before use. Would it be reasonable to force mkfs to use a single 20TB allocation group, and would this be likely to produce contiguous block allocation?
This is kernel 3.0.25 using xfsprogs 3.1.1.
Signature · 7 lines
View this message in context: http://old.nabble.com/How-to-pre-allocate-files-for-sequential-access--tp33564834p33564834.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 25 lines · troby
I am trying to set up a 20 TB filesystem which will contain a single directory with 10000 pre-allocated 2GB files. There will be only a small number of other directories with very little activity. Once the files are preallocated there will be almost no new file creation. The files will be written sequentially, typically with writes of about 120KB, and will not be updated until the filesystem fills, at which point the earliest files will start to be overwritten (not deleted). There will be relatively little read activity. There will be a single writer process using a single thread. The filesystem application is MongoDB. I am trying to minimize seek activity during the write process, and would also like to have contiguous file allocation since the database queries will be retrieving records from a sequentially-related set of files.
The filesystem as currently created looks like this:meta-data=/dev/sdb1 isize=256 agcount=20, agsize=268435448
blks= sectsz=512 attr=2 data = bsize=4096 blocks=5127012091, imaxpct=1 = sunit=8 swidth=56 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
Quoted 6 lines
However what I see is that the earliest created files start about 5TB into the filesystem. The files are not being created in contiguous block ranges. Here is an
xfs_bmapexample of three files created in sequence:
0: [0..4192255]: 24075083136..24079275391
0: [0..4192255]: 26222566720..26226758975
0: [0..4192255]: 28370050304..28374242559
Please try again from scratch after mounting with the -o inode64 mount option, which we will make default Real Soon Now(tm).
That option will more evenly spread inodes & file data throughout your whole 20T.
Quoted 10 lines
Currently a process is doing continuous data inserts into the database, and is writing sequential segments within the files, filling a file in about 6 minutes, and moving on to the next. There is also a small amount of write activity to a single file containing database metadata which is located about 5TB into the filesystem. The database index files are located on a separate disk.
Using seekwatcher I've determined that the actual I/O pattern, even when a small number of files is being written to, is spread over a fairly wide range of filesystem offsets, resulting in about 250 seeks per second. I
Some of that seeking will be log writes, in the middle of the fs.
Quoted 10 lines
don't know how to determine how long the seeks are. (I tried to upload the seekwatcher image but apparently that's not allowed). Seekwatcher shows the I/O activity is in a range between 15 and 17 TB into the filesystem. During this time there was a set of about 4 files being actively written as far as I know.
I'm guessing that the use of multiple allocation groups may explain the non-contiguous block allocation, although I read at one point that even with multiple allocation groups, files within a single directory would use the same group. I don't believe I need multiple allocation groups for this
That's generally true, but only until that group fills. If you fill the whole fs with files in the same dir, of course it will have to spill to other AGs...
Quoted 4 lines
application due to the single writer and the fact that all files will be preallocated before use. Would it be reasonable to force mkfs to use a single 20TB allocation group, and would this be likely to produce contiguous block allocation?
The AG size maxes out at 1T, so you can't make a single AG.
I'd give it another shot with inode64 and see if things look a little better, or at least a bit more predictable.
-Eric
Quoted 3 lines
This is kernel 3.0.25 using xfsprogs 3.1.1.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
On 04.04.2012 16:57, troby wrote:
...
I think the easiest solution would be to create the said number of files, but with dummy-filenames.
Then write a script das does a xfs_bmap on each file, sorts them and then renames the dummy-files to the correct-name in the order they are on disc.
Bis denn
Signature · 9 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Matthias Schniedermeyer
On 04.04.2012 16:57, troby wrote:
...
I think the easiest solution would be to create the said number of files, but with dummy-filenames.
Then write a script das does a
xfs_bmapon each file, sorts them and then renames the dummy-files to the correct-name in the order they are on disc.
Inode64 will likely keep them in order without needing tricks like that.
Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
On 4/4/2012 6:57 PM, troby wrote:
The high points:
Quoted 8 lines
20 TB filesystem
directory with 10000 pre-allocated 2GB files.
written sequentially, writes of about 120KB
overwritten (not deleted)
single writer process using a single thread.
MongoDB.
minimize seek activity
contiguous file allocation
Current filesystem configuration:
Quoted 11 lines
The filesystem as currently created looks like this:
meta-data=/dev/sdb1 isize=256 agcount=20, agsize=268435448
blks= sectsz=512 attr=2 data = bsize=4096 blocks=5127012091, imaxpct=1 = sunit=8 swidth=56 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
RAID configuration based on xfs_info analysis:
Hardware RAID controller, unknown type, unknown cache configuration 8 * 3TB disks, possibly AF and/or 'green', RPM: 5600-7200, cache 32-64MB RAID5 w/32KB chunk, 7 spindle stripe
Quoted 6 lines
However what I see is that the earliest created files start about 5TB into the filesystem. The files are not being created in contiguous block ranges. Here is an
xfs_bmapexample of three files created in sequence:
0: [0..4192255]: 24075083136..24079275391
0: [0..4192255]: 26222566720..26226758975
0: [0..4192255]: 28370050304..28374242559
This is a result of the inode32 allocator behavior. Try the inode64 allocator as Eric recommended.
Quoted 3 lines
Using seekwatcher I've determined that the actual I/O pattern, even when a small number of files is being written to, is spread over a fairly wide range of filesystem offsets, resulting in about 250 seeks per second. I
I don't care for the use of the term "seek" as there is not a 1:1 correlation between these "seeks" and actual disk head seeks. The latter is what is always of concern because that's where all the latency is. Folks most often become head seek bound due to the extra read seeks in RMW operations when using parity RAID. These extra RMW seeks are completely hidden from Seekwatcher when using hardware RAID, though should be somewhat visible with md RAID.
Quoted 5 lines
don't know how to determine how long the seeks are. (I tried to upload the seekwatcher image but apparently that's not allowed). Seekwatcher shows the I/O activity is in a range between 15 and 17 TB into the filesystem. During this time there was a set of about 4 files being actively written as far as I know.
It's coarse, but you might start with iostat interactively to get a rough idea WRT overall latency. Look at the await column, man iostat.
This gives 1 second interval.
$ iostat -d 1 -x /dev/sdb1
Quoted 4 lines
I'm guessing that the use of multiple allocation groups may explain the non-contiguous block allocation, although I read at one point that even with multiple allocation groups, files within a single directory would use the same group.
What you describe is the behavior of the inode64 allocator, which is optional. IIRC the [default] inode32 allocator puts all the directories in the first AG and spreads the files around the remaining AGs, which is what you are seeing.
Quoted 5 lines
I don't believe I need multiple allocation groups for this application due to the single writer and the fact that all files will be preallocated before use. Would it be reasonable to force mkfs to use a single 20TB allocation group, and would this be likely to produce contiguous block allocation?
As Eric mentioned, the max AG size is 1TB, though this may increase in the future as aerial density increases.
One final thought I'd like to share is that parity RAID, level 5 or 6, is rarely, if ever, good for write mostly applications, especially for overwriting applications. You're preallocating all of your files on a fresh XFS, so you should be able to avoid the extra seeks of RMW until you start overwriting them. Once that happens your request latency will double, if not more, and your throughput will take a 2x or more nose dive. If you can get by with 11-12TB, I'd rebuild those 8 drives as a RAID10 array. Here are your advantages:
1. No RMW penalty, near zero RAID computation, high throughput 2. RAID5 write throughput drops 5-20x when degraded/rebuilding 3. RAID10 loses zero performance when degraded 4. RAID10 rebuild time is 10-20x faster than RAID5 5. RAID10 suffers only a small performance drop during rebuild
Need all 20TB, or something in between 10-20? What is your current hardware setup and allowed upgrade budget, if any? I'd be willing to offer you some optimal hardware choices/advice based on your current setup, to achieve the optimal RAID10, if you would like.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Thanks all for your help. Due to unrelated collateral damage within the database software, I had the "opportunity" to rebuild the filesystems using some of your suggestions. I set up external log devices for the two busiest filesystems (RAID5 for the row data, a RAID10 on 4 drives for indexes) and configured the stripe widths correctly this time. The inode64 mount option did result in sequential allocation, with a single 2GB segment per file, for all files which were pre-allocated by the MongoDB software. A small number of similar files I created manually using dd from /dev/zero with a single 2GB block showed 3 or 4 segments coming from different AG's, with corresponding disparate block ranges, but there aren't enough of those to cause problems.
One thing that puzzles me is that despite my configuration of the underlying RAID stripe geometry both at filesystem creation and mount time, all the filesystems show average request sizes (mostly writes at this time) of around 240 sectors. This is correct for some of them, but the RAID1 stripe is twice that wide and the RAID5 almost 4 times as wide. The files being written are all memory-mapped, so I'm wondering if that means the kernel uses some other settings besides the fstab mount options to determine the request size. The flush activity only happens a few times a minute and only lasts a second or two, so I don't think there's really a significant performance impact under the current load. And since the writes are actually going to a 1GB controller cache, I suspect there is enough time for the controller to assemble a full RAID5 stripe before writing to disk.
Signature · 7 lines
View this message in context: http://old.nabble.com/How-to-pre-allocate-files-for-sequential-access--tp33564834p33663734.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs