df bigger than ls?
I have created some files spread across 12 XFS filesystems, using a glusterfs "striped" volume. This writes files with lots of holes(*) - so I would expect the space reported by 'du' to be less than the space reported by 'ls'.
However it's the other way round - du is reporting more space used than ls! Here's what I mean: I'm looking at the files directly on the underlying disk mount point, not via glusterfs at all.
$ ls -lh /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff -rw-rw-r-- 1 brian brian 1.1G 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff $ ls -lk /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff -rw-rw-r-- 1 brian brian 1059968 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff $ du -h /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff $ du -k /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2096392 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
And here's what xfs_bmap reports:
root@storage1:~# xfs_bmap -vp /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..255]: 2933325744..2933325999 2 (3059152..3059407) 256 00000 1: [256..3071]: hole 2816 2: [3072..3327]: 2933326000..2933326255 2 (3059408..3059663) 256 00000 3: [3328..6143]: hole 2816 4: [6144..8191]: 2933326472..2933328519 2 (3059880..3061927) 2048 00000 5: [8192..9215]: hole 1024 6: [9216..13311]: 2933369480..2933373575 2 (3102888..3106983) 4096 00000 7: [13312..15359]: hole 2048 8: [15360..23551]: 2933375624..2933383815 2 (3109032..3117223) 8192 00000 9: [23552..24575]: hole 1024 10: [24576..40959]: 2933587168..2933603551 2 (3320576..3336959) 16384 00000 11: [40960..43007]: hole 2048 12: [43008..75775]: 2933623008..2933655775 2 (3356416..3389183) 32768 00000 13: [75776..76799]: hole 1024 14: [76800..142335]: 2933656800..2933722335 2 (3390208..3455743) 65536 00000 15: [142336..144383]: hole 2048 16: [144384..275455]: 2933724384..2933855455 2 (3457792..3588863) 131072 00000 17: [275456..276479]: hole 1024 18: [276480..538623]: 2935019808..2935281951 2 (4753216..5015359) 262144 00000 19: [538624..540671]: hole 2048 20: [540672..1064959]: 2935284000..2935808287 2 (5017408..5541695) 524288 00000 21: [1064960..1065983]: hole 1024 22: [1065984..2114559]: 2935809312..2936857887 2 (5542720..6591295) 1048576 00000 23: [2114560..2116607]: hole 2048 24: [2116608..2119935]: 2943037984..2943041311 2 (12771392..12774719) 3328 00000 root@storage1:~#
Given that these values are all in 512-byte disk blocks, the total file size is (2119935 + 1) * 512 which agrees with ls. And some proportion of it is holes, so du should report less than this, shouldn't it?
(Aside: it starts off being 11/12th holes as expected, but after a while isn't. This may be a different bug, possibly in glusterfs itself)
I guess 'du' gets its info from stat(), and stat() also says the file is using 4192784 blocks which is 2096392 KB:
$ stat /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff File: `/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff' Size: 1085407232 Blocks: 4192784 IO Block: 4096 regular file Device: 810h/2064d Inode: 8595657903 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ brian) Gid: ( 1000/ brian) Access: 2012-03-07 15:20:36.044365215 +0000 Modify: 2012-03-07 15:19:33.640364277 +0000 Change: 2012-03-07 15:19:33.640364277 +0000
Finally, here is xfs_db dump of the inode:
# ls -i /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 8595657903 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff ... xfs_db> inode 8595657903 xfs_db> p core.magic = 0x494e core.mode = 0100664 core.version = 2 core.format = 3 (btree) core.nlinkv2 = 1 core.onlink = 0 core.projid_lo = 0 core.projid_hi = 0 core.uid = 1000 core.gid = 1000 core.flushiter = 231 core.atime.sec = Wed Mar 7 15:20:36 2012 core.atime.nsec = 044365215 core.mtime.sec = Wed Mar 7 15:19:33 2012 core.mtime.nsec = 640364277 core.ctime.sec = Wed Mar 7 15:19:33 2012 core.ctime.nsec = 640364277 core.size = 1085407232 core.nblocks = 262370 core.extsize = 0 core.nextents = 13 core.naextents = 1 core.forkoff = 15 core.aformat = 2 (extents) core.dmevmask = 0 core.dmstate = 0 core.newrtbm = 0 core.prealloc = 0 core.realtime = 0 core.immutable = 0 core.append = 0 core.sync = 0 core.noatime = 0 core.nodump = 0 core.rtinherit = 0 core.projinherit = 0 core.nosymlinks = 0 core.extsz = 0 core.extszinherit = 0 core.nodefrag = 0 core.filestream = 0 core.gen = 1875303423 next_unlinked = null u.bmbt.level = 1 u.bmbt.numrecs = 1 u.bmbt.keys[1] = [startoff] 1:[0] u.bmbt.ptrs[1] = 1:537294687 a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,537253238,1,0] xfs_db>
Platform: ubuntu 11.10
Linux 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Have I missed something obvious, or is there a bug of some sort here?
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Brian Candler
core.size = 1085407232
core.nblocks = 262370
core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB
(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)
This begs the question of where stat() is getting its info from?
Ah... but I've found that after unmounting and remounting the filesystem (which I had to do for xfs_db), du and stat report the correct info.
In fact, dropping the inode caches is sufficient to fix the problem:
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~# echo 3 >/proc/sys/vm/drop_caches root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~#
Very odd, but not really a major problem other than the confusion it causes.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Brian Candler
Quoted 2 lines · Brian Candler
core.size = 1085407232
core.nblocks = 262370core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB
(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)
Quoted 6 lines
This begs the question of where
stat()is getting its info from?Ah... but I've found that after unmounting and remounting the filesystem (which I had to do for
xfs_db), du and stat report the correct info.In fact, dropping the inode caches is sufficient to fix the problem:
Yep.
XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)
$ git log --pretty=oneline fs/xfs | grep specul b8fc82630ae289bb4e661567808afc59e3298dce xfs: speculative delayed allocation uses rounddown_power_of_2 badly 055388a3188f56676c21e92962fc366ac8b5cb72 xfs: dynamic speculative EOF preallocation
so:
# dd if=/dev/zero of=bigfile bs=1M count=1100 &>/dev/null # ls -lh bigfile -rw-r--r--. 1 root root 1.1G Mar 7 11:47 bigfile # du -h bigfile 1.1G bigfile
but:
# rm -f bigfile # for I in `seq 1 1100`; do dd if=/dev/zero of=bigfile conv=notrunc bs=1M seek=$I count=1 &>/dev/null; done # ls -lh bigfile -rw-r--r--. 1 root root 1.1G Mar 7 11:49 bigfile # du -h bigfile 2.0G bigfile
This should get freed when the inode is dropped from the cache; hence your cache drop bringing it back to size.
But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:
# du -hc bigfile* 2.0G bigfile1 1.1G bigfile2 907M bigfile3
Dave, is this working as intended? I know the speculative preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?
-Eric
Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~# echo 3 >/proc/sys/vm/drop_caches root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~#Very odd, but not really a major problem other than the confusion it causes.
Quoted 9 lines
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Eric Sandeen
Quoted 10 lines · Brian Candler
Quoted 2 lines · Brian Candler
core.size = 1085407232
core.nblocks = 262370core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB
(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)
This begs the question of where
stat()is getting its info from?
stat(2) also reported delayed allocation reservations that are only kept in memory.
....
Quoted 8 lines
so:
# dd if=/dev/zero of=bigfile bs=1M count=1100 &>/dev/null # ls -lh bigfile -rw-r--r--. 1 root root 1.1G Mar 7 11:47 bigfile # du -h bigfile 1.1G bigfile
Quoted 8 lines
but:
# rm -f bigfile # for I in `seq 1 1100`; do dd if=/dev/zero of=bigfile conv=notrunc bs=1M seek=$I count=1 &>/dev/null; done # ls -lh bigfile -rw-r--r--. 1 root root 1.1G Mar 7 11:49 bigfile # du -h bigfile 2.0G bigfile
This is tripping the NFS server write pattern heuristic. i.e. it is detecting repeated open/write at EOF/close patterns and so is not truncating away the speculative EOF reservation on close(). This si what prevents fragmentation of files being written concurrently with this pattern.
Quoted 2 lines
This should get freed when the inode is dropped from the cache; hence your cache drop bringing it back to size.
Right. It is assumes that once you've triggered that heuristic, the preallocation needs to last for as long as the inode is in the working set. The inode cache tracks the current working set, so the preallocation release is tied to cache eviction.
Quoted 9 lines
But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:
# du -hc bigfile* 2.0G bigfile1 1.1G bigfile2 907M bigfile3Dave, is this working as intended?
Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/
Quoted 4 lines
I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?
We don't track what files have current active preallocations, we only reduce the preallocation size as the filesystem nears ENOSPC. This generally works just fine in situations where the filesystem size is significantly greater than the maximum extent size. i.e. the common case
The problem you are tripping over here is that the maximum extent size is greater than the filesystem size, so the preallocation size is also greater than the filesystem size and hence can contribute significantly to premature ENOSPC. I see two possible ways to minimise this problem:
1. reduce the maximum speculative preallocation size based on filesystem size at mount time.
2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.
The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.
The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....
Cheers,
Dave.
Quoted 49 lines
-Eric
Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~# echo 3 >/proc/sys/vm/drop_caches root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~#Very odd, but not really a major problem other than the confusion it causes.
Quoted 9 lines
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · Dave Chinner
On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:
...
Quoted 7 lines
Quoted 4 lines
# du -hc bigfile* 2.0G bigfile1 1.1G bigfile2 907M bigfile3Dave, is this working as intended?
Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/
Well, sure ;)
Quoted 11 lines
Quoted 4 lines
I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?We don't track what files have current active preallocations, we only reduce the preallocation size as the filesystem nears ENOSPC. This generally works just fine in situations where the filesystem size is significantly greater than the maximum extent size. i.e. the common case
Quoted 5 lines
The problem you are tripping over here is that the maximum extent size is greater than the filesystem size, so the preallocation size is also greater than the filesystem size and hence can contribute significantly to premature ENOSPC. I see two possible ways to minimise this problem:
FWIW I'm not overly worried about my 4G filesystem, that was just an obviously extreme case to test the behavior.
I'd be more interested to know if the behavior was causing any issues for Brian's case, or if it was just confusing. :)
Quoted 10 lines
1. reduce the maximum speculative preallocation size based on filesystem size at mount time.
2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.
The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.
Might be worth it; even though a 4G fs is obviously not a design point, it's good not to have diabolical behavior. Although I suppose if xfstests doesn't hit it in everything it does, it's probably not a big deal. ISTR you did have to fix one thing up, though.
Quoted 7 lines
The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....
It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.
-Eric
Quoted 53 lines
Cheers,
Dave.
Quoted 49 lines
-Eric
Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 2.0G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~# echo 3 >/proc/sys/vm/drop_caches root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff 1.1G /disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff root@storage1:~#Very odd, but not really a major problem other than the confusion it causes.
Quoted 9 lines
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Eric Sandeen
It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.
And presumably df on the filesystem would also show it approaching 100% utilisation?
I'm used to this where a large file has been unlinked but is still open. The preallocation case is a new one to me though.
How about if the total of all preallocations were limited to some small percentage of the total filesystem size? If you reach this limit and want to preallocate some space for another file you'd have to either drop or shrink an older preallocation.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Brian Candler
Quoted 4 lines · Eric Sandeen
It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.
And presumably df on the filesystem would also show it approaching 100% utilisation?
*nod*
Quoted 7 lines
I'm used to this where a large file has been unlinked but is still open. The preallocation case is a new one to me though.
How about if the total of all preallocations were limited to some small percentage of the total filesystem size? If you reach this limit and want to preallocate some space for another file you'd have to either drop or shrink an older preallocation.
There is no separate accounting for preallocation - it is considered used space so this currently can't be done even if there was some method for tracking and trimming speculatively preallocated space.
Realistically, if you aren't running out of space there is no reason to limit speculative preallocation. Indeed, if we didn't add delalloc blocks to the block count in stat(2) output so they showed up in df, almost no-one would even know about the fact that this is done....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Dave Chinner
Quoted 5 lines · Eric Sandeen
But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:
Quoted 7 lines
Quoted 4 lines
# du -hc bigfile* 2.0G bigfile1 1.1G bigfile2 907M bigfile3Dave, is this working as intended?
Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/
I have seen a similar problem on some very large filesystems too. This is not just dependent upon the size of the filesystem, but also the workload. I think it is also a big problem for folks using quotas.
Quoted 20 lines
Quoted 4 lines
I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?I see two possible ways to
minimise this problem:1. reduce the maximum speculative preallocation size based on filesystem size at mount time.
2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.
The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.
Quoted 7 lines
The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....
Alex and I discussed this problem briefly awhile ago. What is the best way to lose when you hit ENOSPC (project quotas) or EDQUOT in xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.
I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)
Now that block reservations count toward quotas the symptom will probably be a little different.
-Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Ben Myers
Quoted 8 lines · Dave Chinner
Quoted 7 lines · Eric Sandeen
and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:
# du -hc bigfile* 2.0G bigfile1 1.1G bigfile2 907M bigfile3
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Dave, is this working as intended?
Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/
I have seen a similar problem on some very large filesystems too.
So report the problems to the list, don't keep them to yourself. If we don't hear about problems, then as far as we are concerned they don't exist.
Quoted 3 lines
This
is not just dependent upon the size of the filesystem, but also the workload. I think it is also a big problem for folks using quotas.
Nobody has reported significant quota problems, either....
Quoted 6 lines
Alex and I discussed this problem briefly awhile ago. What is the best way to lose when you hit ENOSPC (project quotas) or EDQUOT in
xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.
Quoted 6 lines
I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)
And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).
Quoted 2 lines
Now that block reservations count toward quotas the symptom will probably be a little different.
Block reservations have always counted towards quotas, it's just that they were never reported.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 16 lines · Dave Chinner
Quoted 2 lines · Ben Myers
Quoted 1 line · Dave Chinner
On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:
Alex and I discussed this problem briefly awhile ago. What is the best
Quoted 11 lines
way to lose when you hit ENOSPC (project quotas) or EDQUOT in
xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)
Quoted 4 lines
And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).
Yeah, maybe not. A single list of inodes with speculative prealloc does seem a good place to start. Later maybe you would not want to scan/filter through all of them and we could add additional lists for dquots. My suggestion of using LRU was because I think that the oldest unused speculative preallocs should be the first to go. One chronic over-quota user shouldn't be able to punish everyone else.
Quoted 5 lines
Quoted 2 lines
Now that block reservations count toward quotas the symptom will probably be a little different.
Block reservations have always counted towards quotas, it's just that they were never reported.
Aw. My mistake. ;)
-Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Ben Myers
Quoted 16 lines · Dave Chinner
Quoted 2 lines · Ben Myers
Quoted 1 line · Dave Chinner
On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:
Alex and I discussed this problem briefly awhile ago. What is the best
Quoted 11 lines
way to lose when you hit ENOSPC (project quotas) or EDQUOT in
xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)
Quoted 7 lines
Quoted 4 lines
And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).
Yeah, maybe not. A single list of inodes with speculative prealloc does seem a good place to start.
Except it involves growing the inode by a struct listhead. Keeping per-inode memory usage down is extremely important, and growing it by 16 bytes for a rare corner case is not a particularly good tradeoff. Especially as using a radix tree tag for tracking doesn't involve any increase in memory usage and tag based scans are lock-free and quite efficient.
Quoted 5 lines
Later maybe you would not want to
scan/filter through all of them and we could add additional lists for dquots. My suggestion of using LRU was because I think that the oldest unused speculative preallocs should be the first to go. One chronic over-quota user shouldn't be able to punish everyone else.
Sure. But we only need to check if the inode with prealloc belongs to the user/group/project that hit EDQUOT before taking action. We don't need an LRU for that.
Indeed, I don't believe that removing the entire preallocation on each inode is the right thing to do, either. Chopping it in half is probably the right thing to do so that we free up lots of space in the case of excessive (large preallocation) but don't cause significant extra fragmentation for the work that currently requires it.
If we still get EDQUOT after trimming speculative preallocations, we can try a second time. If reducing speculative preallocations by 75% doesn't avoid the EDQUOT (or ENOSPC), then I think we can consider it a real error....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Eric Sandeen
XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)
What was the default before speculative preallocation was added (or changed somewhere in 2.6.3x line afaik) ?
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Arkadiusz Miśkiewicz
Quoted 4 lines · Eric Sandeen
XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)What was the default before speculative preallocation was added (or changed somewhere in 2.6.3x line afaik) ?
It's changed many times over the life of XFS.
Originally it was quite a large number and could happen anywhere in the file - I can't remember exactly what it was but some time around 1997 on Irix it got brought back to 16 blocks per written region because of issues with Irix NFS servers keeping tens of thousands of files open (and hence never having speculative preallocation removed) with multiple preallocations all through sparse files and hence running filesystems and user quotas out of space.
This 16 block size was the default when ported to linux when porting to Linux, but would also round up to stripe unit size if the file was larger than 512kb. In 2004 the speculative preallocation was reduced to EOF preallocation only instead of anywhere in a hole in a file. Somewhere along the line the default size got changed to PAGE_SIZE rather than 16 blocks. Then the allocsize mount option was introduced in 2005 to enable it to be increased for those that needed large preallocation (typically PVR users), and the maximum increased to 1GB.
It got broken by changes in 2008, fixed in 2009, and then made dynamic in 2.6.38 and now it behaves more like it did in 1995 for files being extended.
So there's a long history of changing the speculative preallocation behaviour of XFS to adapt to changing storage characteristics, workloads and capacities.....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Eric Sandeen
XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)
Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!
Here are the extent maps for two of the twelve files in my stripe:
root@storage1:~# xfs_bmap /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff: 0: [0..255]: 2933325744..2933325999 1: [256..3071]: hole 2: [3072..3327]: 2933326000..2933326255 3: [3328..6143]: hole 4: [6144..8191]: 2933326472..2933328519 5: [8192..9215]: hole 6: [9216..13311]: 2933369480..2933373575 7: [13312..15359]: hole 8: [15360..23551]: 2933375624..2933383815 9: [23552..24575]: hole 10: [24576..40959]: 2933587168..2933603551 11: [40960..43007]: hole 12: [43008..75775]: 2933623008..2933655775 13: [75776..76799]: hole 14: [76800..142335]: 2933656800..2933722335 15: [142336..144383]: hole 16: [144384..275455]: 2933724384..2933855455 17: [275456..276479]: hole 18: [276480..538623]: 2935019808..2935281951 19: [538624..540671]: hole 20: [540672..1064959]: 2935284000..2935808287 21: [1064960..1065983]: hole 22: [1065984..2114559]: 2935809312..2936857887 23: [2114560..2116607]: hole 24: [2116608..2119935]: 2943037984..2943041311 root@storage1:~# xfs_bmap /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff: 0: [0..255]: hole 1: [256..511]: 2933194944..2933195199 2: [512..3327]: hole 3: [3328..3839]: 2933195200..2933195711 4: [3840..6399]: hole 5: [6400..8447]: 2933204416..2933206463 6: [8448..9471]: hole 7: [9472..13567]: 2933328792..2933332887 8: [13568..15615]: hole 9: [15616..23807]: 2933334936..2933343127 10: [23808..24831]: hole 11: [24832..41215]: 2933344152..2933360535 12: [41216..43263]: hole 13: [43264..76031]: 2934672032..2934704799 14: [76032..77055]: hole 15: [77056..142591]: 2934705824..2934771359 16: [142592..144639]: hole 17: [144640..275711]: 2934773408..2934904479 18: [275712..276735]: hole 19: [276736..538879]: 2934343328..2934605471 20: [538880..540927]: hole 21: [540928..1065215]: 2935498152..2936022439 22: [1065216..1066239]: hole 23: [1066240..2114815]: 2936023464..2937072039 24: [2114816..2116863]: hole 25: [2116864..2120191]: 2937074088..2937077415
You can see that at the start it works fine. There is a stripe size of 256 blocks, so:
* disk 1: data for 1 x 256 blocks <-- stripe 0, chunk 0
hole for 11 x 256 blocks
data for 1 x 256 block <-- stripe 0, chunk 1
...
* disk 2: hole for 1 x 256 blocks
data for 1 x 256 blocks <-- stripe 1, chunk 0
hole for 11 x 256 blocks
data for 1 x 256 blocks <-- stripe 1, chunk 1
...
But after four chunks it gets screwed up. By the end the files are mostly extent and hardly any hole. The extent sizes increase in roughly powers of two which seems to match the speculative preallocation algorithm.
I think this ought to be fixable. For example, if you seek *into* the preallocated area and start writing, could you change the preallocation to start at this location with a hole before?
(But would that mess up some 'seeky' workloads like databases? But they would have ended up creating holes in filesystems which don't have preallocation, so I doubt they do this)
Or for a more sledgehammer approach: if a file already contains any holes then you could just disable preallocation completely.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Brian Candler
Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!
Here is a standalone testcase.
$ for i in {0..19}; do dd if=/dev/zero of=testfile bs=128k count=1 seek=$[$i * 12]; done
$ xfs_bmap testfile
testfile:
0: [0..255]: 1465133392..1465133647
1: [256..3071]: hole
2: [3072..5119]: 1465136464..1465138511
3: [5120..6143]: hole
4: [6144..10239]: 1465139536..1465143631
5: [10240..12287]: hole
6: [12288..20479]: 1465145680..1465153871
7: [20480..21503]: hole
8: [21504..37887]: 1465154896..1465171279
9: [37888..39935]: hole
10: [39936..58623]: 1465173328..1465192015I expected to see: 20 extents of 256 blocks and 19 holes of 2816 blocks.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Brian Candler
Quoted 3 lines · Brian Candler
Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!
How is the filesystem supposed to know that? All it sees is extending writes, which is what triggers speculative preallocation.
Quoted 3 lines
Here is a standalone testcase.
$ for i in {0..19}; do dd if=/dev/zero of=testfile bs=128k count=1 seek=$[$i * 12]; done
Yup, that's behaving exactly as expected there. When you seek past the existing EOF, and there is a speculative preallocation between the old EOF and the new EOF, it writes zeros to that range because the assumption is that you are going to fill it with data.
There are applications that do this to trigger that exact preallocation - Samba is a classic case because windows clients will write one byte 128k beyond the current EOF to get NTFS to trigger large preallocation, then send back and write the real data to the server. In cases like these, you want the hole allocated and filled with zeros before the real writes come in just in case the server crashes between the single byte write and the real data being written....
FWIW, XFS on Irix had special code to detect out of order NFS writes and do a similar hole filling trick to avoid fragmentation.
There's no one correct behaviour when dealing with writes of this sort. In some cases the current behaviour is perfect (and samba on XFS is widely used), in other cases it won't be exactly what you want.
Indeed, you can avoid this problem by using ftruncate() to extend the file before writing, write the regions in reverse order, using fallocate to allocate the exact blocks you want before writing, or use the allocsize mount option to turn of the dynamic behaviour.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs