Skip to contentSkip to conversation

df bigger than ls?

DISCUSSION Brian Candler 5 participants 16 messages Mar 7, 2012 – Mar 9, 2012

lore
Share & tools
lore
#0 lore raw

I have created some files spread across 12 XFS filesystems, using a glusterfs "striped" volume. This writes files with lots of holes(*) - so I would expect the space reported by 'du' to be less than the space reported by 'ls'.

However it's the other way round - du is reporting more space used than ls! Here's what I mean: I'm looking at the files directly on the underlying disk mount point, not via glusterfs at all.

$ ls -lh /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1.1G 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ ls -lk /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1059968 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -h /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -k /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2096392	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff

And here's what xfs_bmap reports:

root@storage1:~# xfs_bmap -vp /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
 EXT: FILE-OFFSET         BLOCK-RANGE            AG AG-OFFSET              TOTAL FLAGS
   0: [0..255]:           2933325744..2933325999  2 (3059152..3059407)       256 00000
   1: [256..3071]:        hole                                              2816
   2: [3072..3327]:       2933326000..2933326255  2 (3059408..3059663)       256 00000
   3: [3328..6143]:       hole                                              2816
   4: [6144..8191]:       2933326472..2933328519  2 (3059880..3061927)      2048 00000
   5: [8192..9215]:       hole                                              1024
   6: [9216..13311]:      2933369480..2933373575  2 (3102888..3106983)      4096 00000
   7: [13312..15359]:     hole                                              2048
   8: [15360..23551]:     2933375624..2933383815  2 (3109032..3117223)      8192 00000
   9: [23552..24575]:     hole                                              1024
  10: [24576..40959]:     2933587168..2933603551  2 (3320576..3336959)     16384 00000
  11: [40960..43007]:     hole                                              2048
  12: [43008..75775]:     2933623008..2933655775  2 (3356416..3389183)     32768 00000
  13: [75776..76799]:     hole                                              1024
  14: [76800..142335]:    2933656800..2933722335  2 (3390208..3455743)     65536 00000
  15: [142336..144383]:   hole                                              2048
  16: [144384..275455]:   2933724384..2933855455  2 (3457792..3588863)    131072 00000
  17: [275456..276479]:   hole                                              1024
  18: [276480..538623]:   2935019808..2935281951  2 (4753216..5015359)    262144 00000
  19: [538624..540671]:   hole                                              2048
  20: [540672..1064959]:  2935284000..2935808287  2 (5017408..5541695)    524288 00000
  21: [1064960..1065983]: hole                                              1024
  22: [1065984..2114559]: 2935809312..2936857887  2 (5542720..6591295)   1048576 00000
  23: [2114560..2116607]: hole                                              2048
  24: [2116608..2119935]: 2943037984..2943041311  2 (12771392..12774719)    3328 00000
root@storage1:~# 

Given that these values are all in 512-byte disk blocks, the total file size is (2119935 + 1) * 512 which agrees with ls. And some proportion of it is holes, so du should report less than this, shouldn't it?

(Aside: it starts off being 11/12th holes as expected, but after a while isn't. This may be a different bug, possibly in glusterfs itself)

I guess 'du' gets its info from stat(), and stat() also says the file is using 4192784 blocks which is 2096392 KB:

$ stat /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
  File: `/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff'
  Size: 1085407232	Blocks: 4192784    IO Block: 4096   regular file
Device: 810h/2064d	Inode: 8595657903  Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   brian)   Gid: ( 1000/   brian)
Access: 2012-03-07 15:20:36.044365215 +0000
Modify: 2012-03-07 15:19:33.640364277 +0000
Change: 2012-03-07 15:19:33.640364277 +0000

Finally, here is xfs_db dump of the inode:

# ls -i /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
8595657903 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
...
xfs_db> inode 8595657903
xfs_db> p
core.magic = 0x494e
core.mode = 0100664
core.version = 2
core.format = 3 (btree)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 231
core.atime.sec = Wed Mar  7 15:20:36 2012
core.atime.nsec = 044365215
core.mtime.sec = Wed Mar  7 15:19:33 2012
core.mtime.nsec = 640364277
core.ctime.sec = Wed Mar  7 15:19:33 2012
core.ctime.nsec = 640364277
core.size = 1085407232
core.nblocks = 262370
core.extsize = 0
core.nextents = 13
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1875303423
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:537294687
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,537253238,1,0]
xfs_db> 

Platform: ubuntu 11.10
Linux 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Have I missed something obvious, or is there a bug of some sort here?

Regards,

Brian.

(*) http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 2 lines · Brian Candler

core.size = 1085407232
core.nblocks = 262370

core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB

(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)

This begs the question of where stat() is getting its info from?

Ah... but I've found that after unmounting and remounting the filesystem (which I had to do for xfs_db), du and stat report the correct info.

In fact, dropping the inode caches is sufficient to fix the problem:

root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# echo 3 >/proc/sys/vm/drop_caches 
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# 

Very odd, but not really a major problem other than the confusion it causes.

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 9 lines · Brian Candler
Quoted 2 lines · Brian Candler

core.size = 1085407232
core.nblocks = 262370

core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB

(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)

Quoted 6 lines

This begs the question of where stat() is getting its info from?

Ah... but I've found that after unmounting and remounting the filesystem (which I had to do for xfs_db), du and stat report the correct info.

In fact, dropping the inode caches is sufficient to fix the problem:

Yep.

XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)

$ git log --pretty=oneline fs/xfs | grep specul
b8fc82630ae289bb4e661567808afc59e3298dce xfs: speculative delayed allocation uses rounddown_power_of_2 badly
055388a3188f56676c21e92962fc366ac8b5cb72 xfs: dynamic speculative EOF preallocation

so:

# dd if=/dev/zero of=bigfile bs=1M count=1100 &>/dev/null
# ls -lh bigfile
-rw-r--r--. 1 root root 1.1G Mar  7 11:47 bigfile
# du -h bigfile
1.1G	bigfile

but:

# rm -f bigfile
# for I in `seq 1 1100`; do dd if=/dev/zero of=bigfile conv=notrunc bs=1M seek=$I count=1 &>/dev/null; done
# ls -lh bigfile
-rw-r--r--. 1 root root 1.1G Mar  7 11:49 bigfile
# du -h bigfile
2.0G	bigfile

This should get freed when the inode is dropped from the cache; hence your cache drop bringing it back to size.

But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:

# du -hc bigfile*
2.0G	bigfile1
1.1G	bigfile2
907M	bigfile3

Dave, is this working as intended? I know the speculative preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?

-Eric

Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# echo 3 >/proc/sys/vm/drop_caches 
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# 

Very odd, but not really a major problem other than the confusion it causes.

Quoted 9 lines

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 11 lines · Eric Sandeen
Quoted 10 lines · Brian Candler
Quoted 2 lines · Brian Candler

core.size = 1085407232
core.nblocks = 262370

core.nblocks is correct here: space used = 262370 * 4 = 1049480 KB

(If I add up all the non-hole extents I get 2098944 blocks = 1049472 KB so there are two extra blocks of something)

This begs the question of where stat() is getting its info from?

stat(2) also reported delayed allocation reservations that are only kept in memory.

....

Quoted 8 lines

so:

# dd if=/dev/zero of=bigfile bs=1M count=1100 &>/dev/null
# ls -lh bigfile
-rw-r--r--. 1 root root 1.1G Mar  7 11:47 bigfile
# du -h bigfile
1.1G	bigfile
Quoted 8 lines

but:

# rm -f bigfile
# for I in `seq 1 1100`; do dd if=/dev/zero of=bigfile conv=notrunc bs=1M seek=$I count=1 &>/dev/null; done
# ls -lh bigfile
-rw-r--r--. 1 root root 1.1G Mar  7 11:49 bigfile
# du -h bigfile
2.0G	bigfile

This is tripping the NFS server write pattern heuristic. i.e. it is detecting repeated open/write at EOF/close patterns and so is not truncating away the speculative EOF reservation on close(). This si what prevents fragmentation of files being written concurrently with this pattern.

Quoted 2 lines

This should get freed when the inode is dropped from the cache; hence your cache drop bringing it back to size.

Right. It is assumes that once you've triggered that heuristic, the preallocation needs to last for as long as the inode is in the working set. The inode cache tracks the current working set, so the preallocation release is tied to cache eviction.

Quoted 9 lines

But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:

# du -hc bigfile* 2.0G	bigfile1 1.1G	bigfile2 907M
bigfile3

Dave, is this working as intended?

Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/

Quoted 4 lines

I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?

We don't track what files have current active preallocations, we only reduce the preallocation size as the filesystem nears ENOSPC. This generally works just fine in situations where the filesystem size is significantly greater than the maximum extent size. i.e. the common case

The problem you are tripping over here is that the maximum extent size is greater than the filesystem size, so the preallocation size is also greater than the filesystem size and hence can contribute significantly to premature ENOSPC. I see two possible ways to minimise this problem:

1. reduce the maximum speculative preallocation size based on filesystem size at mount time.

2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.

The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.

The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....

Cheers,

Dave.

Quoted 49 lines

-Eric

Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# echo 3 >/proc/sys/vm/drop_caches 
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# 

Very odd, but not really a major problem other than the confusion it causes.

Quoted 9 lines

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 1 line · Dave Chinner

On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:

...

Quoted 7 lines
Quoted 4 lines
# du -hc bigfile* 2.0G	bigfile1 1.1G	bigfile2 907M
bigfile3

Dave, is this working as intended?

Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/

Well, sure ;)

Quoted 11 lines
Quoted 4 lines

I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?

We don't track what files have current active preallocations, we only reduce the preallocation size as the filesystem nears ENOSPC. This generally works just fine in situations where the filesystem size is significantly greater than the maximum extent size. i.e. the common case

Quoted 5 lines

The problem you are tripping over here is that the maximum extent size is greater than the filesystem size, so the preallocation size is also greater than the filesystem size and hence can contribute significantly to premature ENOSPC. I see two possible ways to minimise this problem:

FWIW I'm not overly worried about my 4G filesystem, that was just an obviously extreme case to test the behavior.

I'd be more interested to know if the behavior was causing any issues for Brian's case, or if it was just confusing. :)

Quoted 10 lines

1. reduce the maximum speculative preallocation size based on filesystem size at mount time.

2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.

The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.

Might be worth it; even though a 4G fs is obviously not a design point, it's good not to have diabolical behavior. Although I suppose if xfstests doesn't hit it in everything it does, it's probably not a big deal. ISTR you did have to fix one thing up, though.

Quoted 7 lines

The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....

It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.

-Eric

Quoted 53 lines

Cheers,

Dave.

Quoted 49 lines

-Eric

Quoted 31 lines
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# echo 3 >/proc/sys/vm/drop_caches 
root@storage1:~# du -h /disk*/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk10/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk11/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk12/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk3/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk4/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk5/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk6/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk7/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk8/scratch2/work/PRSRA1/PRSRA1.1.0.bff
1.1G	/disk9/scratch2/work/PRSRA1/PRSRA1.1.0.bff
root@storage1:~# 

Very odd, but not really a major problem other than the confusion it causes.

Quoted 9 lines

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw
In reply to Eric Sandeen
Quoted 4 lines · Eric Sandeen

XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)

What was the default before speculative preallocation was added (or changed somewhere in 2.6.3x line afaik) ?

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#13 lore raw
In reply to Eric Sandeen
Quoted 4 lines · Eric Sandeen

XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)

Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!

Here are the extent maps for two of the twelve files in my stripe:

root@storage1:~# xfs_bmap /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff 
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
	0: [0..255]: 2933325744..2933325999
	1: [256..3071]: hole
	2: [3072..3327]: 2933326000..2933326255
	3: [3328..6143]: hole
	4: [6144..8191]: 2933326472..2933328519
	5: [8192..9215]: hole
	6: [9216..13311]: 2933369480..2933373575
	7: [13312..15359]: hole
	8: [15360..23551]: 2933375624..2933383815
	9: [23552..24575]: hole
	10: [24576..40959]: 2933587168..2933603551
	11: [40960..43007]: hole
	12: [43008..75775]: 2933623008..2933655775
	13: [75776..76799]: hole
	14: [76800..142335]: 2933656800..2933722335
	15: [142336..144383]: hole
	16: [144384..275455]: 2933724384..2933855455
	17: [275456..276479]: hole
	18: [276480..538623]: 2935019808..2935281951
	19: [538624..540671]: hole
	20: [540672..1064959]: 2935284000..2935808287
	21: [1064960..1065983]: hole
	22: [1065984..2114559]: 2935809312..2936857887
	23: [2114560..2116607]: hole
	24: [2116608..2119935]: 2943037984..2943041311
root@storage1:~# xfs_bmap /disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff 
/disk2/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
	0: [0..255]: hole
	1: [256..511]: 2933194944..2933195199
	2: [512..3327]: hole
	3: [3328..3839]: 2933195200..2933195711
	4: [3840..6399]: hole
	5: [6400..8447]: 2933204416..2933206463
	6: [8448..9471]: hole
	7: [9472..13567]: 2933328792..2933332887
	8: [13568..15615]: hole
	9: [15616..23807]: 2933334936..2933343127
	10: [23808..24831]: hole
	11: [24832..41215]: 2933344152..2933360535
	12: [41216..43263]: hole
	13: [43264..76031]: 2934672032..2934704799
	14: [76032..77055]: hole
	15: [77056..142591]: 2934705824..2934771359
	16: [142592..144639]: hole
	17: [144640..275711]: 2934773408..2934904479
	18: [275712..276735]: hole
	19: [276736..538879]: 2934343328..2934605471
	20: [538880..540927]: hole
	21: [540928..1065215]: 2935498152..2936022439
	22: [1065216..1066239]: hole
	23: [1066240..2114815]: 2936023464..2937072039
	24: [2114816..2116863]: hole
	25: [2116864..2120191]: 2937074088..2937077415

You can see that at the start it works fine. There is a stripe size of 256 blocks, so:

* disk 1: data for 1 x 256 blocks <-- stripe 0, chunk 0
hole for 11 x 256 blocks
data for 1 x 256 block <-- stripe 0, chunk 1
...

* disk 2: hole for 1 x 256 blocks
data for 1 x 256 blocks <-- stripe 1, chunk 0
hole for 11 x 256 blocks
data for 1 x 256 blocks <-- stripe 1, chunk 1
...

But after four chunks it gets screwed up. By the end the files are mostly extent and hardly any hole. The extent sizes increase in roughly powers of two which seems to match the speculative preallocation algorithm.

I think this ought to be fixable. For example, if you seek *into* the preallocated area and start writing, could you change the preallocation to start at this location with a hole before?

(But would that mess up some 'seeky' workloads like databases? But they would have ended up creating holes in filesystems which don't have preallocation, so I doubt they do this)

Or for a more sledgehammer approach: if a file already contains any holes then you could just disable preallocation completely.

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
In reply to Eric Sandeen
Quoted 4 lines · Eric Sandeen

It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.

And presumably df on the filesystem would also show it approaching 100% utilisation?

I'm used to this where a large file has been unlinked but is still open. The preallocation case is a new one to me though.

How about if the total of all preallocations were limited to some small percentage of the total filesystem size? If you reach this limit and want to preallocate some space for another file you'd have to either drop or shrink an older preallocation.

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 8 lines · Brian Candler
Quoted 4 lines · Eric Sandeen

It seems worth thinking about. I guess I'm still a little concerned about the ENOSPC case; it could lead to some confusion - I could imagine several hundreds of gigs under preallocation, with a reasonable-sized filesystem returning ENOSPC quite early.

And presumably df on the filesystem would also show it approaching 100% utilisation?

*nod*

Quoted 7 lines

I'm used to this where a large file has been unlinked but is still open. The preallocation case is a new one to me though.

How about if the total of all preallocations were limited to some small percentage of the total filesystem size? If you reach this limit and want to preallocate some space for another file you'd have to either drop or shrink an older preallocation.

There is no separate accounting for preallocation - it is considered used space so this currently can't be done even if there was some method for tracking and trimming speculatively preallocated space.

Realistically, if you aren't running out of space there is no reason to limit speculative preallocation. Indeed, if we didn't add delalloc blocks to the block count in stat(2) output so they showed up in df, almost no-one would even know about the fact that this is done....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw
In reply to Brian Candler
Quoted 3 lines · Brian Candler

Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!

Here is a standalone testcase.

$ for i in {0..19}; do dd if=/dev/zero of=testfile bs=128k count=1 seek=$[$i * 12]; done
$ xfs_bmap testfile
testfile:
	0: [0..255]: 1465133392..1465133647
	1: [256..3071]: hole
	2: [3072..5119]: 1465136464..1465138511
	3: [5120..6143]: hole
	4: [6144..10239]: 1465139536..1465143631
	5: [10240..12287]: hole
	6: [12288..20479]: 1465145680..1465153871
	7: [20480..21503]: hole
	8: [21504..37887]: 1465154896..1465171279
	9: [37888..39935]: hole
	10: [39936..58623]: 1465173328..1465192015

I expected to see: 20 extents of 256 blocks and 19 holes of 2816 blocks.

Regards,

Brian.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
In reply to Arkadiusz Miśkiewicz
Quoted 9 lines · Arkadiusz Miśkiewicz
Quoted 4 lines · Eric Sandeen

XFS speculatively preallocates space off the end of a file. The amount of space allocated depends on the present size of the file, and the amount of available free space. This can be overridden
with mount -o allocsize=64k (or other size for example)

What was the default before speculative preallocation was added (or changed somewhere in 2.6.3x line afaik) ?

It's changed many times over the life of XFS.

Originally it was quite a large number and could happen anywhere in the file - I can't remember exactly what it was but some time around 1997 on Irix it got brought back to 16 blocks per written region because of issues with Irix NFS servers keeping tens of thousands of files open (and hence never having speculative preallocation removed) with multiple preallocations all through sparse files and hence running filesystems and user quotas out of space.

This 16 block size was the default when ported to linux when porting to Linux, but would also round up to stripe unit size if the file was larger than 512kb. In 2004 the speculative preallocation was reduced to EOF preallocation only instead of anywhere in a hole in a file. Somewhere along the line the default size got changed to PAGE_SIZE rather than 16 blocks. Then the allocsize mount option was introduced in 2005 to enable it to be increased for those that needed large preallocation (typically PVR users), and the maximum increased to 1GB.

It got broken by changes in 2008, fixed in 2009, and then made dynamic in 2.6.38 and now it behaves more like it did in 1995 for files being extended.

So there's a long history of changing the speculative preallocation behaviour of XFS to adapt to changing storage characteristics, workloads and capacities.....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#15 lore raw
In reply to Brian Candler
Quoted 4 lines · Brian Candler
Quoted 3 lines · Brian Candler

Aha. This may well be what is screwing up gluster's disk usage on a striped volume - I believe XFS is preallocating space which is actually going to end up being a hole!

How is the filesystem supposed to know that? All it sees is extending writes, which is what triggers speculative preallocation.

Quoted 3 lines

Here is a standalone testcase.

$ for i in {0..19}; do dd if=/dev/zero of=testfile bs=128k count=1 seek=$[$i * 12]; done

Yup, that's behaving exactly as expected there. When you seek past the existing EOF, and there is a speculative preallocation between the old EOF and the new EOF, it writes zeros to that range because the assumption is that you are going to fill it with data.

There are applications that do this to trigger that exact preallocation - Samba is a classic case because windows clients will write one byte 128k beyond the current EOF to get NTFS to trigger large preallocation, then send back and write the real data to the server. In cases like these, you want the hole allocated and filled with zeros before the real writes come in just in case the server crashes between the single byte write and the real data being written....

FWIW, XFS on Irix had special code to detect out of order NFS writes and do a similar hole filling trick to avoid fragmentation.

There's no one correct behaviour when dealing with writes of this sort. In some cases the current behaviour is perfect (and samba on XFS is widely used), in other cases it won't be exactly what you want.

Indeed, you can avoid this problem by using ftruncate() to extend the file before writing, write the regions in reverse order, using fallocate to allocate the exact blocks you want before writing, or use the allocsize mount option to turn of the dynamic behaviour.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
In reply to Dave Chinner
Quoted 6 lines · Dave Chinner
Quoted 5 lines · Eric Sandeen

But there does seem to be an issue here; if I make a 4G filesystem and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:

Quoted 7 lines
Quoted 4 lines
# du -hc bigfile* 2.0G	bigfile1 1.1G	bigfile2 907M
bigfile3

Dave, is this working as intended?

Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/

I have seen a similar problem on some very large filesystems too. This is not just dependent upon the size of the filesystem, but also the workload. I think it is also a big problem for folks using quotas.

Quoted 20 lines
Quoted 4 lines

I know the speculative
preallocation amount for new files is supposed to go down as the fs fills, but is there no way to discard prealloc space to avoid ENOSPC on other files?

I see two possible ways to
minimise this problem:

1. reduce the maximum speculative preallocation size based on filesystem size at mount time.

2. track inodes with active speculative preallocation and have an enospc based trigger that can find them and truncate away excess idle speculative preallocation.

The first is relatively easy to do, but will only reduce the incidence of your problem - we still need to allow significant preallocation sizes (e.g. 64MB) to avoid the fragmentation problems.

Quoted 7 lines

The second is needed to reclaim the space we've already preallocated but is not being used. That's more complex to do - probably a radix tree bit and a periodic background scan to reduce the time window the preallocation sits around from cache lifetime to "idle for some time" along with a on-demand, synchronous ENOSPC scan. This will need some more thought as to how to do it effectively, but isn't impossible to do....

Alex and I discussed this problem briefly awhile ago. What is the best way to lose when you hit ENOSPC (project quotas) or EDQUOT in xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.

I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)

Now that block reservations count toward quotas the symptom will probably be a little different.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
Quoted 9 lines · Ben Myers
Quoted 8 lines · Dave Chinner
Quoted 7 lines · Eric Sandeen

and repeat the above test 3 times, the 3rd run gets ENOSPC, and the last file written comes up short, while the first one retains all it's extra preallocated space:

# du -hc bigfile* 2.0G	bigfile1 1.1G	bigfile2 907M
bigfile3
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Dave, is this working as intended?

Yes. Your problem is that you have a very small filesystem, which is not the case that we optimise XFS for. :/

I have seen a similar problem on some very large filesystems too.

So report the problems to the list, don't keep them to yourself. If we don't hear about problems, then as far as we are concerned they don't exist.

Quoted 3 lines

This
is not just dependent upon the size of the filesystem, but also the workload. I think it is also a big problem for folks using quotas.

Nobody has reported significant quota problems, either....

Quoted 6 lines

Alex and I discussed this problem briefly awhile ago. What is the best way to lose when you hit ENOSPC (project quotas) or EDQUOT in xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.

Quoted 6 lines

I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)

And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).

Quoted 2 lines

Now that block reservations count toward quotas the symptom will probably be a little different.

Block reservations have always counted towards quotas, it's just that they were never reported.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw
Quoted 16 lines · Dave Chinner
Quoted 2 lines · Ben Myers
Quoted 1 line · Dave Chinner

On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:

Alex and I discussed this problem briefly awhile ago. What is the best

Quoted 11 lines

way to lose when you hit ENOSPC (project quotas) or EDQUOT in xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.

I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)

Quoted 4 lines

And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).

Yeah, maybe not. A single list of inodes with speculative prealloc does seem a good place to start. Later maybe you would not want to scan/filter through all of them and we could add additional lists for dquots. My suggestion of using LRU was because I think that the oldest unused speculative preallocs should be the first to go. One chronic over-quota user shouldn't be able to punish everyone else.

Quoted 5 lines
Quoted 2 lines

Now that block reservations count toward quotas the symptom will probably be a little different.

Block reservations have always counted towards quotas, it's just that they were never reported.

Aw. My mistake. ;)

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
Quoted 17 lines · Ben Myers
Quoted 16 lines · Dave Chinner
Quoted 2 lines · Ben Myers
Quoted 1 line · Dave Chinner

On Wed, Mar 07, 2012 at 12:04:26PM -0600, Eric Sandeen wrote:

Alex and I discussed this problem briefly awhile ago. What is the best

Quoted 11 lines

way to lose when you hit ENOSPC (project quotas) or EDQUOT in xfs_iomap_write_delay? You want to be fair; one user hitting his quota shouldn't be able to steal some other user's block reservations unless you really are near ENOSPC for the entire filesystem.

I suggested something like... track inodes with preallocated block reservations in LRU order and by dquot, so that the poor fella who is at EDQUOT will first clean up the preallocations that resulted in quota being enforced, try again, and then work on preallocations of other users only if it can help in his situation. IIRC Alex shut me down when he heard LRU. ;)

Quoted 7 lines
Quoted 4 lines

And I agree with Alex. Nothing additional needs to be tracked on top of inodes with speculative prealloc. Just the search filter would need to be different (i.e. only select inodes with a specific dquot attached).

Yeah, maybe not. A single list of inodes with speculative prealloc does seem a good place to start.

Except it involves growing the inode by a struct listhead. Keeping per-inode memory usage down is extremely important, and growing it by 16 bytes for a rare corner case is not a particularly good tradeoff. Especially as using a radix tree tag for tracking doesn't involve any increase in memory usage and tag based scans are lock-free and quite efficient.

Quoted 5 lines

Later maybe you would not want to
scan/filter through all of them and we could add additional lists for dquots. My suggestion of using LRU was because I think that the oldest unused speculative preallocs should be the first to go. One chronic over-quota user shouldn't be able to punish everyone else.

Sure. But we only need to check if the inode with prealloc belongs to the user/group/project that hit EDQUOT before taking action. We don't need an LRU for that.

Indeed, I don't believe that removing the entire preallocation on each inode is the right thing to do, either. Chopping it in half is probably the right thing to do so that we free up lots of space in the case of excessive (large preallocation) but don't cause significant extra fragmentation for the work that currently requires it.

If we still get EDQUOT after trimming speculative preallocations, we can try a second time. If reducing speculative preallocations by 75% doesn't avoid the EDQUOT (or ENOSPC), then I think we can consider it a real error....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs