Strange hole creation behavior

So this coreutils test is failing on XFS:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=blob;f=tests/dd/sparse.sh;h=06efc7017
Specifically the last hole check on line 66.

In summary what's happening is that a write(1MiB), lseek(1MiB), write(1MiB) creates only a 64KiB hole. Is that expected?

Now a 1MiB hole is supported using truncate:
dd if=/dev/urandom of=file.in bs=1M count=1 iflag=fullblock
truncate -s+1M file.in
dd if=/dev/urandom of=file.in bs=1M count=1 iflag=fullblock conv=notrunc oflag=append

  $ du -k file.in
  2048  file.in

But when trying to create the 1MiB hole with dd (lseek) it fails?

# Create 3MiB input file file

  $ dd if=/dev/urandom of=file.in bs=1M count=3 iflag=fullblock
  $ dd if=/dev/zero    of=file.in bs=1M count=1 seek=1 conv=notrunc
  $ du -k file.in
  3072  file.in

# Convert to 1MiB hole doesn't work :(

  $ dd if=file.in of=file.out bs=1M conv=sparse
  $ du -k file.out
  3008  file.out

# Again with syscall details:

  $ strace -e write,lseek dd if=file.in of=file.out bs=1M conv=sparse
  write(1, "...", 1048576) = 1048576
  lseek(1, 1048576, SEEK_CUR)             = 2097152
  write(1, "...", 1048576) = 1048576

So it seems that the lseeks are treated differently to the truncate that was done in the first example, which is surprising. If we look at the file layout we can see the hole is only at the last 64KiB of the middle 1MiB of zeros, rather than for the whole middle 1MiB as in the first example??

  $ filefrag -v file.out
  Filesystem type is: 58465342
  File size of file.out is 3145728 (768 blocks of 4096 bytes)
   ext:     logical_offset:        physical_offset: length:   expected: flags:
     0:        0..     495:      31271..     31766:    496:
     1:      512..     767:      31783..     32038:    256:      31767: eof

thanks,
Pádraig.

Versions etc. in case useful

$ uname -a
Linux tp2 3.12.6-300.fc20.x86_64 #1 SMP Mon Dec 23 16:44:31 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ xfs_info .
meta-data=/dev/loop2             isize=256    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

This is expected behavior due to speculative preallocation. An FAQ with regard to this behavior is pending, but see here for reference:

http://oss.sgi.com/archives/xfs/2014-04/msg00083.html

In that particular write(1MB), lseek(+1MB), write(1MB) workload, each write is preallocating some extra space beyond the current EOF. The seek then moves past that space, but the space doesn't go away. The subsequent writes will extend EOF. The previously preallocated space now resides in the middle of the file and can't be trimmed away when the file is closed.

This works simply because it is broken into multiple commands. When the first dd exits, the excess space is trimmed off (the file descriptor is closed). The subsequent truncate extends the file size without any extra space getting caught between the old and new EOF.

You can confirm this by using the 'allocsize=4k' mount option to the XFS mount. If you wanted something more generic for the purpose of testing the coreutils functionality, you could also set the size of file.out in advance. E.g., with preallocation in effect:

# dd if=file.in of=file.out bs=1M conv=sparse
# xfs_bmap -v file.out 
file.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..3967]:       9773944..9777911  1 (9080..13047)     3968
   1: [3968..4095]:    hole                                   128
   2: [4096..6143]:    9778040..9780087  1 (13176..15223)    2048

... and then prevent preallocation by ensuring writes do not extend the file:

# rm -f file.out 
# truncate --size=3M file.out
# dd if=file.in of=file.out bs=1M conv=sparse,notrunc
# xfs_bmap -v file.out 
file.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..2047]:       9773944..9775991  1 (9080..11127)     2048
   1: [2048..4095]:    hole                                  2048
   2: [4096..6143]:    9778040..9780087  1 (13176..15223)    2048

Hope that helps.

Brian

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Excellent info thanks.
With that I can adjust the test so it passes (patch attached).

So for reference this means that cp can no longer recreate holes <= 1MiB from source to dest (with the default XFS allocation size):

$ cp --sparse=always file.in cp.out
$ xfs_bmap -v !$
xfs_bmap -v cp.out
cp.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..3967]:       219104..223071    0 (219104..223071)  3968
   1: [3968..4095]:    hole                                   128
   2: [4096..6143]:    225720..227767    0 (225720..227767)  2048

$ xfs_bmap -v file.out
file.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..2047]:       229816..231863    0 (229816..231863)  2048
   1: [2048..4095]:    hole                                  2048
   2: [4096..6143]:    233912..235959    0 (233912..235959)  2048

$ cp file.out cp.out
$ xfs_bmap -v cp.out
cp.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..3967]:       250296..254263    0 (250296..254263)  3968
   1: [3968..4095]:    hole                                   128
   2: [4096..6143]:    254392..256439    0 (254392..256439)  2048

Though if we bump up the hole size the representation is better:

$ dd if=/dev/urandom of=bigfile.in bs=1M count=1 iflag=fullblock
$ truncate -s+10M bigfile.in
$ dd if=/dev/urandom of=bigfile.in bs=1M count=1 iflag=fullblock conv=notrunc oflag=append

$ xfs_bmap -v bigfile.in
bigfile.in:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..2047]:       231864..233911    0 (231864..233911)  2048
   1: [2048..22527]:   hole                                 20480
   2: [22528..24575]:  256440..258487    0 (256440..258487)  2048

$ cp bigfile.in bigfile.out
$ xfs_bmap -v bigfile.out
bigfile.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..3967]:       260408..264375    0 (260408..264375)  3968
   1: [3968..22527]:   hole                                 18560
   2: [22528..24575]:  264376..266423    0 (264376..266423)  2048

We could I suppose use FALLOC_FL_PUNCH_HOLE where available to cater for this case. I'll see whether this is worth adding. That can be used after the fact anyway:

$ fallocate --dig-holes bigfile.out
$ xfs_bmap -v bigfile.out
bigfile.out:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL
   0: [0..2047]:       260408..262455    0 (260408..262455)  2048
   1: [2048..22527]:   hole                                 20480
   2: [22528..24575]:  264376..266423    0 (264376..266423)  2048

thanks,
Pádraig.

From: =?UTF-8?q?P=C3=A1draig=20Brady?= <[email protected]>
Date: Fri, 11 Apr 2014 23:44:13 +0100
Subject: [PATCH] tests: fix false dd conv=sparse failure on newer XFS

* tests/dd/sparse.sh: When testing that a hole is created, use an existing sparse destination file, so that we're not write extending the file size, and thus avoiding speculative preallocation which can result in smaller holes than requested.
Workaround suggested by Brian Foster

---
 THANKS.in          |    1 +
 tests/dd/sparse.sh |    9 ++++++++-
 2 files changed, 9 insertions(+), 1 deletions(-)

index e7298ef..a92540a 100644
@@ -95,6 +95,7 @@ Bjorn Helgaas                       [email protected]
 Bob McCracken                       [email protected]
 Branden Robinson                    [email protected]
 Brendan O'Dea                       [email protected]
+Brian Foster                        [email protected]
 Brian Kimball                       [email protected]
 Brian M. Carlson                    [email protected]
 Brian Silverman                     [email protected]

index 06efc70..a7e90d2 100755
@@ -61,8 +61,15 @@ if test $(kb_alloc file.in) -gt 3000; then
   dd if=file.in of=file.out bs=2M conv=sparse
   test 2500 -lt $(kb_alloc file.out) || fail=1
 
+  # Note we recreate a sparse file first to avoid
+  # speculative preallocation seen in XFS, where a write() that
+  # extends a file can preallocate some extra space that
+  # a subsequent seek will not convert to a hole.
+  rm -f file.out
+  truncate --size=3M file.out
+
   # Ensure that this 1MiB string of NULs *is* converted to a hole.
-  dd if=file.in of=file.out bs=1M conv=sparse
+  dd if=file.in of=file.out bs=1M conv=sparse,notrunc
   test $(kb_alloc file.out) -lt 2500 || fail=1
 
 fi

1.7.7.6

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Well, the allocation size changes based on the filesize; there's a heuristic involved. So I fear that if you hard-code it into your tests, you risk failing again in the future...

That might make sense.

But filesystems get to pick their layout; even ext4 may opportunistically fill in holes, etc - so I think you need to be pretty careful with these sorts of tests...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs