Skip to contentSkip to conversation

iotests: Fix 125

PATCH In review Max Reitz 2 participants 9 messages Sep 25, 2019 – Sep 27, 2019

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

cover iotests: Fix 125

Hi,

iotest 125 is very broken. It uses qemu-img info’s “disk size” to determine an image’s on-disk size, but it does so in a wrong way: It just fetches the first number ([0-9]+), but that isn’t very useful because qemu-img info emits human-readable values that include units and decimal points.

We should ust stat -c %b instead. That’s done in patch 3. Unfortunately, doing so exposed more problems.

Patch 1 fixes a stupid bug in the test itself that we never noticed because of what patch 3 fixes. (Pull patch 3 before patch 1 and you’ll see.)

The other thing is actually a bug in XFS. Its fallocate() implementation rounds up the length independently of the offset, so if you try to fallocate an unaligned range, chances are that it might not allocate the last block your range touches. Patch 2 detects that case and skips the test then. (Pull patch 3 before patch 2 and you’ll see the test fail on XFS.)

Max Reitz (3):
iotests: Fix 125 for growth_mode = metadata
iotests: Disable 125 on broken XFS versions
iotests: Use stat -c %b in 125

 tests/qemu-iotests/125 | 45 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/3 iotests: Fix 125 for growth_mode = metadata

If we use growth_mode = metadata, it is very much possible that the file uses more disk space after we have written something to the added area. We did indeed want to test for this case, but unfortunately we evidently just copied the code from the "Test creation preallocation" section and forgot to replace "$create_mode" by "$growth_mode".

We never noticed because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the patch after the next one.

---
 tests/qemu-iotests/125 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/125 +1 -1
index dc4b8f5fb9..df328a63a6 100755
@@ -111,7 +111,7 @@ for GROWTH_SIZE in 16 48 80; do
                 if [ $file_length_2 -gt $file_length_1 ]; then
                     echo "ERROR (grow): Image length has grown from $file_length_1 to $file_length_2"
                 fi
-                if [ $create_mode != metadata ]; then
+                if [ $growth_mode != metadata ]; then
                     # The host size should not have grown either
                     if [ $host_size_2 -gt $host_size_1 ]; then
                         echo "ERROR (grow): Host size has grown from $host_size_1 to $host_size_2"
Signature · 1 line
2.21.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] iotests: Fix 125 for growth_mode = metadata

Quoted 11 lines · Max Reitz

If we use growth_mode = metadata, it is very much possible that the file uses more disk space after we have written something to the added area. We did indeed want to test for this case, but unfortunately we evidently just copied the code from the "Test creation preallocation" section and forgot to replace "$create_mode" by "$growth_mode".

We never noticed because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the patch after the next one.

Quoted 5 lines
---
  tests/qemu-iotests/125 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
tests/qemu-iotests/125 +1 -1
index dc4b8f5fb9..df328a63a6 100755
@@ -111,7 +111,7 @@ for GROWTH_SIZE in 16 48 80; do
                  if [ $file_length_2 -gt $file_length_1 ]; then
                      echo "ERROR (grow): Image length has grown from $file_length_1 to $file_length_2"
                  fi
-                if [ $create_mode != metadata ]; then
+                if [ $growth_mode != metadata ]; then
                      # The host size should not have grown either
                      if [ $host_size_2 -gt $host_size_1 ]; then
                          echo "ERROR (grow): Host size has grown from $host_size_1 to $host_size_2"
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#3 lore raw

PATCH 2/3 iotests: Disable 125 on broken XFS versions

In reply to Max Reitz

And by that I mean all XFS versions, as far as I can tell. All details are in the comment below.

We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the next patch.

---
 tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
tests/qemu-iotests/125 +40 -0
index df328a63a6..0ef51f1e21 100755
@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; then
     TEST_IMG_FILE=$TEST_IMG
 fi
 
+# Test whether we are running on a broken XFS version.  There is this
+# bug:
+
+# $ rm -f foo
+# $ touch foo
+# $ block_size=4096 # Your FS's block size
+# $ fallocate -o $((block_size / 2)) -l $block_size foo
+# $ LANG=C xfs_bmap foo | grep hole
+#         1: [8..15]: hole
+#
+# The problem is that the XFS driver rounds down the offset and
+# rounds up the length to the block size, but independently.  As
+# such, it only allocates the first block in the example above,
+# even though it should allocate the first two blocks (because our
+# request is to fallocate something that touches both the first
+# two blocks).
+#
+# This means that when you then write to the beginning of the
+# second block, the disk usage of the first two blocks grows.
+#
+# That is precisely what fallocate() promises, though: That when you
+# write to an area that you have fallocated, no new blocks will have
+# to be allocated.
+
+touch "$TEST_IMG_FILE"
+# Assuming there is no FS with a block size greater than 64k
+fallocate -o 65535 -l 2 "$TEST_IMG_FILE"
+len0=$(get_image_size_on_host)
+
+# Write to something that in theory we have just fallocated
+# (Thus, the on-disk size should not increase)
+poke_file "$TEST_IMG_FILE" 65536 42
+len1=$(get_image_size_on_host)
+
+if [ $len1 -gt $len0 ]; then
+    _notrun "the test filesystem's fallocate() is broken"
+fi
+
+rm -f "$TEST_IMG_FILE"
+
 # Generally, we create some image with or without existing preallocation and
 # then resize it. Then we write some data into the image and verify that its
 # size does not change if we have used preallocation.
Signature · 1 line
2.21.0
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] iotests: Disable 125 on broken XFS versions

Quoted 23 lines · Max Reitz

And by that I mean all XFS versions, as far as I can tell. All details are in the comment below.

We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the next patch.

---
  tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 40 insertions(+)
tests/qemu-iotests/125 +2 -0
index df328a63a6..0ef51f1e21 100755
@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; then
      TEST_IMG_FILE=$TEST_IMG
  fi
 
+# Test whether we are running on a broken XFS version.  There is this
+# bug:

tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then · line 63

Quoted 10 lines
tests/qemu-iotests/125 +10 -0
+
+# $ rm -f foo
+# $ touch foo
+# $ block_size=4096 # Your FS's block size
+# $ fallocate -o $((block_size / 2)) -l $block_size foo
+# $ LANG=C xfs_bmap foo | grep hole
+#         1: [8..15]: hole
+#
+# The problem is that the XFS driver rounds down the offset and
+# rounds up the length to the block size, but independently.

Eww. I concur you uncovered a bug. Have you reported this to xfs folks?

Quoted 6 lines
tests/qemu-iotests/125 +6 -0
+
+touch "$TEST_IMG_FILE"
+# Assuming there is no FS with a block size greater than 64k
+fallocate -o 65535 -l 2 "$TEST_IMG_FILE"
+len0=$(get_image_size_on_host)
+

tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then · line 90

Quoted 10 lines
tests/qemu-iotests/125 +10 -0
+# Write to something that in theory we have just fallocated
+# (Thus, the on-disk size should not increase)
+poke_file "$TEST_IMG_FILE" 65536 42
+len1=$(get_image_size_on_host)
+
+if [ $len1 -gt $len0 ]; then
+    _notrun "the test filesystem's fallocate() is broken"
+fi
+
+rm -f "$TEST_IMG_FILE"
Quoted 5 lines
tests/qemu-iotests/125 +1 -0
+
  # Generally, we create some image with or without existing preallocation and
  # then resize it. Then we write some data into the image and verify that its
  # size does not change if we have used preallocation.
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#5 lore raw

Re: [PATCH 2/3] iotests: Disable 125 on broken XFS versions

Quoted 27 lines · Eric Blake
Quoted 26 lines · Max Reitz

And by that I mean all XFS versions, as far as I can tell.  All details are in the comment below.

We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point).  That will be fixed in the next patch.

---
  tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++

  1 file changed, 40 insertions(+)
tests/qemu-iotests/125 +0 -0
index df328a63a6..0ef51f1e21 100755
@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; then

      TEST_IMG_FILE=$TEST_IMG
  fi
  +# Test whether we are running on a broken XFS version.  There is this

tests/qemu-iotests/125 +5 -0
+# bug:
+
+# $ rm -f foo
+# $ touch foo
+# $ block_size=4096 # Your FS's block size
Quoted 8 lines

tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then

Quoted 6 lines
tests/qemu-iotests/125 +6 -0
+# $ fallocate -o $((block_size / 2)) -l $block_size foo
+# $ LANG=C xfs_bmap foo | grep hole
+#         1: [8..15]: hole
+#
+# The problem is that the XFS driver rounds down the offset and
+# rounds up the length to the block size, but independently.

Eww. I concur you uncovered a bug.  Have you reported this to xfs folks?

I have now. Took a bit of kernel compiling to see whether what I think would fix it works.

Max

author
#6 lore raw

PATCH 3/3 iotests: Use stat -c %b in 125

In reply to Max Reitz

125 should not use qemu-img to get the on-disk image size, because that reports it in a human-readable format that is useless to us. Just use stat instead (like we do to get the image file length).

---
 tests/qemu-iotests/125 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/125 +1 -2
index 0ef51f1e21..4e31aa4e5f 100755
@@ -34,8 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 get_image_size_on_host()
 {
-    $QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "disk size" \
-        | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/'
+    echo $(($(stat -c '%b * %B' "$TEST_IMG_FILE")))
 }
 
 # get standard environment and filters
Signature · 1 line
2.21.0
Reviewed-by on v1 3/3
#7 lore raw

Re: [PATCH 3/3] iotests: Use stat -c %b in 125

Quoted 14 lines · Max Reitz

125 should not use qemu-img to get the on-disk image size, because that reports it in a human-readable format that is useless to us. Just use stat instead (like we do to get the image file length).

---
  tests/qemu-iotests/125 | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/125 +0 -0
index 0ef51f1e21..4e31aa4e5f 100755

tests/qemu-iotests/125 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 37

Quoted 6 lines
tests/qemu-iotests/125 +1 -2
@@ -34,8 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
  get_image_size_on_host()
  {
-    $QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "disk size" \
-        | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/'
+    echo $(($(stat -c '%b * %B' "$TEST_IMG_FILE")))

Cute use of $(()) around $().

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#8 lore raw
In reply to Max Reitz
Quoted 23 lines · Max Reitz

Hi,

iotest 125 is very broken. It uses qemu-img info’s “disk size” to determine an image’s on-disk size, but it does so in a wrong way: It just fetches the first number ([0-9]+), but that isn’t very useful because qemu-img info emits human-readable values that include units and decimal points.

We should ust stat -c %b instead. That’s done in patch 3. Unfortunately, doing so exposed more problems.

Patch 1 fixes a stupid bug in the test itself that we never noticed because of what patch 3 fixes. (Pull patch 3 before patch 1 and you’ll see.)

The other thing is actually a bug in XFS. Its fallocate() implementation rounds up the length independently of the offset, so if you try to fallocate an unaligned range, chances are that it might not allocate the last block your range touches. Patch 2 detects that case and skips the test then. (Pull patch 3 before patch 2 and you’ll see the test fail on XFS.)

Quoted 7 lines

Max Reitz (3):
iotests: Fix 125 for growth_mode = metadata
iotests: Disable 125 on broken XFS versions
iotests: Use stat -c %b in 125

 tests/qemu-iotests/125 | 45 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)

Thanks for the review, applied to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block

And for the record, I’ve sent a patch to the XFS driver:

https://www.spinics.net/lists/linux-xfs/msg32174.html

Max