iotests: Fix 125
cover iotests: Fix 125
Hi,
iotest 125 is very broken. It uses qemu-img info’s “disk size” to determine an image’s on-disk size, but it does so in a wrong way: It just fetches the first number ([0-9]+), but that isn’t very useful because qemu-img info emits human-readable values that include units and decimal points.
We should ust stat -c %b instead. That’s done in patch 3. Unfortunately, doing so exposed more problems.
Patch 1 fixes a stupid bug in the test itself that we never noticed because of what patch 3 fixes. (Pull patch 3 before patch 1 and you’ll see.)
The other thing is actually a bug in XFS. Its fallocate() implementation rounds up the length independently of the offset, so if you try to fallocate an unaligned range, chances are that it might not allocate the last block your range touches. Patch 2 detects that case and skips the test then. (Pull patch 3 before patch 2 and you’ll see the test fail on XFS.)
Max Reitz (3):
iotests: Fix 125 for growth_mode = metadata
iotests: Disable 125 on broken XFS versions
iotests: Use stat -c %b in 125
tests/qemu-iotests/125 | 45 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/3 iotests: Fix 125 for growth_mode = metadata
If we use growth_mode = metadata, it is very much possible that the file uses more disk space after we have written something to the added area. We did indeed want to test for this case, but unfortunately we evidently just copied the code from the "Test creation preallocation" section and forgot to replace "$create_mode" by "$growth_mode".
We never noticed because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the patch after the next one.
--- tests/qemu-iotests/125 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/125 +1 -1
@@ -111,7 +111,7 @@ for GROWTH_SIZE in 16 48 80; do if [ $file_length_2 -gt $file_length_1 ]; then echo "ERROR (grow): Image length has grown from $file_length_1 to $file_length_2" fi - if [ $create_mode != metadata ]; then + if [ $growth_mode != metadata ]; then # The host size should not have grown either if [ $host_size_2 -gt $host_size_1 ]; then echo "ERROR (grow): Host size has grown from $host_size_1 to $host_size_2"
Signature · 1 line
2.21.0
Re: [PATCH 1/3] iotests: Fix 125 for growth_mode = metadata
Quoted 11 lines · Max Reitz
If we use
growth_mode= metadata, it is very much possible that the file uses more disk space after we have written something to the added area. We did indeed want to test for this case, but unfortunately we evidently just copied the code from the "Test creation preallocation" section and forgot to replace "$create_mode" by "$growth_mode".We never noticed because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the patch after the next one.
Quoted 5 lines
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/125 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
tests/qemu-iotests/125 +1
-1@@ -111,7 +111,7 @@ for GROWTH_SIZE in 16 48 80; do if [ $file_length_2 -gt $file_length_1 ]; then echo "ERROR (grow): Image length has grown from $file_length_1 to $file_length_2" fi - if [ $create_mode != metadata ]; then + if [ $growth_mode != metadata ]; then # The host size should not have grown either if [ $host_size_2 -gt $host_size_1 ]; then echo "ERROR (grow): Host size has grown from $host_size_1 to $host_size_2"
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 2/3 iotests: Disable 125 on broken XFS versions
In reply to Max ReitzAnd by that I mean all XFS versions, as far as I can tell. All details are in the comment below.
We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the next patch.
--- tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
tests/qemu-iotests/125 +40 -0
@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; then TEST_IMG_FILE=$TEST_IMG fi +# Test whether we are running on a broken XFS version. There is this +# bug: + +# $ rm -f foo +# $ touch foo +# $ block_size=4096 # Your FS's block size +# $ fallocate -o $((block_size / 2)) -l $block_size foo +# $ LANG=C xfs_bmap foo | grep hole +# 1: [8..15]: hole +# +# The problem is that the XFS driver rounds down the offset and +# rounds up the length to the block size, but independently. As +# such, it only allocates the first block in the example above, +# even though it should allocate the first two blocks (because our +# request is to fallocate something that touches both the first +# two blocks). +# +# This means that when you then write to the beginning of the +# second block, the disk usage of the first two blocks grows. +# +# That is precisely what fallocate() promises, though: That when you +# write to an area that you have fallocated, no new blocks will have +# to be allocated. + +touch "$TEST_IMG_FILE" +# Assuming there is no FS with a block size greater than 64k +fallocate -o 65535 -l 2 "$TEST_IMG_FILE" +len0=$(get_image_size_on_host) + +# Write to something that in theory we have just fallocated +# (Thus, the on-disk size should not increase) +poke_file "$TEST_IMG_FILE" 65536 42 +len1=$(get_image_size_on_host) + +if [ $len1 -gt $len0 ]; then + _notrun "the test filesystem's fallocate() is broken" +fi + +rm -f "$TEST_IMG_FILE" + # Generally, we create some image with or without existing preallocation and # then resize it. Then we write some data into the image and verify that its # size does not change if we have used preallocation.
Signature · 1 line
2.21.0
Re: [PATCH 2/3] iotests: Disable 125 on broken XFS versions
Quoted 23 lines · Max Reitz
And by that I mean all XFS versions, as far as I can tell. All details are in the comment below.
We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the next patch.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)tests/qemu-iotests/125 +2
-0@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; then TEST_IMG_FILE=$TEST_IMG fi +# Test whether we are running on a broken XFS version. There is this +# bug:
tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then · line 63
Quoted 10 lines
tests/qemu-iotests/125 +10
-0+ +# $ rm -f foo +# $ touch foo +# $ block_size=4096 # Your FS's block size +# $ fallocate -o $((block_size / 2)) -l $block_size foo +# $ LANG=C xfs_bmap foo | grep hole +# 1: [8..15]: hole +# +# The problem is that the XFS driver rounds down the offset and +# rounds up the length to the block size, but independently.
Eww. I concur you uncovered a bug. Have you reported this to xfs folks?
Quoted 6 lines
tests/qemu-iotests/125 +6
-0+ +touch "$TEST_IMG_FILE" +# Assuming there is no FS with a block size greater than 64k +fallocate -o 65535 -l 2 "$TEST_IMG_FILE" +len0=$(get_image_size_on_host) +
tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then · line 90
Quoted 10 lines
tests/qemu-iotests/125 +10
-0+# Write to something that in theory we have just fallocated +# (Thus, the on-disk size should not increase) +poke_file "$TEST_IMG_FILE" 65536 42 +len1=$(get_image_size_on_host) + +if [ $len1 -gt $len0 ]; then + _notrun "the test filesystem's fallocate() is broken" +fi + +rm -f "$TEST_IMG_FILE"
Quoted 5 lines
tests/qemu-iotests/125 +1
-0+ # Generally, we create some image with or without existing preallocation and # then resize it. Then we write some data into the image and verify that its # size does not change if we have used preallocation.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/3] iotests: Disable 125 on broken XFS versions
Quoted 27 lines · Eric Blake
Quoted 26 lines · Max Reitz
And by that I mean all XFS versions, as far as I can tell. All details are in the comment below.
We never noticed this problem because we only read the first number from qemu-img info's "disk size" output -- and that is effectively useless, because qemu-img prints a human-readable value (which generally includes a decimal point). That will be fixed in the next patch.
Signed-off-by Max Reitz <[email protected]>---
tests/qemu-iotests/125 | 40 ++++++++++++++++++++++++++++++++++++++++1 file changed, 40 insertions(+)tests/qemu-iotests/125 +0
-0@@ -49,6 +49,46 @@ if [ -z "$TEST_IMG_FILE" ]; thenTEST_IMG_FILE=$TEST_IMG
fi
+# Test whether we are running on a broken XFS version. There is thistests/qemu-iotests/125 +5
-0+# bug: + +# $ rm -f foo +# $ touch foo +# $ block_size=4096 # Your FS's block size
Quoted 8 lines
tests/qemu-iotests/125 if [ -z "$TEST_IMG_FILE" ]; then
Quoted 6 lines
tests/qemu-iotests/125 +6
-0+# $ fallocate -o $((block_size / 2)) -l $block_size foo +# $ LANG=C xfs_bmap foo | grep hole +# 1: [8..15]: hole +# +# The problem is that the XFS driver rounds down the offset and +# rounds up the length to the block size, but independently.Eww. I concur you uncovered a bug. Have you reported this to xfs folks?
I have now. Took a bit of kernel compiling to see whether what I think would fix it works.
Max
PATCH 3/3 iotests: Use stat -c %b in 125
In reply to Max Reitz125 should not use qemu-img to get the on-disk image size, because that reports it in a human-readable format that is useless to us. Just use stat instead (like we do to get the image file length).
--- tests/qemu-iotests/125 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/125 +1 -2
@@ -34,8 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 get_image_size_on_host() { - $QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "disk size" \ - | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/' + echo $(($(stat -c '%b * %B' "$TEST_IMG_FILE"))) } # get standard environment and filters
Signature · 1 line
2.21.0
Re: [PATCH 3/3] iotests: Use stat -c %b in 125
Quoted 14 lines · Max Reitz
125 should not use qemu-img to get the on-disk image size, because that reports it in a human-readable format that is useless to us. Just use stat instead (like we do to get the image file length).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/125 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/125 +0
-0
tests/qemu-iotests/125 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 37
Quoted 6 lines
tests/qemu-iotests/125 +1
-2@@ -34,8 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 get_image_size_on_host() { - $QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "disk size" \ - | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/' + echo $(($(stat -c '%b * %B' "$TEST_IMG_FILE")))
Cute use of $(()) around $().
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Quoted 23 lines · Max Reitz
Hi,
iotest 125 is very broken. It uses qemu-img info’s “disk size” to determine an image’s on-disk size, but it does so in a wrong way: It just fetches the first number ([0-9]+), but that isn’t very useful because qemu-img info emits human-readable values that include units and decimal points.
We should ust stat -c %b instead. That’s done in patch 3. Unfortunately, doing so exposed more problems.
Patch 1 fixes a stupid bug in the test itself that we never noticed because of what patch 3 fixes. (Pull patch 3 before patch 1 and you’ll see.)
The other thing is actually a bug in XFS. Its
fallocate()implementation rounds up the length independently of the offset, so if you try to fallocate an unaligned range, chances are that it might not allocate the last block your range touches. Patch 2 detects that case and skips the test then. (Pull patch 3 before patch 2 and you’ll see the test fail on XFS.)
Quoted 7 lines
Max Reitz (3):
iotests: Fix 125 forgrowth_mode= metadata
iotests: Disable 125 on broken XFS versions
iotests: Use stat -c %b in 125tests/qemu-iotests/125 | 45 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-)
Thanks for the review, applied to my block branch:
https://git.xanclic.moe/XanClic/qemu/commits/branch/block
And for the record, I’ve sent a patch to the XFS driver:
https://www.spinics.net/lists/linux-xfs/msg32174.html
Max