qemu-img convert -n: Keep qcow2 v2 target sparse
cover qemu-img convert -n: Keep qcow2 v2 target sparse
Kevin Wolf (2):
qcow2: Implement v2 zero writes with discard if possible
iotests: Test sparseness for qemu-img convert -n
block/qcow2-cluster.c | 9 ++++++++- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 3 files changed, 59 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.4
PATCH 1/2 qcow2: Implement v2 zero writes with discard if possible
qcow2 version 2 images don't support the zero flag for clusters, so for write_zeroes requests, we return -ENOTSUP and get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.
This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
block/qcow2-cluster.c +8 -1
@@ -1797,8 +1797,15 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,assert (QEMU_IS_ALIGNED (end_offset, s->cluster_size) || end_offset >= bs->total_sectors <<BDRV_SECTOR_BITS ); -/* The zero flag is only supported by version 3 and newer */ +/* +* The zero flag is only supported by version 3 and newer. However, if we +* have no backing file, we can resort to discard in version 2. +*/ if (s->qcow_version <3 ) { +if (!bs->backing) { +return qcow2_cluster_discard (bs, offset, bytes, +QCOW2_DISCARD_REQUEST , false); + }return -ENOTSUP ; }
Signature · 1 line
2.25.4
Re: [PATCH for-5.1 1/2] qcow2: Implement v2 zero writes with discard if possible
Quoted 27 lines · Kevin Wolf
qcow2 version 2 images don't support the zero flag for clusters, so for
write_zeroesrequests, we return-ENOTSUPand get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
Reported-by Nir Soffer <[email protected]>Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)block/qcow2-cluster.c +1
-1@@ -1797,8 +1797,15 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,assert (QEMU_IS_ALIGNED (end_offset, s->cluster_size) || end_offset >= bs->total_sectors <<BDRV_SECTOR_BITS ); -/* The zero flag is only supported by version 3 and newer */ +/*
block/qcow2-cluster.c int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset, · line 1810
Quoted 10 lines
block/qcow2-cluster.c +7
-0+* The zero flag is only supported by version 3 and newer. However, if we +* have no backing file, we can resort to discard in version 2. +*/ if (s->qcow_version <3 ) { +if (!bs->backing) { +return qcow2_cluster_discard (bs, offset, bytes, +QCOW2_DISCARD_REQUEST , false); + }return -ENOTSUP ; }
Looks good to me.
Quoted 4 lines
block/qcow2-cluster.c +0
-1--2.25.4
Re: [PATCH for-5.1 1/2] qcow2: Implement v2 zero writes with discard if possible
In reply to Kevin WolfQuoted 10 lines · Kevin Wolf
qcow2 version 2 images don't support the zero flag for clusters, so for
write_zeroesrequests, we return-ENOTSUPand get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
Quoted 5 lines
Reported-by Nir Soffer <[email protected]>Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
Re: [PATCH for-5.1 1/2] qcow2: Implement v2 zero writes with discard if possible
In reply to Kevin WolfQuoted 27 lines · Kevin Wolf
qcow2 version 2 images don't support the zero flag for clusters, so for
write_zeroesrequests, we return-ENOTSUPand get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
Reported-by Nir Soffer <[email protected]>Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)block/qcow2-cluster.c +2
-1@@ -1797,8 +1797,15 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,assert (QEMU_IS_ALIGNED (end_offset, s->cluster_size) || end_offset >= bs->total_sectors <<BDRV_SECTOR_BITS ); -/* The zero flag is only supported by version 3 and newer */ +/* +* The zero flag is only supported by version 3 and newer. However, if we
block/qcow2-cluster.c int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset, · line 1811
Quoted 10 lines
block/qcow2-cluster.c +6
-0+* have no backing file, we can resort to discard in version 2. +*/ if (s->qcow_version <3 ) { +if (!bs->backing) { +return qcow2_cluster_discard (bs, offset, bytes, +QCOW2_DISCARD_REQUEST , false); + }return -ENOTSUP ; }
From my knowelege of nvme, I remember that discard doesn't have to zero the blocks. There is special namespace capability the indicates the contents of the discarded block. (Deallocate Logical Block Features)
If and only if the discard behavier flag indicates that discarded areas are zero, then the write-zero command can have special 'deallocate' flag that hints the controller to discard the sectors.
So woudn't discarding the clusters have theoretical risk of introducing garbage there?
Best regards,
Maxim Levitsky
Re: [PATCH for-5.1 1/2] qcow2: Implement v2 zero writes with discard if possible
Am 22.07.2020 um 19:01 hat Maxim Levitsky geschrieben:
Quoted 43 lines · Kevin Wolf
Quoted 27 lines
qcow2 version 2 images don't support the zero flag for clusters, so for
write_zeroesrequests, we return-ENOTSUPand get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
Reported-by Nir Soffer <[email protected]>Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)block/qcow2-cluster.c +2
-1@@ -1797,8 +1797,15 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,assert (QEMU_IS_ALIGNED (end_offset, s->cluster_size) || end_offset >= bs->total_sectors <<BDRV_SECTOR_BITS ); -/* The zero flag is only supported by version 3 and newer */ +/* +* The zero flag is only supported by version 3 and newer. However, if we block/qcow2-cluster.c int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset, · line 1811
Quoted 10 lines
block/qcow2-cluster.c +6
-0+* have no backing file, we can resort to discard in version 2. +*/ if (s->qcow_version <3 ) { +if (!bs->backing) { +return qcow2_cluster_discard (bs, offset, bytes, +QCOW2_DISCARD_REQUEST , false); + }return -ENOTSUP ; }From my knowelege of nvme, I remember that discard doesn't have to zero the blocks. There is special namespace capability the indicates the contents of the discarded block. (Deallocate Logical Block Features)
Quoted 5 lines
If and only if the discard behavier flag indicates that discarded areas are zero, then the write-zero command can have special 'deallocate' flag that hints the controller to discard the sectors.
So woudn't discarding the clusters have theoretical risk of introducing garbage there?
No, qcow2_cluster_discard() has a defined behaviour. For v2 images, it unallocates the cluster in the L2 table (this is only safe without a backing file), for v3 images it converts them to zero clusters.
Kevin
Re: [PATCH for-5.1 1/2] qcow2: Implement v2 zero writes with discard if possible
Quoted 48 lines · Kevin Wolf
Am 22.07.2020 um 19:01 hat Maxim Levitsky geschrieben:
Quoted 47 lines · Kevin Wolf
Quoted 27 lines
qcow2 version 2 images don't support the zero flag for clusters, so for
write_zeroesrequests, we return-ENOTSUPand get explicit zero buffer writes. If the image doesn't have a backing file, we can do better: Just discard the respective clusters.This is relevant for 'qemu-img convert -O qcow2 -n', where qemu-img has to assume that the existing target image may contain any data, so it has to write zeroes. Without this patch, this results in a fully allocated target image, even if the source image was empty.
Reported-by Nir Soffer <[email protected]>Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cluster.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)block/qcow2-cluster.c +2
-1@@ -1797,8 +1797,15 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,assert (QEMU_IS_ALIGNED (end_offset, s->cluster_size) || end_offset >= bs->total_sectors <<BDRV_SECTOR_BITS ); -/* The zero flag is only supported by version 3 and newer */ +/* +* The zero flag is only supported by version 3 and newer. However, if we block/qcow2-cluster.c int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset, · line 1811
Quoted 10 lines
block/qcow2-cluster.c +6
-0+* have no backing file, we can resort to discard in version 2. +*/ if (s->qcow_version <3 ) { +if (!bs->backing) { +return qcow2_cluster_discard (bs, offset, bytes, +QCOW2_DISCARD_REQUEST , false); + }return -ENOTSUP ; }From my knowelege of nvme, I remember that discard doesn't have to zero the blocks. There is special namespace capability the indicates the contents of the discarded block. (Deallocate Logical Block Features)
If and only if the discard behavier flag indicates that discarded areas are zero, then the write-zero command can have special 'deallocate' flag that hints the controller to discard the sectors.
Quoted 5 lines
Quoted 1 line
So woudn't discarding the clusters have theoretical risk of introducing garbage there?
No,
qcow2_cluster_discard()has a defined behaviour. For v2 images, it unallocates the cluster in the L2 table (this is only safe without a backing file), for v3 images it converts them to zero clusters.
All right then!
Best regards,
Maxim Levitsky
Quoted 2 lines
Kevin
PATCH 2/2 iotests: Test sparseness for qemu-img convert -n
In reply to Kevin Wolf--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)
tests/qemu-iotests/122 +34 -0
@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig +echo +echo '=== -n to an empty image ===' +echo + +_make_test_img 64M + +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +echo +echo '=== -n to an empty image with a backing file ===' +echo + +_make_test_img 64M +TEST_IMG="$TEST_IMG".base _make_test_img 64M + +# Convert with -n, which should still not result in a fully allocated image for +# compat=1.1 (because it can use zero clusters), but it should be fully +# allocated with compat=0.10 +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + echo echo '=== -n -B to an image without a backing file ===' echo
tests/qemu-iotests/122.out +17 -0
@@ -229,6 +229,23 @@ wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Images are identical. +=== -n to an empty image === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] + +=== -n to an empty image with a backing file === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": false, "data": true, "offset": 327680}] + === -n -B to an image without a backing file === Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864
Signature · 1 line
2.25.4
Re: [PATCH for-5.1 2/2] iotests: Test sparseness for qemu-img convert -n
Quoted 12 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)tests/qemu-iotests/122 +0
-0
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 288
Quoted 8 lines
tests/qemu-iotests/122 +5
-0@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig +echo +echo '=== -n to an empty image ===' +echo + +_make_test_img 64M
Why make a test image here? We create it again below twice
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 294
Quoted 6 lines
tests/qemu-iotests/122 +6
-0+ +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig
This looks reversed - "$TEST_IMG".orig is the original image, and "$TEST_IMG" is the target image. So maybe use "$TEST_IMG".target?
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 298
Quoted 4 lines
tests/qemu-iotests/122 +4
-0+ +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig
Since the only difference is the compat, why not use a loop?
for compat in 0.10 1.1; do
...
Quoted 4 lines
tests/qemu-iotests/122 +4
-0+ +echo +echo '=== -n to an empty image with a backing file ===' +echo
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 312
Quoted 10 lines
tests/qemu-iotests/122 +10
-0+ +_make_test_img 64M +TEST_IMG="$TEST_IMG".base _make_test_img 64M + +# Convert with -n, which should still not result in a fully allocated image for +# compat=1.1 (because it can use zero clusters), but it should be fully +# allocated with compat=0.10 +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig
Do we have a real use case for this convert? Doesn't this hide all the data in the
backing file by data from source?
Assuming source is:
src-top: A0--
dst-bas: --B0
And destination is:
dst-top: ----
dst-bas: CCCC
After the convert we will have:
dst-top: A0B0
dst-bas: CCCC
So entire backing of dst is hidden.
Nir
Quoted 38 lines
tests/qemu-iotests/122 +4
-0+TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + echo echo '=== -n -B to an image without a backing file ===' echotests/qemu-iotests/122.out +17
-1@@ -229,6 +229,23 @@ wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Images are identical. +=== -n to an empty image === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] + +=== -n to an empty image with a backing file === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": false, "data": true, "offset": 327680}] + === -n -B to an image without a backing file === Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 --2.25.4
Re: [PATCH for-5.1 2/2] iotests: Test sparseness for qemu-img convert -n
Am 20.07.2020 um 16:47 hat Nir Soffer geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 15 lines
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)tests/qemu-iotests/122 +0
-0@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig
Quoted 7 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 288
Quoted 5 lines
tests/qemu-iotests/122 +5
-0+echo +echo '=== -n to an empty image ===' +echo + +_make_test_img 64MWhy make a test image here? We create it again below twice
This is a different image because the invocations below change the TEST_IMG variable.
Quoted 9 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 294
Quoted 6 lines
tests/qemu-iotests/122 +6
-0+ +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".origThis looks reversed - "$TEST_IMG".orig is the original image, and "$TEST_IMG" is the target image. So maybe use "$TEST_IMG".target?
I'll use .orig for the source and without a suffix for the target (which are filenames that _cleanup_test_img covers automatically).
Quoted 9 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 298
Quoted 4 lines
tests/qemu-iotests/122 +4
-0+ +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".origSince the only difference is the compat, why not use a loop?
for compat in 0.10 1.1; do
...
Makes sense.
Quoted 9 lines
Quoted 9 lines
tests/qemu-iotests/122 +9
-0+ +echo +echo '=== -n to an empty image with a backing file ===' +echo + +_make_test_img 64M +TEST_IMG="$TEST_IMG".base _make_test_img 64M + +# Convert with -n, which should still not result in a fully allocated image for
Quoted 8 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 312
Quoted 5 lines
tests/qemu-iotests/122 +5
-0+# compat=1.1 (because it can use zero clusters), but it should be fully +# allocated with compat=0.10 +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".origDo we have a real use case for this convert? Doesn't this hide all the data in the backing file by data from source?
There is probably no real use case for this. But it has a defined behaviour and it's always good to cover corner cases with tests so that unintentional changes can be found (which may potentially affect more relevant cases, too).
Kevin
Re: [PATCH for-5.1 2/2] iotests: Test sparseness for qemu-img convert -n
In reply to Kevin WolfQuoted 17 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)tests/qemu-iotests/122 +3
-0@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig +echo +echo '=== -n to an empty image ===' +echo
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 296
Quoted 10 lines
tests/qemu-iotests/122 +10
-0+ +_make_test_img 64M + +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64M
It’s a shame that with this, the test will no longer pass with refcount_bits=1. (Or an external data file.)
But, well. Maybe we don’t care and then should just put both options into _unsupported_imgopts.
Apart from that, the test looks good to me.
Max
Quoted 53 lines
tests/qemu-iotests/122 +21
-0+$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +echo +echo '=== -n to an empty image with a backing file ===' +echo + +_make_test_img 64M +TEST_IMG="$TEST_IMG".base _make_test_img 64M + +# Convert with -n, which should still not result in a fully allocated image for +# compat=1.1 (because it can use zero clusters), but it should be fully +# allocated with compat=0.10 +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -b "$TEST_IMG".base -F $IMGFMT -o compat=0.10 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + echo echo '=== -n -B to an image without a backing file ===' echotests/qemu-iotests/122.out +17
-0@@ -229,6 +229,23 @@ wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Images are identical. +=== -n to an empty image === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] + +=== -n to an empty image with a backing file === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +[{ "start": 0, "length": 67108864, "depth": 0, "zero": false, "data": true, "offset": 327680}] + === -n -B to an image without a backing file === Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864
Re: [PATCH for-5.1 2/2] iotests: Test sparseness for qemu-img convert -n
Am 21.07.2020 um 12:19 hat Max Reitz geschrieben:
Quoted 15 lines · Kevin Wolf
Quoted 14 lines
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)tests/qemu-iotests/122 +0
-0@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig
Quoted 16 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 296
Quoted 13 lines
tests/qemu-iotests/122 +13
-0+echo +echo '=== -n to an empty image ===' +echo + +_make_test_img 64M + +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64MIt’s a shame that with this, the test will no longer pass with refcount_bits=1. (Or an external data file.)
You mean because of the compat=0.10? We already use that in this test case, however just with "$QEMU_IMG convert" so that $IMGOPTS doesn't apply.
I guess I could just override $IMGOPTS for this line to get the same behaviour here and make sure that none of these options are used.
Kevin
Re: [PATCH for-5.1 2/2] iotests: Test sparseness for qemu-img convert -n
Quoted 33 lines · Kevin Wolf
Am 21.07.2020 um 12:19 hat Max Reitz geschrieben:
Quoted 15 lines · Kevin Wolf
Quoted 14 lines
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 17 +++++++++++++++++ 2 files changed, 51 insertions(+)tests/qemu-iotests/122 +0
-0@@ -281,6 +281,40 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".origQuoted 16 lines
tests/qemu-iotests/122 $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig · line 296
Quoted 13 lines
tests/qemu-iotests/122 +13
-0+echo +echo '=== -n to an empty image ===' +echo + +_make_test_img 64M + +# Convert with -n, which should not result in a fully allocated image, not even +# with compat=0.10 (because the target doesn't have a backing file) +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=1.1 64M +$QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map --output=json "$TEST_IMG".orig + +TEST_IMG="$TEST_IMG".orig _make_test_img -o compat=0.10 64MIt’s a shame that with this, the test will no longer pass with refcount_bits=1. (Or an external data file.)
Quoted 6 lines
You mean because of the compat=0.10? We already use that in this test case, however just with "$QEMU_IMG convert" so that $IMGOPTS doesn't apply.
I guess I could just override $IMGOPTS for this line to get the same behaviour here and make sure that none of these options are used.
Well... Not my favorite, but probably because I just never thought of that.
I suppose it works, so why not, actually.
Max