Fix convert to qcow2 compressed to NBD
cover Fix convert to qcow2 compressed to NBD
Fix qemu-img convert -O qcow2 -c to NBD URL and add missing test for this usage.
This already works now, but unfortunately qemu-img fails when trying to truncate the target image to the same size at the end of the operation.
Nir Soffer (2):
block: nbd: Fix convert qcow2 compressed to nbd
qemu-iotests: Test convert to qcow2 compressed to NBD
block/nbd.c | 27 +++++++++++++ tests/qemu-iotests/302 | 83 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 27 +++++++++++++ tests/qemu-iotests/group | 1 + 4 files changed, 138 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.out
Signature · 1 line
2.25.4
PATCH 1/2 block: nbd: Fix convert qcow2 compressed to nbd
When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement bdrv_co_truncate().
For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.
Example failing without this change:
In one shell starts qemu-nbd:
$ truncate -s 1g test.tar $ qemu-nbd --socket=/tmp/nbd.sock --persistent --format=raw --offset 1536 test.tar
In another shell convert an image to qcow2 compressed via NBD:
$ echo "disk data" > disk.raw $ truncate -s 1g disk.raw $ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=/tmp/nbd.sock; echo $? 1
qemu-img failed, but the conversion was successful:
$ qemu-img info nbd+unix:///?socket=/tmp/nbd.sock image: nbd+unix://?socket=/tmp/nbd.sock file format: qcow2 virtual size: 1 GiB (1073741824 bytes) ...
$ qemu-img check nbd+unix:///?socket=/tmp/nbd.sock No errors were found on the image. 1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters Image end offset: 393216
$ qemu-img compare disk.raw nbd+unix:///?socket=/tmp/nbd.sock Images are identical.
--- block/nbd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
block/nbd.c +27 -0
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)nbd_clear_bdrvstate (s); } +/* +* NBD cannot truncate, but if the caller ask to truncate to the same size, or +* to a smaller size with extact=false, there is not reason to fail the +* operation. +*/ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, +bool exact,PreallocMode prealloc, +BdrvRequestFlags flags,Error **errp) +{ +BDRVNBDState *s = bs->opaque; + +if (offset != s->info.size && exact) { +error_setg (errp,"Cannot resize NBD nodes" ); +return -ENOTSUP ; + } + +if (offset > s->info.size) { +error_setg (errp,"Cannot grow NBD nodes" ); +return -EINVAL ; + } + +return 0 ; +} +static int64_t nbd_getlength (BlockDriverState *bs) {BDRVNBDState *s = bs->opaque;
@@ -2045,6 +2069,7 @@ static BlockDriver bdrv_nbd = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context,
@@ -2072,6 +2097,7 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context,
@@ -2099,6 +2125,7 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context,
Signature · 1 line
2.25.4
Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd
Quoted 67 lines · Nir Soffer
When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to
bdrv_co_truncate(). This call always fail for the nbd driver since it does not implementbdrv_co_truncate().For block devices, which have the same limits, the call succeeds since file driver implements
bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.Example failing without this change:
In one shell starts qemu-nbd:
$ truncate -s 1g test.tar $ qemu-nbd --socket=/tmp/nbd.sock --persistent --format=raw --offset 1536 test.tarIn another shell convert an image to qcow2 compressed via NBD:
$ echo "disk data" > disk.raw $ truncate -s 1g disk.raw $ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=/tmp/nbd.sock; echo $? 1qemu-img failed, but the conversion was successful:
$ qemu-img info nbd+unix:///?socket=/tmp/nbd.sock image: nbd+unix://?socket=/tmp/nbd.sock file format: qcow2 virtual size: 1 GiB (1073741824 bytes) ...$ qemu-img check nbd+unix:///?socket=/tmp/nbd.sock No errors were found on the image. 1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters Image end offset: 393216$ qemu-img compare disk.raw nbd+unix:///?socket=/tmp/nbd.sock Images are identical.Signed-off-by Nir Soffer <[email protected]>--- block/nbd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)block/nbd.c +13
-0@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)nbd_clear_bdrvstate (s); } +/* +* NBD cannot truncate, but if the caller ask to truncate to the same size, or +* to a smaller size with extact=false, there is not reason to fail the +* operation. +*/ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, +bool exact,PreallocMode prealloc, +BdrvRequestFlags flags,Error **errp) +{ +BDRVNBDState *s = bs->opaque; + +if (offset != s->info.size && exact) { +error_setg (errp,"Cannot resize NBD nodes" );
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1991
Quoted 10 lines
block/nbd.c +10
-0+return -ENOTSUP ; + } + +if (offset > s->info.size) { +error_setg (errp,"Cannot grow NBD nodes" ); +return -EINVAL ; + } + +return 0 ; +}
Can we also check @prealloc, like iscsi.c does it?
(Yes, preallocation only makes sense for growing, but you can for example truncate a 2 GB NBD node to 1 GB, and then grow it to 1.5 GB, and for the latter operation it might make sense to give a preallocation parameter. (Except block_resize doesn’t allow that, but, well.))
Max
Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd
In reply to Nir SofferQuoted 3 lines · Nir Soffer
When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to
bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement
fails
Quoted 1 line
bdrv_co_truncate().
Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.
Quoted 21 lines
For block devices, which have the same limits, the call succeeds since file driver implements
bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.Example failing without this change:
Signed-off-by Nir Soffer <[email protected]>--- block/nbd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)block/nbd.c +0
-0
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970
Quoted 5 lines
block/nbd.c +2
-0@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)nbd_clear_bdrvstate (s); } +/* +* NBD cannot truncate, but if the caller ask to truncate to the same size, or
asks
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971
Quoted 1 line
block/nbd.c +1
-0+* to a smaller size with extact=false, there is not reason to fail the
exact, no
Quoted 9 lines
block/nbd.c +9
-0+* operation. +*/ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, +bool exact,PreallocMode prealloc, +BdrvRequestFlags flags,Error **errp) +{ +BDRVNBDState *s = bs->opaque; + +if (offset != s->info.size && exact) {
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990
Quoted 10 lines
block/nbd.c +10
-0+error_setg (errp,"Cannot resize NBD nodes" ); +return -ENOTSUP ; + } + +if (offset > s->info.size) { +error_setg (errp,"Cannot grow NBD nodes" ); +return -EINVAL ; + } + +return 0 ;
Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since s->info.size is constant, but the BDS size is not if inexact resize succeeds).
As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd
Quoted 6 lines · Eric Blake
Quoted 3 lines · Nir Soffer
When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to
bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement
Quoted 6 lines
fails
Quoted 1 line
bdrv_co_truncate().Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.
I started with ignoring ENOTSUP in qcow2, but felt less safe about this approach since the same issue may happen in other flows, and making nbd driver behave like a block device looks like a safer change.
Quoted 58 lines
Quoted 20 lines
For block devices, which have the same limits, the call succeeds since file driver implements
bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.Example failing without this change:
Signed-off-by Nir Soffer <[email protected]>--- block/nbd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)block/nbd.c +0
-0block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970
Quoted 5 lines
block/nbd.c +2
-0@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)nbd_clear_bdrvstate (s); } +/* +* NBD cannot truncate, but if the caller ask to truncate to the same size, or asks
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971
Quoted 1 line
block/nbd.c +1
-0+* to a smaller size with extact=false, there is not reason to fail the exact, no
Quoted 9 lines
block/nbd.c +9
-0+* operation. +*/ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, +bool exact,PreallocMode prealloc, +BdrvRequestFlags flags,Error **errp) +{ +BDRVNBDState *s = bs->opaque; + +if (offset != s->info.size && exact) {block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990
Quoted 10 lines
block/nbd.c +10
-0+error_setg (errp,"Cannot resize NBD nodes" ); +return -ENOTSUP ; + } + +if (offset > s->info.size) { +error_setg (errp,"Cannot grow NBD nodes" ); +return -EINVAL ; + } + +return 0 ;Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since
s->info.sizeis constant, but the BDS size is not if inexact resize succeeds).
Quoted 2 lines
As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.
I'll post v2 with the test fixes later today.
Quoted 6 lines
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd
In reply to Eric BlakeQuoted 63 lines · Eric Blake
Quoted 3 lines · Nir Soffer
When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to
bdrv_co_truncate(). This call always fail for the nbd driver since it does not implementfails
Quoted 1 line
bdrv_co_truncate().Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.
Quoted 21 lines
For block devices, which have the same limits, the call succeeds since file driver implements
bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.Example failing without this change:
Signed-off-by Nir Soffer <[email protected]>--- block/nbd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)block/nbd.c +0
-0block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970
Quoted 5 lines
block/nbd.c +2
-0@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)nbd_clear_bdrvstate (s); } +/* +* NBD cannot truncate, but if the caller ask to truncate to the same size, or asks
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971
Quoted 1 line
block/nbd.c +1
-0+* to a smaller size with extact=false, there is not reason to fail the exact, no
Quoted 17 lines
block/nbd.c +17
-0+* operation. +*/ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, +bool exact,PreallocMode prealloc, +BdrvRequestFlags flags,Error **errp) +{ +BDRVNBDState *s = bs->opaque; + +if (offset != s->info.size && exact) { +error_setg (errp,"Cannot resize NBD nodes" ); +return -ENOTSUP ; + } + +if (offset > s->info.size) { +error_setg (errp,"Cannot grow NBD nodes" ); +return -EINVAL ; + }
Quoted 8 lines
block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990
Quoted 2 lines
block/nbd.c +2
-0+ +return 0 ;Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since
s->info.sizeis constant, but the BDS size is not if inexact resize succeeds).
Do we want to fail if someone specifies -o preallocation={falloc,full}?
I see we convert DRV_REQ_MAY_UNMAP to NBD_CMD_FLAG_NO_HOLE
so using -o preallocation=falloc,full should be correct. But the last request zero
length write request does not do anything, so failing does not look useful.
Quoted 8 lines
As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
PATCH 2/2 qemu-iotests: Test convert to qcow2 compressed to NBD
In reply to Nir SofferAdd test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.
--- tests/qemu-iotests/302 | 83 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 27 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 111 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.out
tests/qemu-iotests/302 new file +83 -0
@@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +# +# Tests conveting qcow2 compressed to NBD +# +# Copyright (c) 2020 Nir Soffer <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +import json +import iotests + +from iotests import ( + file_path, + qemu_img, + qemu_img_create, + qemu_img_log, + qemu_img_pipe, + qemu_io, + qemu_nbd, +) + +iotests.script_initialize(supported_fmts=["qcow2"]) + +# Create source disk, format does not matter. +src_disk = file_path("disk.img") +qemu_img_create("-f", "raw", src_disk, "10m") +qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk) + +# The use case is writing qcow2 image directly into a tar file. Code to create +# real tar file not included. +# +# offset content +# ------------------------------- +# 0 first memebr header +# 512 first member data +# 1024 second memeber header +# 1536 second member data + +tar_file = file_path("test.tar") +out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk) +measure = json.loads(out) +qemu_img_create("-f", "raw", tar_file, str(measure["required"])) + +nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) +nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + +# Use raw format to allow creating qcow2 directy into tar file. +qemu_nbd( + "--socket", nbd_sock, + "--persistent", + "--export-name", "exp", + "--format", "raw", + "--offset", "1536", + tar_file) + +iotests.log("=== Target image info ===") +qemu_img_log("info", nbd_uri) + +# Write image into the tar file. In a real applicatio we would write a tar +# entry after writing the image. +qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri) + +iotests.log("=== Converted image info ===") +qemu_img_log("info", nbd_uri) + +iotests.log("=== Converted image check ===") +qemu_img_log("check", nbd_uri) + +iotests.log("=== Comparing to source disk ===") +qemu_img_log("compare", src_disk, nbd_uri)
tests/qemu-iotests/302.out new file +27 -0
@@ -0,0 +1,27 @@ +=== Target image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: raw +virtual size: 446 KiB (457216 bytes) +disk size: unavailable + +=== Converted image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: qcow2 +virtual size: 10 MiB (10485760 bytes) +disk size: unavailable +cluster_size: 65536 +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== Converted image check === +No errors were found on the image. +1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216 + +=== Comparing to source disk === +Images are identical. +
tests/qemu-iotests/group +1 -0
@@ -308,3 +308,4 @@ 297 meta 299 auto quick 301 backing quick +302 quick
Signature · 1 line
2.25.4
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
Quoted 18 lines · Nir Soffer
Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.
Signed-off-by Nir Soffer <[email protected]>--- tests/qemu-iotests/302 | 83 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 27 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 111 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.outtests/qemu-iotests/302 new file +0
-0
tests/qemu-iotests/302 line 3
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0@@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +# +# Tests conveting qcow2 compressed to NBD
*converting
Quoted 26 lines
tests/qemu-iotests/302 new file +26
-0+# +# Copyright (c) 2020 Nir Soffer <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +import json +import iotests + +from iotests import ( + file_path, + qemu_img, + qemu_img_create, + qemu_img_log,
tests/qemu-iotests/302 line 39
Quoted 10 lines
tests/qemu-iotests/302 new file +10
-0+ qemu_img_pipe, + qemu_io, + qemu_nbd, +) + +iotests.script_initialize(supported_fmts=["qcow2"]) + +# Create source disk, format does not matter. +src_disk = file_path("disk.img") +qemu_img_create("-f", "raw", src_disk, "10m")
If the format doesn’t matter, why not just use qcow2 and so put iotests.imgfmt here? (And everywhere else where you now have -f raw.)
tests/qemu-iotests/302 line 40
Quoted 1 line
tests/qemu-iotests/302 new file +1
-0+qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk)
(Except I think qemu_io already has -f qcow2 in its arguments by default, so specifying the format wouldn’t even be necessary here.)
tests/qemu-iotests/302 line 47
Quoted 6 lines
tests/qemu-iotests/302 new file +6
-0+# The use case is writing qcow2 image directly into a tar file. Code to create +# real tar file not included. +# +# offset content +# ------------------------------- +# 0 first memebr header
*member
tests/qemu-iotests/302 line 49
Quoted 2 lines
tests/qemu-iotests/302 new file +2
-0+# 512 first member data +# 1024 second memeber header
*member
tests/qemu-iotests/302 line 55
Quoted 6 lines
tests/qemu-iotests/302 new file +6
-0+# 1536 second member data + +tar_file = file_path("test.tar") +out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk) +measure = json.loads(out) +qemu_img_create("-f", "raw", tar_file, str(measure["required"]))
Should this be measure["required"] + 1536?
Quoted 7 lines
tests/qemu-iotests/302 new file +7
-0+ +nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) +nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + +# Use raw format to allow creating qcow2 directy into tar file. +qemu_nbd( + "--socket", nbd_sock,
tests/qemu-iotests/302 line 72
Quoted 10 lines
tests/qemu-iotests/302 new file +10
-0+ "--persistent", + "--export-name", "exp", + "--format", "raw", + "--offset", "1536", + tar_file) + +iotests.log("=== Target image info ===") +qemu_img_log("info", nbd_uri) + +# Write image into the tar file. In a real applicatio we would write a tar
*application
Quoted 30 lines
tests/qemu-iotests/302 new file +11
-0+# entry after writing the image. +qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri) + +iotests.log("=== Converted image info ===") +qemu_img_log("info", nbd_uri) + +iotests.log("=== Converted image check ===") +qemu_img_log("check", nbd_uri) + +iotests.log("=== Comparing to source disk ===") +qemu_img_log("compare", src_disk, nbd_uri)tests/qemu-iotests/302.out new file +13
-0@@ -0,0 +1,27 @@ +=== Target image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: raw +virtual size: 446 KiB (457216 bytes) +disk size: unavailable + +=== Converted image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: qcow2 +virtual size: 10 MiB (10485760 bytes) +disk size: unavailable +cluster_size: 65536 +Format specific information:
tests/qemu-iotests/302.out line 23
Quoted 10 lines
tests/qemu-iotests/302.out new file +10
-0+ compat: 1.1 + compression type: zlib + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== Converted image check === +No errors were found on the image. +1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216
I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)
Max
Quoted 13 lines
tests/qemu-iotests/302.out new file +3
-0+=== Comparing to source disk === +Images are identical. +tests/qemu-iotests/group +1
-0@@ -308,3 +308,4 @@ 297 meta 299 auto quick 301 backing quick +302 quick
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
Quoted 17 lines · Max Reitz
tests/qemu-iotests/302 line 47
Quoted 13 lines · Nir Soffer
Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.
Signed-off-by Nir Soffer <[email protected]>---
tests/qemu-iotests/302 new file +6
-0+# The use case is writing qcow2 image directly into a tar file. Code to create +# real tar file not included. +# +# offset content +# ------------------------------- +# 0 first memebr header*member
Quoted 8 lines
tests/qemu-iotests/302 line 49
Quoted 2 lines
tests/qemu-iotests/302 new file +2
-0+# 512 first member data +# 1024 second memeber header*member
tests/qemu-iotests/302 line 52
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0+# 1536 second member data + +tar_file = file_path("test.tar")
I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.
Quoted 5 lines
tests/qemu-iotests/302 line 55
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk) +measure = json.loads(out) +qemu_img_create("-f", "raw", tar_file, str(measure["required"]))Should this be measure["required"] + 1536?
The test works without it (because of compression), but yes, if you are going to test writing into an offset, you should oversize your file by that same offset.
Quoted 22 lines
Quoted 8 lines
tests/qemu-iotests/302 new file +7
-0+ +nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) +nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + +# Use raw format to allow creating qcow2 directy into tar file. +qemu_nbd( + "--socket", nbd_sock,tests/qemu-iotests/302 line 72
Quoted 10 lines
tests/qemu-iotests/302 new file +10
-0+ "--persistent", + "--export-name", "exp", + "--format", "raw", + "--offset", "1536", + tar_file) + +iotests.log("=== Target image info ===") +qemu_img_log("info", nbd_uri) + +# Write image into the tar file. In a real applicatio we would write a tar*application
Quoted 7 lines
tests/qemu-iotests/302.out line 23
Quoted 4 lines
+=== Converted image check ===
tests/qemu-iotests/302.out new file +3
-0+No errors were found on the image. +1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)
Indeed - time to see what CI thinks of this.
At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
Quoted 10 lines · Eric Blake
Quoted 8 lines · Max Reitz
Quoted 5 lines · Nir Soffer
Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.
Signed-off-by Nir Soffer <[email protected]>---
Quoted 8 lines
Quoted 8 lines
tests/qemu-iotests/302 line 47
Quoted 6 lines
tests/qemu-iotests/302 new file +6
-0+# The use case is writing qcow2 image directly into a tar file. Code to create +# real tar file not included. +# +# offset content +# ------------------------------- +# 0 first memebr header*member
Sorry for the typos, I need to setup automated spelling check :-)
Quoted 6 lines
Quoted 6 lines
tests/qemu-iotests/302 line 49
Quoted 3 lines
tests/qemu-iotests/302 new file +2
-0+# 512 first member data +# 1024 second memeber header*member
Quoted 10 lines
Quoted 3 lines
tests/qemu-iotests/302 line 52
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0+# 1536 second member data + +tar_file = file_path("test.tar")I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.
The real code is more complicated, something like:
offset = tar.fileobj.tell() + BLOCK_SIZE
with open(tar.name, "r+") as f:
f.truncate(offset + measure["required"])
convert_image(image, tar.name, offset)
check = check_image(tar.name, offset)
size = check["image-end-offset"]
member = tarfile.TarInfo(name)
member.size = size
tar.addfile(member)
tar_size = offset + round_up(size)
tar.fileobj.seek(tar_size)
with open(tar.name, "r+") as f:
f.truncate(tar_size)
I'm not sure it helps qemu developers working on these tests.
Quoted 9 lines
Quoted 5 lines
tests/qemu-iotests/302 line 55
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk) +measure = json.loads(out) +qemu_img_create("-f", "raw", tar_file, str(measure["required"]))Should this be measure["required"] + 1536?
The test works without it (because of compression), but yes, if you are going to test writing into an offset, you should oversize your file by that same offset.
Right, in the real code using this I indeed use offset + required.
Quoted 29 lines
Quoted 21 lines
Quoted 7 lines
tests/qemu-iotests/302 new file +7
-0+ +nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) +nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + +# Use raw format to allow creating qcow2 directy into tar file. +qemu_nbd( + "--socket", nbd_sock,tests/qemu-iotests/302 line 72
Quoted 10 lines
tests/qemu-iotests/302 new file +10
-0+ "--persistent", + "--export-name", "exp", + "--format", "raw", + "--offset", "1536", + tar_file) + +iotests.log("=== Target image info ===") +qemu_img_log("info", nbd_uri) + +# Write image into the tar file. In a real applicatio we would write a tar*application
Quoted 7 lines
tests/qemu-iotests/302.out line 23
Quoted 4 lines
+=== Converted image check ===
tests/qemu-iotests/302.out new file +3
-0+No errors were found on the image. +1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)
Quoted 5 lines
Indeed - time to see what CI thinks of this.
At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.
Thanks Max and Eric.
Should I post a fixed version later today?
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
Quoted 23 lines · Nir Soffer
Quoted 6 lines
I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.
The real code is more complicated, something like:
offset =
tar.fileobj.tell()+BLOCK_SIZEwith
open(tar.name, "r+")as f:
f.truncate(offset + measure["required"])
convert_image(image, tar.name, offset)check =
check_image(tar.name, offset)
size = check["image-end-offset"]member =
tarfile.TarInfo(name)
member.size = size
tar.addfile(member)
Quoted 7 lines
tar_size= offset +round_up(size)
tar.fileobj.seek(tar_size)
withopen(tar.name, "r+")as f:
f.truncate(tar_size)I'm not sure it helps qemu developers working on these tests.
The closer the iotest is to reality, the more likely it will serve as a good regression test. Cutting corners risks a test that passes in isolation even when we've done something that breaks the overall process in one of the corners you cut.
Quoted 8 lines
Quoted 4 lines
At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.
Thanks Max and Eric.
Should I post a fixed version later today?
A v2 would be helpful.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
Quoted 32 lines · Eric Blake
Quoted 29 lines · Nir Soffer
Quoted 6 lines
I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.
The real code is more complicated, something like:
offset =
tar.fileobj.tell()+BLOCK_SIZEwith
open(tar.name, "r+")as f:
f.truncate(offset + measure["required"])
convert_image(image, tar.name, offset)check =
check_image(tar.name, offset)
size = check["image-end-offset"]member =
tarfile.TarInfo(name)
member.size = size
tar.addfile(member)
tar_size= offset +round_up(size)
tar.fileobj.seek(tar_size)
withopen(tar.name, "r+")as f:
f.truncate(tar_size)
Quoted 6 lines
Quoted 1 line
I'm not sure it helps qemu developers working on these tests.
The closer the iotest is to reality, the more likely it will serve as a good regression test. Cutting corners risks a test that passes in isolation even when we've done something that breaks the overall process in one of the corners you cut.
I'll add this code then.
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.
Thanks Max and Eric.
Should I post a fixed version later today?
A v2 would be helpful.
Will post later today.
Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD
In reply to Max ReitzQuoted 57 lines · Max Reitz
Quoted 18 lines · Nir Soffer
Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.
Signed-off-by Nir Soffer <[email protected]>--- tests/qemu-iotests/302 | 83 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 27 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 111 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.outtests/qemu-iotests/302 new file +0
-0tests/qemu-iotests/302 line 3
Quoted 3 lines
tests/qemu-iotests/302 new file +3
-0@@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +# +# Tests conveting qcow2 compressed to NBD*converting
Quoted 31 lines
tests/qemu-iotests/302 new file +31
-0+# +# Copyright (c) 2020 Nir Soffer <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +import json +import iotests + +from iotests import ( + file_path, + qemu_img, + qemu_img_create, + qemu_img_log, + qemu_img_pipe, + qemu_io, + qemu_nbd, +) +
Quoted 8 lines
tests/qemu-iotests/302 line 39
Quoted 5 lines
tests/qemu-iotests/302 new file +5
-0+iotests.script_initialize(supported_fmts=["qcow2"]) + +# Create source disk, format does not matter. +src_disk = file_path("disk.img") +qemu_img_create("-f", "raw", src_disk, "10m")If the format doesn’t matter, why not just use qcow2 and so put iotests.imgfmt here? (And everywhere else where you now have -f raw.)
I tried to use the simplest setup that is less likely to break, but thinking about
CI environments with strange storage, maybe using qcow2 source disk will be more robust.
Quoted 83 lines
tests/qemu-iotests/302 line 40
Quoted 1 line
tests/qemu-iotests/302 new file +1
-0+qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk)(Except I think
qemu_ioalready has -f qcow2 in its arguments by default, so specifying the format wouldn’t even be necessary here.)tests/qemu-iotests/302 line 47
Quoted 6 lines
tests/qemu-iotests/302 new file +6
-0+# The use case is writing qcow2 image directly into a tar file. Code to create +# real tar file not included. +# +# offset content +# ------------------------------- +# 0 first memebr header*member
tests/qemu-iotests/302 line 49
Quoted 2 lines
tests/qemu-iotests/302 new file +2
-0+# 512 first member data +# 1024 second memeber header*member
tests/qemu-iotests/302 line 55
Quoted 6 lines
tests/qemu-iotests/302 new file +6
-0+# 1536 second member data + +tar_file = file_path("test.tar") +out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk) +measure = json.loads(out) +qemu_img_create("-f", "raw", tar_file, str(measure["required"]))Should this be measure["required"] + 1536?
Quoted 7 lines
tests/qemu-iotests/302 new file +7
-0+ +nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) +nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + +# Use raw format to allow creating qcow2 directy into tar file. +qemu_nbd( + "--socket", nbd_sock,tests/qemu-iotests/302 line 72
Quoted 10 lines
tests/qemu-iotests/302 new file +10
-0+ "--persistent", + "--export-name", "exp", + "--format", "raw", + "--offset", "1536", + tar_file) + +iotests.log("=== Target image info ===") +qemu_img_log("info", nbd_uri) + +# Write image into the tar file. In a real applicatio we would write a tar*application
Quoted 35 lines
tests/qemu-iotests/302 new file +11
-0+# entry after writing the image. +qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri) + +iotests.log("=== Converted image info ===") +qemu_img_log("info", nbd_uri) + +iotests.log("=== Converted image check ===") +qemu_img_log("check", nbd_uri) + +iotests.log("=== Comparing to source disk ===") +qemu_img_log("compare", src_disk, nbd_uri)tests/qemu-iotests/302.out new file +18
-0@@ -0,0 +1,27 @@ +=== Target image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: raw +virtual size: 446 KiB (457216 bytes) +disk size: unavailable + +=== Converted image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: qcow2 +virtual size: 10 MiB (10485760 bytes) +disk size: unavailable +cluster_size: 65536 +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + refcount bits: 16 + corrupt: false
Quoted 8 lines
tests/qemu-iotests/302.out line 23
Quoted 5 lines
tests/qemu-iotests/302.out new file +5
-0+ +=== Converted image check === +No errors were found on the image. +1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)
I don't think the check would be affected, but maybe the compare step can use the strict
option if we change to qcow2 source disk.
Quoted 18 lines
Max
Quoted 13 lines
tests/qemu-iotests/302.out new file +3
-0+=== Comparing to source disk === +Images are identical. +tests/qemu-iotests/group +1
-0@@ -308,3 +308,4 @@ 297 meta 299 auto quick 301 backing quick +302 quick