Skip to contentSkip to conversation

Fix convert to qcow2 compressed to NBD

PATCH Superseded Nir Soffer 4 participants 13 messages Jul 26, 2020 – Jul 27, 2020 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Nir Soffer

author
#0 lore raw

cover Fix convert to qcow2 compressed to NBD

Fix qemu-img convert -O qcow2 -c to NBD URL and add missing test for this usage.

This already works now, but unfortunately qemu-img fails when trying to truncate the target image to the same size at the end of the operation.

Nir Soffer (2):
block: nbd: Fix convert qcow2 compressed to nbd
qemu-iotests: Test convert to qcow2 compressed to NBD

 block/nbd.c                | 27 +++++++++++++
 tests/qemu-iotests/302     | 83 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/302.out | 27 +++++++++++++
 tests/qemu-iotests/group   |  1 +
 4 files changed, 138 insertions(+)
 create mode 100755 tests/qemu-iotests/302
 create mode 100644 tests/qemu-iotests/302.out
Signature · 1 line
2.25.4
author
#1 lore raw

PATCH 1/2 block: nbd: Fix convert qcow2 compressed to nbd

When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement bdrv_co_truncate().

For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.

Example failing without this change:

In one shell starts qemu-nbd:

$ truncate -s 1g test.tar
$ qemu-nbd --socket=/tmp/nbd.sock --persistent --format=raw --offset 1536 test.tar

In another shell convert an image to qcow2 compressed via NBD:

$ echo "disk data" > disk.raw
$ truncate -s 1g disk.raw
$ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=/tmp/nbd.sock; echo $?
1

qemu-img failed, but the conversion was successful:

$ qemu-img info nbd+unix:///?socket=/tmp/nbd.sock
image: nbd+unix://?socket=/tmp/nbd.sock
file format: qcow2
virtual size: 1 GiB (1073741824 bytes)
...
$ qemu-img check nbd+unix:///?socket=/tmp/nbd.sock
No errors were found on the image.
1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters
Image end offset: 393216
$ qemu-img compare disk.raw nbd+unix:///?socket=/tmp/nbd.sock
Images are identical.
---
 block/nbd.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
block/nbd.c +27 -0
index 65a4f56924..2154113af3 100644
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)
     nbd_clear_bdrvstate(s);
 }
 
+/*
+ * NBD cannot truncate, but if the caller ask to truncate to the same size, or
+ * to a smaller size with extact=false, there is not reason to fail the
+ * operation.
+ */
+static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
+                                        bool exact, PreallocMode prealloc,
+                                        BdrvRequestFlags flags, Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (offset != s->info.size && exact) {
+        error_setg(errp, "Cannot resize NBD nodes");
+        return -ENOTSUP;
+    }
+
+    if (offset > s->info.size) {
+        error_setg(errp, "Cannot grow NBD nodes");
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
 static int64_t nbd_getlength(BlockDriverState *bs)
 {
     BDRVNBDState *s = bs->opaque;
@@ -2045,6 +2069,7 @@ static BlockDriver bdrv_nbd = {
     .bdrv_co_flush_to_os        = nbd_co_flush,
     .bdrv_co_pdiscard           = nbd_client_co_pdiscard,
     .bdrv_refresh_limits        = nbd_refresh_limits,
+    .bdrv_co_truncate           = nbd_co_truncate,
     .bdrv_getlength             = nbd_getlength,
     .bdrv_detach_aio_context    = nbd_client_detach_aio_context,
     .bdrv_attach_aio_context    = nbd_client_attach_aio_context,
@@ -2072,6 +2097,7 @@ static BlockDriver bdrv_nbd_tcp = {
     .bdrv_co_flush_to_os        = nbd_co_flush,
     .bdrv_co_pdiscard           = nbd_client_co_pdiscard,
     .bdrv_refresh_limits        = nbd_refresh_limits,
+    .bdrv_co_truncate           = nbd_co_truncate,
     .bdrv_getlength             = nbd_getlength,
     .bdrv_detach_aio_context    = nbd_client_detach_aio_context,
     .bdrv_attach_aio_context    = nbd_client_attach_aio_context,
@@ -2099,6 +2125,7 @@ static BlockDriver bdrv_nbd_unix = {
     .bdrv_co_flush_to_os        = nbd_co_flush,
     .bdrv_co_pdiscard           = nbd_client_co_pdiscard,
     .bdrv_refresh_limits        = nbd_refresh_limits,
+    .bdrv_co_truncate           = nbd_co_truncate,
     .bdrv_getlength             = nbd_getlength,
     .bdrv_detach_aio_context    = nbd_client_detach_aio_context,
     .bdrv_attach_aio_context    = nbd_client_attach_aio_context,
Signature · 1 line
2.25.4
#2 lore raw

Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd

Quoted 67 lines · Nir Soffer

When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement bdrv_co_truncate().

For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.

Example failing without this change:

In one shell starts qemu-nbd:

$ truncate -s 1g test.tar
$ qemu-nbd --socket=/tmp/nbd.sock --persistent --format=raw --offset 1536 test.tar

In another shell convert an image to qcow2 compressed via NBD:

$ echo "disk data" > disk.raw
$ truncate -s 1g disk.raw
$ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=/tmp/nbd.sock; echo $?
1

qemu-img failed, but the conversion was successful:

$ qemu-img info nbd+unix:///?socket=/tmp/nbd.sock
image: nbd+unix://?socket=/tmp/nbd.sock
file format: qcow2
virtual size: 1 GiB (1073741824 bytes)
...
$ qemu-img check nbd+unix:///?socket=/tmp/nbd.sock
No errors were found on the image.
1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters
Image end offset: 393216
$ qemu-img compare disk.raw nbd+unix:///?socket=/tmp/nbd.sock
Images are identical.
---
 block/nbd.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
block/nbd.c +13 -0
index 65a4f56924..2154113af3 100644
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)
     nbd_clear_bdrvstate(s);
 }
 
+/*
+ * NBD cannot truncate, but if the caller ask to truncate to the same size, or
+ * to a smaller size with extact=false, there is not reason to fail the
+ * operation.
+ */
+static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
+                                        bool exact, PreallocMode prealloc,
+                                        BdrvRequestFlags flags, Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (offset != s->info.size && exact) {
+        error_setg(errp, "Cannot resize NBD nodes");

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1991

Quoted 10 lines
block/nbd.c +10 -0
+        return -ENOTSUP;
+    }
+
+    if (offset > s->info.size) {
+        error_setg(errp, "Cannot grow NBD nodes");
+        return -EINVAL;
+    }
+
+    return 0;
+}

Can we also check @prealloc, like iscsi.c does it?

(Yes, preallocation only makes sense for growing, but you can for example truncate a 2 GB NBD node to 1 GB, and then grow it to 1.5 GB, and for the latter operation it might make sense to give a preallocation parameter. (Except block_resize doesn’t allow that, but, well.))

Max

#3 lore raw

Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd

In reply to Nir Soffer
Quoted 3 lines · Nir Soffer

When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement

fails

Quoted 1 line

bdrv_co_truncate().

Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.

Quoted 21 lines

For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.

Example failing without this change:

---
  block/nbd.c | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)
block/nbd.c +0 -0
index 65a4f56924..2154113af3 100644

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970

Quoted 5 lines
block/nbd.c +2 -0
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)
      nbd_clear_bdrvstate(s);
  }
 
+/*
+ * NBD cannot truncate, but if the caller ask to truncate to the same size, or

asks

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971

Quoted 1 line
block/nbd.c +1 -0
+ * to a smaller size with extact=false, there is not reason to fail the

exact, no

Quoted 9 lines
block/nbd.c +9 -0
+ * operation.
+ */
+static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
+                                        bool exact, PreallocMode prealloc,
+                                        BdrvRequestFlags flags, Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (offset != s->info.size && exact) {

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990

Quoted 10 lines
block/nbd.c +10 -0
+        error_setg(errp, "Cannot resize NBD nodes");
+        return -ENOTSUP;
+    }
+
+    if (offset > s->info.size) {
+        error_setg(errp, "Cannot grow NBD nodes");
+        return -EINVAL;
+    }
+
+    return 0;

Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since s->info.size is constant, but the BDS size is not if inexact resize succeeds).

As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#4 lore raw

Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd

Quoted 6 lines · Eric Blake
Quoted 3 lines · Nir Soffer

When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement

Quoted 6 lines

fails

Quoted 1 line

bdrv_co_truncate().

Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.

I started with ignoring ENOTSUP in qcow2, but felt less safe about this approach since the same issue may happen in other flows, and making nbd driver behave like a block device looks like a safer change.

Quoted 58 lines
Quoted 20 lines

For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.

Example failing without this change:

---
  block/nbd.c | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)
block/nbd.c +0 -0
index 65a4f56924..2154113af3 100644

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970

Quoted 5 lines
block/nbd.c +2 -0
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)
      nbd_clear_bdrvstate(s);
  }
 
+/*
+ * NBD cannot truncate, but if the caller ask to truncate to the same size, or

asks

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971

Quoted 1 line
block/nbd.c +1 -0
+ * to a smaller size with extact=false, there is not reason to fail the

exact, no

Quoted 9 lines
block/nbd.c +9 -0
+ * operation.
+ */
+static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
+                                        bool exact, PreallocMode prealloc,
+                                        BdrvRequestFlags flags, Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (offset != s->info.size && exact) {

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990

Quoted 10 lines
block/nbd.c +10 -0
+        error_setg(errp, "Cannot resize NBD nodes");
+        return -ENOTSUP;
+    }
+
+    if (offset > s->info.size) {
+        error_setg(errp, "Cannot grow NBD nodes");
+        return -EINVAL;
+    }
+
+    return 0;

Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since s->info.size is constant, but the BDS size is not if inexact resize succeeds).

Quoted 2 lines

As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.

I'll post v2 with the test fixes later today.

Quoted 6 lines

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org

#5 lore raw

Re: [PATCH 1/2] block: nbd: Fix convert qcow2 compressed to nbd

In reply to Eric Blake
Quoted 63 lines · Eric Blake
Quoted 3 lines · Nir Soffer

When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fail for the nbd driver since it does not implement

fails

Quoted 1 line

bdrv_co_truncate().

Arguably, qemu-img should be taught to ignore the failure, since it is not unique to the nbd driver. But I can live with your approach here.

Quoted 21 lines

For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd.

Example failing without this change:

---
  block/nbd.c | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)
block/nbd.c +0 -0
index 65a4f56924..2154113af3 100644

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1970

Quoted 5 lines
block/nbd.c +2 -0
@@ -1966,6 +1966,30 @@ static void nbd_close(BlockDriverState *bs)
      nbd_clear_bdrvstate(s);
  }
 
+/*
+ * NBD cannot truncate, but if the caller ask to truncate to the same size, or

asks

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1971

Quoted 1 line
block/nbd.c +1 -0
+ * to a smaller size with extact=false, there is not reason to fail the

exact, no

Quoted 17 lines
block/nbd.c +17 -0
+ * operation.
+ */
+static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
+                                        bool exact, PreallocMode prealloc,
+                                        BdrvRequestFlags flags, Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (offset != s->info.size && exact) {
+        error_setg(errp, "Cannot resize NBD nodes");
+        return -ENOTSUP;
+    }
+
+    if (offset > s->info.size) {
+        error_setg(errp, "Cannot grow NBD nodes");
+        return -EINVAL;
+    }
Quoted 8 lines

block/nbd.c static void nbd_close(BlockDriverState *bs) · line 1990

Quoted 2 lines
block/nbd.c +2 -0
+
+    return 0;

Looks reasonable. As Max said, I wonder if we want to reject particular preallocation modes (looking at block/file-posix.c:raw_co_truncate), in the case where the image was resized down and then back up (since s->info.size is constant, but the BDS size is not if inexact resize succeeds).

Do we want to fail if someone specifies -o preallocation={falloc,full}?

I see we convert DRV_REQ_MAY_UNMAP to NBD_CMD_FLAG_NO_HOLE
so using -o preallocation=falloc,full should be correct. But the last request zero
length write request does not do anything, so failing does not look useful.

Quoted 8 lines

As you have a bugzilla entry, I think this is safe for -rc2; I'll be touching up the typos and queuing it through my NBD tree later today.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org

author
#6 lore raw

PATCH 2/2 qemu-iotests: Test convert to qcow2 compressed to NBD

In reply to Nir Soffer

Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.

---
 tests/qemu-iotests/302     | 83 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/302.out | 27 +++++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 111 insertions(+)
 create mode 100755 tests/qemu-iotests/302
 create mode 100644 tests/qemu-iotests/302.out
tests/qemu-iotests/302 new file +83 -0
new file mode 100755
index 0000000000..cefde1f7cf
@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+#
+# Tests conveting qcow2 compressed to NBD
+#
+# Copyright (c) 2020 Nir Soffer <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+import json
+import iotests
+
+from iotests import (
+    file_path,
+    qemu_img,
+    qemu_img_create,
+    qemu_img_log,
+    qemu_img_pipe,
+    qemu_io,
+    qemu_nbd,
+)
+
+iotests.script_initialize(supported_fmts=["qcow2"])
+
+# Create source disk, format does not matter.
+src_disk = file_path("disk.img")
+qemu_img_create("-f", "raw", src_disk, "10m")
+qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk)
+
+# The use case is writing qcow2 image directly into a tar file. Code to create
+# real tar file not included.
+#
+# offset    content
+# -------------------------------
+#      0    first memebr header
+#    512    first member data
+#   1024    second memeber header
+#   1536    second member data
+
+tar_file = file_path("test.tar")
+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk)
+measure = json.loads(out)
+qemu_img_create("-f", "raw", tar_file, str(measure["required"]))
+
+nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir)
+nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock
+
+# Use raw format to allow creating qcow2 directy into tar file.
+qemu_nbd(
+    "--socket", nbd_sock,
+    "--persistent",
+    "--export-name", "exp",
+    "--format", "raw",
+    "--offset", "1536",
+    tar_file)
+
+iotests.log("=== Target image info ===")
+qemu_img_log("info", nbd_uri)
+
+# Write image into the tar file. In a real applicatio we would write a tar
+# entry after writing the image.
+qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri)
+
+iotests.log("=== Converted image info ===")
+qemu_img_log("info", nbd_uri)
+
+iotests.log("=== Converted image check ===")
+qemu_img_log("check", nbd_uri)
+
+iotests.log("=== Comparing to source disk ===")
+qemu_img_log("compare", src_disk, nbd_uri)
tests/qemu-iotests/302.out new file +27 -0
new file mode 100644
index 0000000000..babef3d574
@@ -0,0 +1,27 @@
+=== Target image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: raw
+virtual size: 446 KiB (457216 bytes)
+disk size: unavailable
+
+=== Converted image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: qcow2
+virtual size: 10 MiB (10485760 bytes)
+disk size: unavailable
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    refcount bits: 16
+    corrupt: false
+
+=== Converted image check ===
+No errors were found on the image.
+1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters
+Image end offset: 393216
+
+=== Comparing to source disk ===
+Images are identical.
+
tests/qemu-iotests/group +1 -0
index 1d0252e1f0..1e1cb27bc8 100644
@@ -308,3 +308,4 @@
 297 meta
 299 auto quick
 301 backing quick
+302 quick
Signature · 1 line
2.25.4
#7 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

Quoted 18 lines · Nir Soffer

Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.

---
 tests/qemu-iotests/302     | 83 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/302.out | 27 +++++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 111 insertions(+)
 create mode 100755 tests/qemu-iotests/302
 create mode 100644 tests/qemu-iotests/302.out
tests/qemu-iotests/302 new file +0 -0
new file mode 100755
index 0000000000..cefde1f7cf

tests/qemu-iotests/302 line 3

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+#
+# Tests conveting qcow2 compressed to NBD

*converting

Quoted 26 lines
tests/qemu-iotests/302 new file +26 -0
+#
+# Copyright (c) 2020 Nir Soffer <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+import json
+import iotests
+
+from iotests import (
+    file_path,
+    qemu_img,
+    qemu_img_create,
+    qemu_img_log,

tests/qemu-iotests/302 line 39

Quoted 10 lines
tests/qemu-iotests/302 new file +10 -0
+    qemu_img_pipe,
+    qemu_io,
+    qemu_nbd,
+)
+
+iotests.script_initialize(supported_fmts=["qcow2"])
+
+# Create source disk, format does not matter.
+src_disk = file_path("disk.img")
+qemu_img_create("-f", "raw", src_disk, "10m")

If the format doesn’t matter, why not just use qcow2 and so put iotests.imgfmt here? (And everywhere else where you now have -f raw.)

tests/qemu-iotests/302 line 40

Quoted 1 line
tests/qemu-iotests/302 new file +1 -0
+qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk)

(Except I think qemu_io already has -f qcow2 in its arguments by default, so specifying the format wouldn’t even be necessary here.)

tests/qemu-iotests/302 line 47

Quoted 6 lines
tests/qemu-iotests/302 new file +6 -0
+# The use case is writing qcow2 image directly into a tar file. Code to create
+# real tar file not included.
+#
+# offset    content
+# -------------------------------
+#      0    first memebr header

*member

tests/qemu-iotests/302 line 49

Quoted 2 lines
tests/qemu-iotests/302 new file +2 -0
+#    512    first member data
+#   1024    second memeber header

*member

tests/qemu-iotests/302 line 55

Quoted 6 lines
tests/qemu-iotests/302 new file +6 -0
+#   1536    second member data
+
+tar_file = file_path("test.tar")
+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk)
+measure = json.loads(out)
+qemu_img_create("-f", "raw", tar_file, str(measure["required"]))

Should this be measure["required"] + 1536?

Quoted 7 lines
tests/qemu-iotests/302 new file +7 -0
+
+nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir)
+nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock
+
+# Use raw format to allow creating qcow2 directy into tar file.
+qemu_nbd(
+    "--socket", nbd_sock,

tests/qemu-iotests/302 line 72

Quoted 10 lines
tests/qemu-iotests/302 new file +10 -0
+    "--persistent",
+    "--export-name", "exp",
+    "--format", "raw",
+    "--offset", "1536",
+    tar_file)
+
+iotests.log("=== Target image info ===")
+qemu_img_log("info", nbd_uri)
+
+# Write image into the tar file. In a real applicatio we would write a tar

*application

Quoted 30 lines
tests/qemu-iotests/302 new file +11 -0
+# entry after writing the image.
+qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri)
+
+iotests.log("=== Converted image info ===")
+qemu_img_log("info", nbd_uri)
+
+iotests.log("=== Converted image check ===")
+qemu_img_log("check", nbd_uri)
+
+iotests.log("=== Comparing to source disk ===")
+qemu_img_log("compare", src_disk, nbd_uri)
tests/qemu-iotests/302.out new file +13 -0
new file mode 100644
index 0000000000..babef3d574
@@ -0,0 +1,27 @@
+=== Target image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: raw
+virtual size: 446 KiB (457216 bytes)
+disk size: unavailable
+
+=== Converted image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: qcow2
+virtual size: 10 MiB (10485760 bytes)
+disk size: unavailable
+cluster_size: 65536
+Format specific information:

tests/qemu-iotests/302.out line 23

Quoted 10 lines
tests/qemu-iotests/302.out new file +10 -0
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    refcount bits: 16
+    corrupt: false
+
+=== Converted image check ===
+No errors were found on the image.
+1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters
+Image end offset: 393216

I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)

Max

Quoted 13 lines
tests/qemu-iotests/302.out new file +3 -0
+=== Comparing to source disk ===
+Images are identical.
+
tests/qemu-iotests/group +1 -0
index 1d0252e1f0..1e1cb27bc8 100644
@@ -308,3 +308,4 @@
 297 meta
 299 auto quick
 301 backing quick
+302 quick
#8 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

Quoted 17 lines · Max Reitz

tests/qemu-iotests/302 line 47

Quoted 13 lines · Nir Soffer

Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.

---

tests/qemu-iotests/302 new file +6 -0
+# The use case is writing qcow2 image directly into a tar file. Code to create
+# real tar file not included.
+#
+# offset    content
+# -------------------------------
+#      0    first memebr header

*member

Quoted 8 lines

tests/qemu-iotests/302 line 49

Quoted 2 lines
tests/qemu-iotests/302 new file +2 -0
+#    512    first member data
+#   1024    second memeber header

*member

tests/qemu-iotests/302 line 52

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
+#   1536    second member data
+
+tar_file = file_path("test.tar")

I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.

Quoted 5 lines

tests/qemu-iotests/302 line 55

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk)
+measure = json.loads(out)
+qemu_img_create("-f", "raw", tar_file, str(measure["required"]))

Should this be measure["required"] + 1536?

The test works without it (because of compression), but yes, if you are going to test writing into an offset, you should oversize your file by that same offset.

Quoted 22 lines
Quoted 8 lines
tests/qemu-iotests/302 new file +7 -0
+
+nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir)
+nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock
+
+# Use raw format to allow creating qcow2 directy into tar file.
+qemu_nbd(
+    "--socket", nbd_sock,

tests/qemu-iotests/302 line 72

Quoted 10 lines
tests/qemu-iotests/302 new file +10 -0
+    "--persistent",
+    "--export-name", "exp",
+    "--format", "raw",
+    "--offset", "1536",
+    tar_file)
+
+iotests.log("=== Target image info ===")
+qemu_img_log("info", nbd_uri)
+
+# Write image into the tar file. In a real applicatio we would write a tar

*application

Quoted 7 lines

tests/qemu-iotests/302.out line 23

Quoted 4 lines

+=== Converted image check ===

tests/qemu-iotests/302.out new file +3 -0
+No errors were found on the image.
+1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters
+Image end offset: 393216

I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)

Indeed - time to see what CI thinks of this.

At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#9 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

Quoted 10 lines · Eric Blake
Quoted 8 lines · Max Reitz
Quoted 5 lines · Nir Soffer

Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.

---

Quoted 8 lines
Quoted 8 lines

tests/qemu-iotests/302 line 47

Quoted 6 lines
tests/qemu-iotests/302 new file +6 -0
+# The use case is writing qcow2 image directly into a tar file. Code to create
+# real tar file not included.
+#
+# offset    content
+# -------------------------------
+#      0    first memebr header

*member

Sorry for the typos, I need to setup automated spelling check :-)

Quoted 6 lines
Quoted 6 lines

tests/qemu-iotests/302 line 49

Quoted 3 lines
tests/qemu-iotests/302 new file +2 -0
+#    512    first member data
+#   1024    second memeber header

*member

Quoted 10 lines
Quoted 3 lines

tests/qemu-iotests/302 line 52

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
+#   1536    second member data
+
+tar_file = file_path("test.tar")

I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.

The real code is more complicated, something like:

offset = tar.fileobj.tell() + BLOCK_SIZE

with open(tar.name, "r+") as f:
f.truncate(offset + measure["required"])

convert_image(image, tar.name, offset)

check = check_image(tar.name, offset)
size = check["image-end-offset"]

member = tarfile.TarInfo(name)
member.size = size
tar.addfile(member)

tar_size = offset + round_up(size)

tar.fileobj.seek(tar_size)
with open(tar.name, "r+") as f:
f.truncate(tar_size)

I'm not sure it helps qemu developers working on these tests.

Quoted 9 lines
Quoted 5 lines

tests/qemu-iotests/302 line 55

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk)
+measure = json.loads(out)
+qemu_img_create("-f", "raw", tar_file, str(measure["required"]))

Should this be measure["required"] + 1536?

The test works without it (because of compression), but yes, if you are going to test writing into an offset, you should oversize your file by that same offset.

Right, in the real code using this I indeed use offset + required.

Quoted 29 lines
Quoted 21 lines
Quoted 7 lines
tests/qemu-iotests/302 new file +7 -0
+
+nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir)
+nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock
+
+# Use raw format to allow creating qcow2 directy into tar file.
+qemu_nbd(
+    "--socket", nbd_sock,

tests/qemu-iotests/302 line 72

Quoted 10 lines
tests/qemu-iotests/302 new file +10 -0
+    "--persistent",
+    "--export-name", "exp",
+    "--format", "raw",
+    "--offset", "1536",
+    tar_file)
+
+iotests.log("=== Target image info ===")
+qemu_img_log("info", nbd_uri)
+
+# Write image into the tar file. In a real applicatio we would write a tar

*application

Quoted 7 lines

tests/qemu-iotests/302.out line 23

Quoted 4 lines

+=== Converted image check ===

tests/qemu-iotests/302.out new file +3 -0
+No errors were found on the image.
+1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters
+Image end offset: 393216

I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)

Quoted 5 lines

Indeed - time to see what CI thinks of this.

At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.

Thanks Max and Eric.

Should I post a fixed version later today?

#10 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

Quoted 23 lines · Nir Soffer
Quoted 6 lines

I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.

The real code is more complicated, something like:

offset = tar.fileobj.tell() + BLOCK_SIZE

with open(tar.name, "r+") as f:
f.truncate(offset + measure["required"])

convert_image(image, tar.name, offset)

check = check_image(tar.name, offset)
size = check["image-end-offset"]

member = tarfile.TarInfo(name)
member.size = size
tar.addfile(member)

Quoted 7 lines

tar_size = offset + round_up(size)

tar.fileobj.seek(tar_size)
with open(tar.name, "r+") as f:
f.truncate(tar_size)

I'm not sure it helps qemu developers working on these tests.

The closer the iotest is to reality, the more likely it will serve as a good regression test. Cutting corners risks a test that passes in isolation even when we've done something that breaks the overall process in one of the corners you cut.

Quoted 8 lines
Quoted 4 lines

At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.

Thanks Max and Eric.

Should I post a fixed version later today?

A v2 would be helpful.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#11 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

Quoted 32 lines · Eric Blake
Quoted 29 lines · Nir Soffer
Quoted 6 lines

I guess it's okay that you don't create a real tar file here, but listing the commands to create it (even as a comment) is better than just saying "trust me". And it doesn't seem like that much more work - it looks like the key to your test is that you created a tar file containing two files, where the first file was less than 512 bytes and the second file is your target destination that you will be rewriting.

The real code is more complicated, something like:

offset = tar.fileobj.tell() + BLOCK_SIZE

with open(tar.name, "r+") as f:
f.truncate(offset + measure["required"])

convert_image(image, tar.name, offset)

check = check_image(tar.name, offset)
size = check["image-end-offset"]

member = tarfile.TarInfo(name)
member.size = size
tar.addfile(member)

tar_size = offset + round_up(size)

tar.fileobj.seek(tar_size)
with open(tar.name, "r+") as f:
f.truncate(tar_size)

Quoted 6 lines
Quoted 1 line

I'm not sure it helps qemu developers working on these tests.

The closer the iotest is to reality, the more likely it will serve as a good regression test. Cutting corners risks a test that passes in isolation even when we've done something that breaks the overall process in one of the corners you cut.

I'll add this code then.

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

At any rate, given the urgency of getting pull requests for -rc2 in before slamming Peter tomorrow, I'll probably try to touch up the issues Max pointed out and queue it today.

Thanks Max and Eric.

Should I post a fixed version later today?

A v2 would be helpful.

Will post later today.

#12 lore raw

Re: [PATCH 2/2] qemu-iotests: Test convert to qcow2 compressed to NBD

In reply to Max Reitz
Quoted 57 lines · Max Reitz
Quoted 18 lines · Nir Soffer

Add test for "qemu-img convert -O qcow2 -c" to NBD target. The use case is writing compressed disk content to OVA archive.

---
 tests/qemu-iotests/302     | 83 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/302.out | 27 +++++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 111 insertions(+)
 create mode 100755 tests/qemu-iotests/302
 create mode 100644 tests/qemu-iotests/302.out
tests/qemu-iotests/302 new file +0 -0
new file mode 100755
index 0000000000..cefde1f7cf

tests/qemu-iotests/302 line 3

Quoted 3 lines
tests/qemu-iotests/302 new file +3 -0
@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+#
+# Tests conveting qcow2 compressed to NBD

*converting

Quoted 31 lines
tests/qemu-iotests/302 new file +31 -0
+#
+# Copyright (c) 2020 Nir Soffer <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+import json
+import iotests
+
+from iotests import (
+    file_path,
+    qemu_img,
+    qemu_img_create,
+    qemu_img_log,
+    qemu_img_pipe,
+    qemu_io,
+    qemu_nbd,
+)
+
Quoted 8 lines

tests/qemu-iotests/302 line 39

Quoted 5 lines
tests/qemu-iotests/302 new file +5 -0
+iotests.script_initialize(supported_fmts=["qcow2"])
+
+# Create source disk, format does not matter.
+src_disk = file_path("disk.img")
+qemu_img_create("-f", "raw", src_disk, "10m")

If the format doesn’t matter, why not just use qcow2 and so put iotests.imgfmt here? (And everywhere else where you now have -f raw.)

I tried to use the simplest setup that is less likely to break, but thinking about
CI environments with strange storage, maybe using qcow2 source disk will be more robust.

Quoted 83 lines

tests/qemu-iotests/302 line 40

Quoted 1 line
tests/qemu-iotests/302 new file +1 -0
+qemu_io("-f", "raw", "-c", "write 1m 64K", src_disk)

(Except I think qemu_io already has -f qcow2 in its arguments by default, so specifying the format wouldn’t even be necessary here.)

tests/qemu-iotests/302 line 47

Quoted 6 lines
tests/qemu-iotests/302 new file +6 -0
+# The use case is writing qcow2 image directly into a tar file. Code to create
+# real tar file not included.
+#
+# offset    content
+# -------------------------------
+#      0    first memebr header

*member

tests/qemu-iotests/302 line 49

Quoted 2 lines
tests/qemu-iotests/302 new file +2 -0
+#    512    first member data
+#   1024    second memeber header

*member

tests/qemu-iotests/302 line 55

Quoted 6 lines
tests/qemu-iotests/302 new file +6 -0
+#   1536    second member data
+
+tar_file = file_path("test.tar")
+out = qemu_img_pipe("measure", "-O", "qcow2", "--output", "json", src_disk)
+measure = json.loads(out)
+qemu_img_create("-f", "raw", tar_file, str(measure["required"]))

Should this be measure["required"] + 1536?

Quoted 7 lines
tests/qemu-iotests/302 new file +7 -0
+
+nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir)
+nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock
+
+# Use raw format to allow creating qcow2 directy into tar file.
+qemu_nbd(
+    "--socket", nbd_sock,

tests/qemu-iotests/302 line 72

Quoted 10 lines
tests/qemu-iotests/302 new file +10 -0
+    "--persistent",
+    "--export-name", "exp",
+    "--format", "raw",
+    "--offset", "1536",
+    tar_file)
+
+iotests.log("=== Target image info ===")
+qemu_img_log("info", nbd_uri)
+
+# Write image into the tar file. In a real applicatio we would write a tar

*application

Quoted 35 lines
tests/qemu-iotests/302 new file +11 -0
+# entry after writing the image.
+qemu_img("convert", "-f", "raw", "-O", "qcow2", "-c", src_disk, nbd_uri)
+
+iotests.log("=== Converted image info ===")
+qemu_img_log("info", nbd_uri)
+
+iotests.log("=== Converted image check ===")
+qemu_img_log("check", nbd_uri)
+
+iotests.log("=== Comparing to source disk ===")
+qemu_img_log("compare", src_disk, nbd_uri)
tests/qemu-iotests/302.out new file +18 -0
new file mode 100644
index 0000000000..babef3d574
@@ -0,0 +1,27 @@
+=== Target image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: raw
+virtual size: 446 KiB (457216 bytes)
+disk size: unavailable
+
+=== Converted image info ===
+image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
+file format: qcow2
+virtual size: 10 MiB (10485760 bytes)
+disk size: unavailable
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    refcount bits: 16
+    corrupt: false
Quoted 8 lines

tests/qemu-iotests/302.out line 23

Quoted 5 lines
tests/qemu-iotests/302.out new file +5 -0
+
+=== Converted image check ===
+No errors were found on the image.
+1/160 = 0.62% allocated, 100.00% fragmented, 100.00% compressed clusters
+Image end offset: 393216

I hope none of this is fs-dependant. (I don’t think it is, but who knows. I suppose we’ll find out.)

I don't think the check would be affected, but maybe the compare step can use the strict
option if we change to qcow2 source disk.

Quoted 18 lines

Max

Quoted 13 lines
tests/qemu-iotests/302.out new file +3 -0
+=== Comparing to source disk ===
+Images are identical.
+
tests/qemu-iotests/group +1 -0
index 1d0252e1f0..1e1cb27bc8 100644
@@ -308,3 +308,4 @@
 297 meta
 299 auto quick
 301 backing quick
+302 quick