Skip to contentSkip to conversation

VHD/VPC format compatibility

PATCH Superseded Jeff Cody 4 participants 19 messages Feb 24, 2016 Block layer

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jeff Cody

author
#0 lore raw

cover VHD/VPC format compatibility

This is a long-standing issue that has come up many times, and has had several different patches posted to fix it. Virtual PC, and Hyper-V calculate the disk geometry differently for VHD, leading to compatibility issues.

We want to fix these compatibility problems, however we want to make sure we do not break backwards compatibility.

There are two areas of compatibility addressed:

* Reading images (Patch 1)
* Creating images (Patch 3)

Please see the commit messages in Patches 1,3 for details.

Jeff Cody (4):
block/vpc: choose size calculation method based on creator_app field block/vpc: tests for auto-detecting VPC and Hyper-V VHD images block/vpc: give option to force the current_size field in .bdrv_create block/vpc: add tests for image creation force_size parameter

 block/vpc.c                                        | 102 ++++++++++-
 tests/qemu-iotests/146                             | 200 +++++++++++++++++++++
 tests/qemu-iotests/146.out                         |  82 +++++++++
 tests/qemu-iotests/group                           |   1 +
 .../sample_images/hyperv2012r2-dynamic.vhd.bz2     | Bin 0 -> 214 bytes
 .../sample_images/virtualpc-dynamic.vhd.bz2        | Bin 0 -> 212 bytes
 6 files changed, 379 insertions(+), 6 deletions(-)
 create mode 100755 tests/qemu-iotests/146
 create mode 100644 tests/qemu-iotests/146.out
 create mode 100644 tests/qemu-iotests/sample_images/hyperv2012r2-dynamic.vhd.bz2
 create mode 100644 tests/qemu-iotests/sample_images/virtualpc-dynamic.vhd.bz2
Signature · 1 line
1.9.3
author
#1 lore raw

PATCH 1/4 block/vpc: choose size calculation method based on creator_app field

The VHD file format is used by both Virtual PC, and Hyper-V. However, how the virtual disk size is calculated varies between the two.

Virtual PC uses the CHS drive parameters to determine the drive size. Hyper-V, on the other hand, uses the current_size field in the footer when determining image size.

This is problematic for a few reasons:

* VHD images from Hyper-V, using CHS calculations, will likely be trunctated.

* If we just rely always on current_size, then QEMU may have data compatibility issues with Virtual PC (we may write too much data into a VHD file to be used by Virtual PC, for instance).

* Existing VHD images created by QEMU have used the CHS calculations, except for images exceeding the 127GB limit. We want to remain compatible with our own generated images.

Luckily, the VHD specification defines a 'Creator App' field, that is used to indicate what software created the VHD file.

This patch does two things:

1. Uses the 'Creator App' field to help determine how to calculate size, and

2. Adds a VPC format option 'force_size_calc', so that the user can override the 'Creator App' auto-detection, in case there exist VHD images with unknown or contradictory 'Creator App' entries.

N.B.: We currently use the maximum CHS value as an indication to use the current_size field. This patch does not change that, even with the 'force_size_calc' option.

---
 block/vpc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 82 insertions(+), 5 deletions(-)
block/vpc.c +82 -5
index f504536..54a38a7 100644
@@ -128,6 +128,8 @@ typedef struct BDRVVPCState {
 
     uint32_t block_size;
     uint32_t bitmap_size;
+    bool force_use_chs;
+    bool force_use_sz;
 
 #ifdef CACHE
     uint8_t *pageentry_u8;
@@ -140,6 +142,22 @@ typedef struct BDRVVPCState {
     Error *migration_blocker;
 } BDRVVPCState;
 
+#define VPC_OPT_SIZE_CALC "force_size_calc"
+static QemuOptsList vpc_runtime_opts = {
+    .name = "vpc-runtime-opts",
+    .head = QTAILQ_HEAD_INITIALIZER(vpc_runtime_opts.head),
+    .desc = {
+        {
+            .name = VPC_OPT_SIZE_CALC,
+            .type = QEMU_OPT_STRING,
+            .help = "Force disk size calculation to use either CHS geometry, "
+                    "or use the disk current_size specified in the VHD footer. "
+                    "{chs, current_size}"
+        },
+        { /* end of list */ }
+    }
+};
+
 static uint32_t vpc_checksum(uint8_t* buf, size_t size)
 {
     uint32_t res = 0;
@@ -159,6 +177,25 @@ static int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
     return 0;
 }
 
+static void vpc_parse_options(BlockDriverState *bs, QemuOpts *opts,
+                              Error **errp)
+{
+    BDRVVPCState *s = bs->opaque;
+    const char *size_calc;
+
+    size_calc = qemu_opt_get(opts, VPC_OPT_SIZE_CALC);
+
+    if (!size_calc) {
+       /* no override, use autodetect only */
+    } else if (!strcmp(size_calc, "current_size")) {
+        s->force_use_sz = true;
+    } else if (!strcmp(size_calc, "chs")) {
+        s->force_use_chs = true;
+    } else {
+        error_setg(errp, "Invalid size calculation mode: '%s'", size_calc);
+    }
+}
+
 static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
                     Error **errp)
 {
@@ -166,6 +203,9 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
     int i;
     VHDFooter *footer;
     VHDDynDiskHeader *dyndisk_header;
+    QemuOpts *opts = NULL;
+    Error *local_err = NULL;
+    bool use_chs;
     uint8_t buf[HEADER_SIZE];
     uint32_t checksum;
     uint64_t computed_size;
@@ -173,6 +213,21 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
     int disk_type = VHD_DYNAMIC;
     int ret;
 
+    opts = qemu_opts_create(&vpc_runtime_opts, NULL, 0, &error_abort);
+    qemu_opts_absorb_qdict(opts, options, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    vpc_parse_options(bs, opts, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        ret = -EINVAL;
+        goto fail;
+    }
+
     ret = bdrv_pread(bs->file->bs, 0, s->footer_buf, HEADER_SIZE);
     if (ret < 0) {
         goto fail;
@@ -218,12 +273,34 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
     bs->total_sectors = (int64_t)
         be16_to_cpu(footer->cyls) * footer->heads * footer->secs_per_cyl;
 
-    /* Images that have exactly the maximum geometry are probably bigger and
-     * would be truncated if we adhered to the geometry for them. Rely on
-     * footer->current_size for them. */
-    if (bs->total_sectors == VHD_MAX_GEOMETRY) {
+    /* Microsoft Virtual PC and Microsoft Hyper-V produce and read
+     * VHD image sizes differently.  VPC will rely on CHS geometry,
+     * while Hyper-V and disk2vhd use the size specified in the footer.
+     *
+     * We use a couple of approaches to try and determine the correct method:
+     * look at the Creator App field, and look for images that have CHS
+     * geometry that is the maximum value.
+     *
+     * If the CHS geometry is the maximum CHS geometry, then we assume that
+     * the size is the footer->current_size to avoid truncation.  Otherwise,
+     * we follow the table based on footer->creator_app:
+     *
+     *  Known creator apps:
+     *      'vpc '  :  CHS              Virtual PC (uses disk geometry)
+     *      'qemu'  :  CHS              QEMU (uses disk geometry)
+     *      'win '  :  current_size     Hyper-V
+     *      'd2v '  :  current_size     Disk2vhd
+     *
+     *  The user can override the table values via drive options, however
+     *  even with an override we will still use current_size for images
+     *  that have CHS geometry of the maximum size.
+     */
+    use_chs = (!!strncmp(footer->creator_app, "win ", 4) &&
+               !!strncmp(footer->creator_app, "d2v ", 4)) || s->force_use_chs;
+
+    if (!use_chs || bs->total_sectors == VHD_MAX_GEOMETRY || s->force_use_sz) {
         bs->total_sectors = be64_to_cpu(footer->current_size) /
-                            BDRV_SECTOR_SIZE;
+                                        BDRV_SECTOR_SIZE;
     }
 
     /* Allow a maximum disk size of approximately 2 TB */
Signature · 1 line
1.9.3
author
#9 lore raw

PATCH 3/4 block/vpc: give option to force the current_size field in .bdrv_create

In reply to Jeff Cody

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

---
 block/vpc.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
block/vpc.c +14 -1
index 54a38a7..48ef16e 100644
@@ -49,6 +49,8 @@ enum vhd_type {
 #define VHD_MAX_SECTORS       (65535LL * 255 * 255)
 #define VHD_MAX_GEOMETRY      (65535LL *  16 * 255)
 
+#define VPC_OPT_FORCE_SIZE "force_size"
+
 // always big-endian
 typedef struct vhd_footer {
     char        creator[8]; // "conectix"
@@ -850,6 +852,7 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
     int64_t total_size;
     int disk_type;
     int ret = -EIO;
+    bool force_size;
     Error *local_err = NULL;
     BlockDriverState *bs = NULL;
 
@@ -870,6 +873,8 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
         disk_type = VHD_DYNAMIC;
     }
 
+    force_size = qemu_opt_get_bool_del(opts, VPC_OPT_FORCE_SIZE, false);
+
     ret = bdrv_create_file(filename, opts, &local_err);
     if (ret < 0) {
         error_propagate(errp, local_err);
@@ -896,7 +901,8 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
         calculate_geometry(total_sectors + i, &cyls, &heads, &secs_per_cyl);
     }
 
-    if ((int64_t)cyls * heads * secs_per_cyl == VHD_MAX_GEOMETRY) {
+    if ((int64_t)cyls * heads * secs_per_cyl == VHD_MAX_GEOMETRY ||
+            force_size) {
         total_sectors = total_size / BDRV_SECTOR_SIZE;
         /* Allow a maximum disk size of approximately 2 TB */
         if (total_sectors > VHD_MAX_SECTORS) {
@@ -994,6 +1000,13 @@ static QemuOptsList vpc_create_opts = {
                 "Type of virtual hard disk format. Supported formats are "
                 "{dynamic (default) | fixed} "
         },
+        {
+            .name = VPC_OPT_FORCE_SIZE,
+            .type = QEMU_OPT_BOOL,
+            .help = "Force disk size calculation to use the actual size "
+                    "specified, rather than using the nearest CHS-based "
+                    "calculation"
+        },
         { /* end of list */ }
     }
 };
Signature · 1 line
1.9.3
author
#2 lore raw

PATCH 2/4 block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

In reply to Jeff Cody

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

---
 tests/qemu-iotests/146                             | 119 +++++++++++++++++++++
 tests/qemu-iotests/146.out                         |  38 +++++++
 tests/qemu-iotests/group                           |   1 +
 .../sample_images/hyperv2012r2-dynamic.vhd.bz2     | Bin 0 -> 214 bytes
 .../sample_images/virtualpc-dynamic.vhd.bz2        | Bin 0 -> 212 bytes
 5 files changed, 158 insertions(+)
 create mode 100755 tests/qemu-iotests/146
 create mode 100644 tests/qemu-iotests/146.out
 create mode 100644 tests/qemu-iotests/sample_images/hyperv2012r2-dynamic.vhd.bz2
 create mode 100644 tests/qemu-iotests/sample_images/virtualpc-dynamic.vhd.bz2
tests/qemu-iotests/146 new file +119 -0
new file mode 100755
index 0000000..af60849
@@ -0,0 +1,119 @@
+#!/bin/bash
+#
+# Test VHD image format creator detection and override
+#
+# Copyright (C) 2016 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1    # failure is the default!
+
+_cleanup()
+{
+    _cleanup_qemu
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt vpc
+_supported_proto file
+_supported_os Linux
+
+
+qemu_comm_method="monitor"
+silent=
+
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Testing VPC with current_size force ===
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=current_size
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Testing VPC with chs force ===
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=chs
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
+_cleanup_test_img
+
+echo
+echo === Testing Hyper-V Autodetect ===
+echo
+_use_sample_img hyperv2012r2-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Testing Hyper-V with current_size force ===
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=current_size
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Testing Hyper-V with chs force ===
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=chs
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
+
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/146.out new file +38 -0
new file mode 100644
index 0000000..ddcfeb6
@@ -0,0 +1,38 @@
+QA output created by 146
+
+=== Testing VPC Autodetect ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266334240/ 266334240 sectors not allocated at offset 0 bytes (0)
+
+=== Testing VPC with current_size force ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266338304/ 266338304 sectors not allocated at offset 0 bytes (0)
+
+=== Testing VPC with chs force ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266334240/ 266334240 sectors not allocated at offset 0 bytes (0)
+
+=== Testing Hyper-V Autodetect ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266338304/ 266338304 sectors not allocated at offset 0 bytes (0)
+
+=== Testing Hyper-V with current_size force ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266338304/ 266338304 sectors not allocated at offset 0 bytes (0)
+
+=== Testing Hyper-V with chs force ===
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) qqeqemqemuqemu-qemu-iqemu-ioqemu-io qemu-io dqemu-io diqemu-io disqemu-io diskqemu-io disk qemu-io disk "qemu-io disk "mqemu-io disk "maqemu-io disk "mapqemu-io disk "map"
+[                       0]  266334240/ 266334240 sectors not allocated at offset 0 bytes (0)
+*** done
tests/qemu-iotests/group +1 -0
index 47fd40c..1211149 100644
@@ -148,3 +148,4 @@
 143 auto quick
 144 rw auto quick
 145 auto quick
+146 auto quick
tests/qemu-iotests/sample_images/hyperv2012r2-dynamic.vhd.bz2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..bfeccf7b9f2596f119bb35d7e650a29f8e6f17ef
GIT binary patch
tests/qemu-iotests/sample_images/virtualpc-dynamic.vhd.bz2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..783be3c8f068b6eb96d777b1290999899c6d5a25
GIT binary patch
Signature · 1 line
1.9.3
author
#18 lore raw

PATCH 4/4 block/vpc: add tests for image creation force_size parameter

In reply to Jeff Cody
---
 tests/qemu-iotests/146     | 81 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/146.out | 44 +++++++++++++++++++++++++
 2 files changed, 125 insertions(+)
tests/qemu-iotests/146 +81 -0
index af60849..323ea8a 100755
@@ -113,6 +113,87 @@ _send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
 _send_qemu_cmd $h1 'quit' ""
 
 
+_cleanup_test_img
+
+echo
+echo === Testing Image create, default ===
+echo
+
+TEST_IMG="${TEST_DIR}/vpc-create-test.vpc"
+
+_make_test_img 4G
+
+echo
+echo === Read created image, default opts ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Read created image, force_size_calc=chs ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=chs
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
+echo
+echo === Read created image, force_size_calc=current_size ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=current_size
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
+echo
+echo === Testing Image create, force_size ===
+echo
+
+_make_test_img -o force_size 4G
+
+
+echo
+echo === Read created image, default opts ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+echo
+echo === Read created image, force_size_calc=chs ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=chs
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
+echo
+echo === Read created image, force_size_calc=current_size ====
+echo
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc,force_size_calc=current_size
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""
+
+
 
 echo "*** done"
 rm -f $seq.full
tests/qemu-iotests/146.out +44 -0

48 lines not on this page · raw message

Signature · 1 line
1.9.3
#10 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

In reply to Jeff Cody

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 12 lines

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

Quoted 5 lines

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

We need to set a different creator string here that makes vpc_open() recognise the image as current_size based.

Kevin

#3 lore raw

Re: [Qemu-devel] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

In reply to Jeff Cody

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

author
#4 lore raw

Re: [Qemu-devel] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

Quoted 25 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

Quoted 3 lines

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

OK, that sounds fine - I'll change that

author
#11 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

In reply to Kevin Wolf
Quoted 15 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 14 lines

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Quoted 8 lines
Quoted 3 lines

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

We need to set a different creator string here that makes vpc_open() recognise the image as current_size based.

Kevin

How about "qem2"? I initially thought about just changing the case on "qemu", but I was afraid some other software may treat the app creator string as case-insensitive.

I'll also update patch 1, to recognize that string as well.

#12 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Peter

Quoted 32 lines

Am 24.02.2016 um 13:24 schrieb Jeff Cody <[email protected]>:

Quoted 16 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 14 lines

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Quoted 8 lines
Quoted 3 lines

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

We need to set a different creator string here that makes vpc_open() recognise the image as current_size based.

Kevin

How about "qem2"? I initially thought about just changing the case on "qemu", but I was afraid some other software may treat the app creator string as case-insensitive.

I'll also update patch 1, to recognize that string as well.

#13 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

Am 24.02.2016 um 13:44 hat Peter Lieven geschrieben:

Quoted 3 lines

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Does disk2vhd do it this way even if the size is smaller than the maximum that can be represented with CHS?

Quoted 25 lines
Quoted 25 lines

Am 24.02.2016 um 13:24 schrieb Jeff Cody <[email protected]>:

Quoted 23 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 12 lines

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

Quoted 5 lines

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

We need to set a different creator string here that makes vpc_open() recognise the image as current_size based.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Kevin

How about "qem2"? I initially thought about just changing the case on "qemu", but I was afraid some other software may treat the app creator string as case-insensitive.

I'll also update patch 1, to recognize that string as well.

I had the same thoughts about our options, and I wasn't fully convinced of either, so I didn't propose any. I was leaning towards the case change, though, as I don't think it should make a difference and it reads nicer. But "qem2" is okay with me, too.

Kevin

author
#14 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

Quoted 8 lines · Kevin Wolf

Am 24.02.2016 um 13:44 hat Peter Lieven geschrieben:

Quoted 3 lines

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Does disk2vhd do it this way even if the size is smaller than the maximum that can be represented with CHS?

I don't know about disk2vhd, but I just created a 5G dynamic VHD image on Hyper-V, and it produced:

cyl: 10402, heads: 16, secs: 63
virtual size: 5.0G (5368709120 bytes)

(the virtual size as calculated by CHS in that case would have been 5368430592 bytes)

I then tested the reverse - I modified qemu to create a VHD image with 5G as the current_size, but maxed out CHS parameters. I imported it into Hyper-V, and it worked fine - just recognized as a 5G disk with 5368709120 bytes.

But with all that, it seems like it may be better to mimic the Hyper-V behavior, and use a new creator app string, with the normal CHS values.

Quoted 39 lines
Quoted 25 lines
Quoted 25 lines

Am 24.02.2016 um 13:24 schrieb Jeff Cody <[email protected]>:

Quoted 23 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 12 lines

When QEMU creates a VHD image, it goes by the original spec, calculating the current_size based on the nearest CHS geometry (with an exception for disks > 127GB).

Apparently, Azure will only allow images that are sized to the nearest MB, and the current_size as calculated from CHS cannot guarantee that.

Allow QEMU to create images similar to how Hyper-V creates images, by setting current_size to the specified virtual disk size. This introduces an option, force_size, to be passed to the vpc format during image creation, e.g.:

Quoted 5 lines

qemu-img convert -f raw -o force_size -O vpc test.img test.vhd

Bug reference: https://bugs.launchpad.net/qemu/+bug/1490611

We need to set a different creator string here that makes vpc_open() recognise the image as current_size based.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Kevin

How about "qem2"? I initially thought about just changing the case on "qemu", but I was afraid some other software may treat the app creator string as case-insensitive.

I'll also update patch 1, to recognize that string as well.

I had the same thoughts about our options, and I wasn't fully convinced of either, so I didn't propose any. I was leaning towards the case change, though, as I don't think it should make a difference and it reads nicer. But "qem2" is okay with me, too.

Kevin

author
#5 lore raw

Re: [Qemu-devel] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

In reply to Kevin Wolf
Quoted 25 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

Quoted 3 lines

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

The reason I used a qemu process before, was difficulty passing in the drive options to qemu-io. I futzed around a bit with the new --image-opts, but I hadn't tried "file=" in the open command with -o, so I felt a bit silly after your email.

However, this doesn't seem to work, now that I've tried it. Is it broken, or am I doing something wrong? Here is what I get:

# ./qemu-io -c "open -o file=/tmp/hyperv2012r2-dynamic.vhd,format=vpc" -c "map"
can't open: Cannot find device=/tmp/hyperv2012r2-dynamic.vhd nor node_name=/tmp/hyperv2012r2-dynamic.vhd

Technically, I could just rely on image format autodetection since the current test images are dynamic and not fixed, and use -o to pass the vpc specific options. I just hate to rely on autodetection anymore.

-Jeff

#6 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

Quoted 35 lines · Jeff Cody
Quoted 25 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

Quoted 3 lines

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

The reason I used a qemu process before, was difficulty passing in the drive options to qemu-io. I futzed around a bit with the new --image-opts, but I hadn't tried "file=" in the open command with -o, so I felt a bit silly after your email.

Quoted 5 lines

However, this doesn't seem to work, now that I've tried it. Is it broken, or am I doing something wrong? Here is what I get:

# ./qemu-io -c "open -o file=/tmp/hyperv2012r2-dynamic.vhd,format=vpc" -c "map"
can't open: Cannot find device=/tmp/hyperv2012r2-dynamic.vhd nor node_name=/tmp/hyperv2012r2-dynamic.vhd

Try:

$QEMU_IO -c "open -o driver=vpc ${TEST_IMG}"

Max

Quoted 7 lines

Technically, I could just rely on image format autodetection since the current test images are dynamic and not fixed, and use -o to pass the vpc specific options. I just hate to rely on autodetection anymore.

-Jeff

author
#7 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

Quoted 39 lines · Max Reitz
Quoted 38 lines · Jeff Cody
Quoted 25 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

Quoted 3 lines

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

The reason I used a qemu process before, was difficulty passing in the drive options to qemu-io. I futzed around a bit with the new --image-opts, but I hadn't tried "file=" in the open command with -o, so I felt a bit silly after your email.

However, this doesn't seem to work, now that I've tried it. Is it broken, or am I doing something wrong? Here is what I get:

Quoted 8 lines
Quoted 2 lines
# ./qemu-io -c "open -o file=/tmp/hyperv2012r2-dynamic.vhd,format=vpc" -c "map"
can't open: Cannot find device=/tmp/hyperv2012r2-dynamic.vhd nor node_name=/tmp/hyperv2012r2-dynamic.vhd

Try:

$QEMU_IO -c "open -o driver=vpc ${TEST_IMG}"

Max

Thanks!

Quoted 10 lines
Quoted 8 lines

Technically, I could just rely on image format autodetection since the current test images are dynamic and not fixed, and use -o to pass the vpc specific options. I just hate to rely on autodetection anymore.

-Jeff

#8 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH 2/4] block/vpc: tests for auto-detecting VPC and Hyper-V VHD images

In reply to Jeff Cody
Quoted 38 lines · Jeff Cody
Quoted 25 lines · Kevin Wolf

Am 24.02.2016 um 01:47 hat Jeff Cody geschrieben:

Quoted 10 lines

This tests auto-detection, and overrides, of VHD image sizes created by Virtual PC and Hyper-V.

This adds two sample images:

hyperv2012r2-dynamic.vhd.bz2 - dynamic VHD image created with Hyper-V
virtualpc-dynamic.vhd.bz2 - dynamic VHD image created with Virtual PC

tests/qemu-iotests/146 line 59

Quoted 10 lines
tests/qemu-iotests/146 new file +10 -0
+echo
+echo === Testing VPC Autodetect ===
+echo
+_use_sample_img virtualpc-dynamic.vhd.bz2
+
+_launch_qemu -drive file="${TEST_IMG}",id=disk,format=vpc
+h1=$QEMU_HANDLE
+
+_send_qemu_cmd $h1 'qemu-io disk "map"' "sectors"
+_send_qemu_cmd $h1 'quit' ""

I would avoid the big hammer of starting qemu processes when qemu-io can test the same:

Quoted 3 lines

$QEMU_IO -c "open -o file=${TEST_IMG},id=disk,format=vpc" -c "map"

Kevin

The reason I used a qemu process before, was difficulty passing in the drive options to qemu-io. I futzed around a bit with the new --image-opts, but I hadn't tried "file=" in the open command with -o, so I felt a bit silly after your email.

However, this doesn't seem to work, now that I've tried it. Is it broken, or am I doing something wrong? Here is what I get:

Quoted 6 lines
# ./qemu-io -c "open -o file=/tmp/hyperv2012r2-dynamic.vhd,format=vpc" -c "map"
can't open: Cannot find device=/tmp/hyperv2012r2-dynamic.vhd nor node_name=/tmp/hyperv2012r2-dynamic.vhd

Technically, I could just rely on image format autodetection since the current test images are dynamic and not fixed, and use -o to pass the vpc specific options. I just hate to rely on autodetection anymore.

And I completely skipped this block...

$QEMU_IO resolves to _qemu_io_wrapper; this function will eventually invoke "$QEMU_IO_PROG" $QEMU_IO_OPTIONS; and $QEMU_IO_OPTIONS resolves to "-f $IMGFMT --cache $CACHEMODE". Therefore, you don't need to worry about autodetection anyway.

Max

#15 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

In reply to Jeff Cody

Am 24.02.2016 um 14:40 schrieb Jeff Cody:

Quoted 14 lines · Kevin Wolf
Quoted 7 lines

Am 24.02.2016 um 13:44 hat Peter Lieven geschrieben:

Quoted 3 lines

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Does disk2vhd do it this way even if the size is smaller than the maximum that can be represented with CHS?

I don't know about disk2vhd, but I just created a 5G dynamic VHD image on Hyper-V, and it produced:

cyl: 10402, heads: 16, secs: 63
virtual size: 5.0G (5368709120 bytes)

Quoted 7 lines

(the virtual size as calculated by CHS in that case would have been 5368430592 bytes)

I then tested the reverse - I modified qemu to create a VHD image with 5G as the current_size, but maxed out CHS parameters. I imported it into Hyper-V, and it worked fine - just recognized as a 5G disk with 5368709120 bytes.

So the idea to set CHS to MAX is not that bad.

Quoted 4 lines

But with all that, it seems like it may be better to mimic the Hyper-V behavior, and use a new creator app string, with the normal CHS values.

But this means that it is not backwards compatible. Maxing out CHS when forcing the size would mean all old qemu version would look at current size and not CHS.

Might it be that Hyper-V and others use CHS when an ATA disk is emulated and the cur_size otherwise? I think this is what virtualbox does.

Peter

#17 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

In reply to Kevin Wolf

Am 24.02.2016 um 14:07 schrieb Kevin Wolf:

Quoted 6 lines

Am 24.02.2016 um 13:44 hat Peter Lieven geschrieben:

Quoted 3 lines

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Does disk2vhd do it this way even if the size is smaller than the maximum that can be represented with CHS?

Afaik it always sets it to MAX. This is why we first introduced a check for the d2v creator in Qemu because all VHD Images created by disk2vhd where 127GB regardless of their size.

Peter

author
#16 lore raw

Re: [Qemu-devel] [PATCH 3/4] block/vpc: give option to force the current_size field in .bdrv_create

In reply to Peter Lieven
Quoted 18 lines · Peter Lieven

Am 24.02.2016 um 14:40 schrieb Jeff Cody:

Quoted 17 lines · Kevin Wolf
Quoted 7 lines

Am 24.02.2016 um 13:44 hat Peter Lieven geschrieben:

Quoted 3 lines

if the size is forced I would set the chs values to max. this way no new creator String is needed and it is even backwards compatible. this is what disk2vhd does.

Does disk2vhd do it this way even if the size is smaller than the maximum that can be represented with CHS?

I don't know about disk2vhd, but I just created a 5G dynamic VHD image on Hyper-V, and it produced:

cyl: 10402, heads: 16, secs: 63
virtual size: 5.0G (5368709120 bytes)

(the virtual size as calculated by CHS in that case would have been 5368430592 bytes)

Quoted 7 lines
Quoted 4 lines

I then tested the reverse - I modified qemu to create a VHD image with 5G as the current_size, but maxed out CHS parameters. I imported it into Hyper-V, and it worked fine - just recognized as a 5G disk with 5368709120 bytes.

So the idea to set CHS to MAX is not that bad.

I agree. I just did a test with disk2vhd, against an empty (but formatted) 7.5G usb drive. Here is what it created:

# ./qemu-img info /mnt/nfs-2/tmp/WINtsts2.VHD
cyls: 65535, heads: 16, sectors: 255
image: /mnt/nfs-2/tmp/WINtsts2.VHD
file format: vpc
virtual size: 7.5G (8095006720 bytes)

I really like how one company has 3 different tools that all handle this differently.

Quoted 9 lines
Quoted 4 lines

But with all that, it seems like it may be better to mimic the Hyper-V behavior, and use a new creator app string, with the normal CHS values.

But this means that it is not backwards compatible. Maxing out CHS when forcing the size would mean all old qemu version would look at current size and not CHS.

That is a valid concern. So, how about this:

With 'force_size' during image create, set the CHS parameters to MAX (as d2v does), but also set the creator app to 'qem2'. This at least gives other software a chance to learn that new creator app and handle it differently, if needed (and it shouldn't hurt anything, and will still be backwards compatible).

Quoted 2 lines

Might it be that Hyper-V and others use CHS when an ATA disk is emulated and the cur_size otherwise? I think this is what virtualbox does.

I tested the Hyper-V created image under a Linux VM in Hyper-V. The VM is using IDE controllers, and the image size was the current_size, not the CHS calculation.

Jeff