Skip to contentSkip to conversation

Introducing qcow2 extensions + keep backing file format (v4)

PATCH In review Uri Lublin 2 participants 5 messages Feb 18, 2009 – Mar 1, 2009

lore
Share & tools
lore

Uri Lublin

author
#0 lore raw

cover Introducing qcow2 extensions + keep backing file format (v4)

This patch set is an implementation of keeping a backing file format, such that we would not need probing to figure it out.

I introduce qcow2 extensions which can be used in the future. I use one such an extension to keep the backing file format, and use it when opening a file.

Qcow2 extensions are built of magic (id) len (in bytes) and data (len bytes). They reside between the end of the header and the filename.

We keep the backing file format in a such a qcow2 extension, to
1. Provide a way to know the backing file format without probing it (setting the format at creation time).
2. Enable using qcow2 format over host block devices.
(only if the user specifically asks for it, by providing the format
at image creation time).

This patchset is backward compatible in two ways:
1. It does not break old images (no extensions -> probe).
2. New images can be opened by old Qemu (extensions skipped -> probe).

I've added bdrv_create2 and drv->bdrv_create2 (implemented only by block-qcow2 currently) to pass the backing-format to create.

Based on a work done by Shahar Frank.

Also fixes a security flaw found by Daniel P. Berrange on [1] which summarizes: "Autoprobing: just say no."

[1] http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg01083.html

First patch implements qcow2 extensions + bdrv_create2 Second patch modifies qemu-img such that the user can use it.

Currently this is implemented only for qcow2.

Changes from v3:
Rebased to current HEAD.
Removed some unneeded comments.

Thanks,
Uri.

author
#1 lore raw

PATCH 1/2 Introducing qcow2 extensions + keep backing file format (v4)

Qcow2 extensions are build of magic (id) len (in bytes) and data. They reside between the end of the header and the filename.

We can keep the backing file format in a such a qcow2 extension, to
1. Provide a way to know the backing file format without probing it (setting the format at creation time).
2. Enable using qcow2 format over host block devices.
(only if the user specifically asks for it, by providing the format at creation time).

I've added bdrv_create2 and drv->bdrv_create2 (implemented only by block-qcow2 currently) to pass the backing-format to create.

Based on a work done by Shahar Frank.

Also fixes a security flaw found by Daniel P. Berrange on [1] which summarizes: "Autoprobing: just say no."

[1] http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg01083.html

---
 block-qcow2.c |  108 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 block.c       |   29 ++++++++++++++--
 block.h       |    4 ++
 block_int.h   |    6 +++
 4 files changed, 142 insertions(+), 5 deletions(-)
block-qcow2.c +106 -2
index 465dcd6..222d423 100644
@@ -45,6 +45,7 @@
 
 //#define DEBUG_ALLOC
 //#define DEBUG_ALLOC2
+//#define DEBUG_EXT
 
 #define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
 #define QCOW_VERSION 2
@@ -77,6 +78,14 @@ typedef struct QCowHeader {
     uint64_t snapshots_offset;
 } QCowHeader;
 
+
+typedef struct {
+    uint32_t magic;
+    uint32_t len;
+} QCowExtension;
+#define  QCOW_EXT_MAGIC_END 0
+#define  QCOW_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
+
 typedef struct __attribute__((packed)) QCowSnapshotHeader {
     /* header is 8 byte aligned */
     uint64_t l1_table_offset;
@@ -189,6 +198,66 @@ static int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
         return 0;
 }
 
+
+/* read qcow2 extension and fill bs
+ * start reading from start_offset
+ * finish reading upon magic of value 0 or when end_offset reached
+ * unknown magic is skipped (future extension this version knows nothing about)
+ * return 0 upon success, non-0 otherwise
+ */
+static int qcow_read_extensions(BlockDriverState *bs, uint64_t start_offset,
+                                uint64_t end_offset)
+{
+    BDRVQcowState *s = bs->opaque;
+    QCowExtension ext;
+    uint64_t offset;
+
+#ifdef DEBUG_EXT
+    printf("qcow_read_extensions: start=%ld end=%ld\n", start_offset, end_offset);
+#endif
+    offset = start_offset;
+    while (offset < end_offset) {
+
+#ifdef DEBUG_EXT
+        /* Sanity check */
+        if (offset >= s->cluster_size)
+            printf("qcow_handle_extension: suspicious offset %lu\n", offset);
+
+        printf("attemting to read extended header in offset %lu\n", offset);
+#endif
+
+        if (bdrv_pread(s->hd, offset, &ext, sizeof(ext)) != sizeof(ext)) {
+            fprintf(stderr, "qcow_handle_extension: ERROR: pread fail from offset %lu\n",
+                    offset);
+            return 1;
+        }
+        be32_to_cpus(&ext.magic);
+        be32_to_cpus(&ext.len);
+        offset += sizeof(ext);
+        switch (ext.magic) {
+        case QCOW_EXT_MAGIC_END:
+            break;
+        case QCOW_EXT_MAGIC_BACKING_FORMAT:
+            if (ext.len >= sizeof(bs->backing_format)) {
+                fprintf(stderr, "ERROR: ext_backing_format: len=%u too large"
+                        " (>=%lu)\n",
+                        ext.len, sizeof(bs->backing_format));
+                return 2;
+            }
+            if (bdrv_pread(s->hd, offset , bs->backing_format,
+                           ext.len) != ext.len)
+                return 3;
+            bs->backing_format[ext.len] = '\0';
+#ifdef DEBUG_EXT
+            printf("Qcow2: Got format extension %s\n", bs->backing_format);
+#endif
+            offset += ((ext.len + 7) & ~7);
+        }
+    }
+
+    return 0;
+}
+
 static int qcow_open(BlockDriverState *bs, const char *filename, int flags)
 {
     BDRVQcowState *s = bs->opaque;
@@ -278,6 +347,10 @@ static int qcow_open(BlockDriverState *bs, const char *filename, int flags)
 
     scan_refcount(bs, &s->highest_alloc, &s->nc_free);
 
+    /* read qcow2 extensions */
+    if (qcow_read_extensions(bs, sizeof(header), header.backing_file_offset))
+        goto fail;
+
     /* read the backing file name */
     if (header.backing_file_offset != 0) {
         len = header.backing_file_size;
@@ -1462,13 +1535,17 @@ static void create_refcount_update(QCowCreateState *s,
     }
 }
 
-static int qcow_create(const char *filename, int64_t total_size,
-                      const char *backing_file, int flags)
+static int qcow_create2(const char *filename, int64_t total_size,
+                        const char *backing_file, const char *backing_format,
+                        int flags)
 {
     int fd, header_size, backing_filename_len, l1_size, i, shift, l2_bits;
+    int backing_format_len = 0;
     QCowHeader header;
     uint64_t tmp, offset;
     QCowCreateState s1, *s = &s1;
+    QCowExtension ext = {0, 0};
+
 
     memset(s, 0, sizeof(*s));
 
@@ -1482,6 +1559,12 @@ static int qcow_create(const char *filename, int64_t total_size,
     header_size = sizeof(header);
     backing_filename_len = 0;
     if (backing_file) {
+        if (backing_format) {
+            ext.magic = QCOW_EXT_MAGIC_BACKING_FORMAT;
+            backing_format_len = strlen(backing_format);
+            ext.len = (backing_format_len + 7) & ~7;
+            header_size += ((sizeof(ext) + ext.len + 7) & ~7);
+        }
         header.backing_file_offset = cpu_to_be64(header_size);
         backing_filename_len = strlen(backing_file);
         header.backing_file_size = cpu_to_be32(backing_filename_len);
@@ -1526,6 +1609,19 @@ static int qcow_create(const char *filename, int64_t total_size,
     /* write all the data */
     write(fd, &header, sizeof(header));
     if (backing_file) {
+        if (backing_format_len) {
+            char zero[16];
+            int d = ext.len - backing_format_len;
+
+            memset(zero, 0, sizeof(zero));
+            cpu_to_be32s(&ext.magic);
+            cpu_to_be32s(&ext.len);
+            write(fd, &ext, sizeof(ext));
+            write(fd, backing_format, backing_format_len);
+            if (d>0) {
+                write(fd, zero, d);
+            }
+        }
         write(fd, backing_file, backing_filename_len);
     }
     lseek(fd, s->l1_table_offset, SEEK_SET);
@@ -1545,6 +1641,12 @@ static int qcow_create(const char *filename, int64_t total_size,
     return 0;
 }
 
+static int qcow_create(const char *filename, int64_t total_size,
+                       const char *backing_file, int flags)
+{
+    return qcow_create2(filename, total_size, backing_file, NULL, flags);
+}
+
 static int qcow_make_empty(BlockDriverState *bs)
 {
 #if 0
@@ -2670,4 +2772,6 @@ BlockDriver bdrv_qcow2 = {
     .bdrv_snapshot_delete = qcow_snapshot_delete,
     .bdrv_snapshot_list = qcow_snapshot_list,
     .bdrv_get_info = qcow_get_info,
+
+    .bdrv_create2 = qcow_create2,
 };
block.c +26 -3
index 4f4bf7c..31d6d5b 100644
@@ -172,6 +172,20 @@ BlockDriver *bdrv_find_format(const char *format_name)
     return NULL;
 }
 
+int bdrv_create2(BlockDriver *drv,
+                const char *filename, int64_t size_in_sectors,
+                const char *backing_file, const char *backing_format,
+                int flags)
+{
+    if (drv->bdrv_create2)
+        return drv->bdrv_create2(filename, size_in_sectors, backing_file,
+                                 backing_format, flags);
+    if (drv->bdrv_create)
+        return drv->bdrv_create(filename, size_in_sectors, backing_file,
+                                flags);
+    return -ENOTSUP;
+}
+
 int bdrv_create(BlockDriver *drv,
                 const char *filename, int64_t size_in_sectors,
                 const char *backing_file, int flags)
@@ -348,6 +362,9 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
 
         /* if there is a backing file, use it */
         bs1 = bdrv_new("");
+/*         if (drv) */
+/*             pstrcpy(bs1->backing_format, sizeof(bs->backing_format), */
+/*                     drv->format_name); */
         if (!bs1) {
             return -ENOMEM;
         }
@@ -371,8 +388,10 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
         else
             realpath(filename, backing_filename);
 
-        if (bdrv_create(&bdrv_qcow2, tmp_filename,
-                        total_size, backing_filename, 0) < 0) {
+        if (bdrv_create2(&bdrv_qcow2, tmp_filename,
+                         total_size, backing_filename,
+                         (drv ? drv->format_name : NULL),
+                         0) < 0) {
             return -1;
         }
         filename = tmp_filename;
@@ -420,6 +439,7 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
 #endif
     if (bs->backing_file[0] != '\0') {
         /* if there is a backing file, use it */
+        BlockDriver *back_drv = NULL;
         bs->backing_hd = bdrv_new("");
         if (!bs->backing_hd) {
         fail:
@@ -428,7 +448,10 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
         }
         path_combine(backing_filename, sizeof(backing_filename),
                      filename, bs->backing_file);
-        if (bdrv_open(bs->backing_hd, backing_filename, open_flags) < 0)
+        if (bs->backing_format)
+            back_drv = bdrv_find_format(bs->backing_format);
+        if (bdrv_open2(bs->backing_hd, backing_filename, open_flags,
+                       back_drv) < 0)
             goto fail;
     }
 
block.h +4 -0
index e1927dd..e19fd8f 100644
@@ -64,6 +64,10 @@ BlockDriver *bdrv_find_format(const char *format_name);
 int bdrv_create(BlockDriver *drv,
                 const char *filename, int64_t size_in_sectors,
                 const char *backing_file, int flags);
+int bdrv_create2(BlockDriver *drv,
+                 const char *filename, int64_t size_in_sectors,
+                 const char *backing_file, const char *backing_format,
+                 int flags);
 BlockDriverState *bdrv_new(const char *device_name);
 void bdrv_delete(BlockDriverState *bs);
 int bdrv_file_open(BlockDriverState **pbs, const char *filename, int flags);
block_int.h +6 -0
index 781789c..8c373ff 100644
@@ -85,6 +85,11 @@ struct BlockDriver {
     /* to control generic scsi devices */
     int (*bdrv_ioctl)(BlockDriverState *bs, unsigned long int req, void *buf);
 
+    /* new create with backing file format */
+    int (*bdrv_create2)(const char *filename, int64_t total_sectors,
+                        const char *backing_file, const char *backing_format,
+                        int flags);
+
     BlockDriverAIOCB *free_aiocb;
     struct BlockDriver *next;
 };
@@ -107,6 +112,7 @@ struct BlockDriverState {
     char filename[1024];
     char backing_file[1024]; /* if non zero, the image is a diff of
                                 this file image */
+    char backing_format[16]; /* if non-zero and backing_file exists */
     int is_temporary;
     int media_changed;
 
Signature · 1 line
1.6.0.6
author
#2 lore raw

PATCH 2/2 qemu-img: adding a "-F base_fmt" option to "qemu-img create -b" (v4)

If the user specifies the backing file format, then when opening the backing file, there is no need to probe the (backing file) image to figure out its format.

This follows my previous patch implementing bdrv_create2 which keeps (for qcow2 only) the backing file format as a qcow2-extension

Suggested by Daniel P. Berrange.

---
 qemu-img.c    |   24 ++++++++++++++++++++----
 qemu-img.texi |    4 +++-
 2 files changed, 23 insertions(+), 5 deletions(-)
qemu-img.c +20 -4
index d83ffe3..2cf0e1f 100644
@@ -25,6 +25,7 @@
 #include "osdep.h"
 #include "block_int.h"
 #include <assert.h>
+#include <stdio.h>
 
 #ifdef _WIN32
 #define WIN32_LEAN_AND_MEAN
@@ -58,7 +59,7 @@ static void help(void)
            "QEMU disk image utility\n"
            "\n"
            "Command syntax:\n"
-           "  create [-e] [-6] [-b base_image] [-f fmt] filename [size]\n"
+           "  create [-e] [-6] [-F fmt] [-b base_image] [-f fmt] filename [size]\n"
            "  commit [-f fmt] filename\n"
            "  convert [-c] [-e] [-6] [-f fmt] [-O output_fmt] [-B output_base_image] filename [filename2 [...]] output_filename\n"
            "  info [-f fmt] filename\n"
@@ -218,6 +219,7 @@ static int img_create(int argc, char **argv)
 {
     int c, ret, flags;
     const char *fmt = "raw";
+    const char *base_fmt = NULL;
     const char *filename;
     const char *base_filename = NULL;
     uint64_t size;
@@ -226,13 +228,16 @@ static int img_create(int argc, char **argv)
 
     flags = 0;
     for(;;) {
-        c = getopt(argc, argv, "b:f:he6");
+        c = getopt(argc, argv, "F:b:f:he6");
         if (c == -1)
             break;
         switch(c) {
         case 'h':
             help();
             break;
+        case 'F':
+            base_fmt = optarg;
+            break;
         case 'b':
             base_filename = optarg;
             break;
@@ -253,7 +258,15 @@ static int img_create(int argc, char **argv)
     size = 0;
     if (base_filename) {
         BlockDriverState *bs;
-        bs = bdrv_new_open(base_filename, NULL);
+        BlockDriver *base_drv = NULL;
+        
+        if (base_fmt) {
+            base_drv = bdrv_find_format(base_fmt);
+            if (base_drv == NULL)
+                error("Unknown basefile format '%s'", base_fmt);
+        }
+        
+        bs = bdrv_new_open(base_filename, base_fmt);
         bdrv_get_geometry(bs, &size);
         size *= 512;
         bdrv_delete(bs);
@@ -284,9 +297,12 @@ static int img_create(int argc, char **argv)
     if (base_filename) {
         printf(", backing_file=%s",
                base_filename);
+         if (base_fmt)
+             printf(", backing_fmt=%s",
+                    base_fmt);
     }
     printf(", size=%" PRIu64 " kB\n", size / 1024);
-    ret = bdrv_create(drv, filename, size / 512, base_filename, flags);
+    ret = bdrv_create2(drv, filename, size / 512, base_filename, base_fmt, flags);
     if (ret < 0) {
         if (ret == -ENOTSUP) {
             error("Formatting or formatting option not supported for file format '%s'", fmt);
qemu-img.texi +3 -1
index a40f841..deef2ab 100644
@@ -8,7 +8,7 @@ usage: qemu-img command [command options]
 
 The following commands are supported:
 @table @option
-@item create [-e] [-6] [-b @var{base_image}] [-f @var{fmt}] @var{filename} [@var{size}]
+@item create [-e] [-6] [-F @var{base_fmt}] [-b @var{base_image}] [-f @var{fmt}] @var{filename} [@var{size}]
 @item commit [-f @var{fmt}] @var{filename}
 @item convert [-c] [-e] [-6] [-f @var{fmt}] [-O @var{output_fmt}] [-B @var{output_base_image}] @var{filename} [@var{filename2} [...]] @var{output_filename}
 @item info [-f @var{fmt}] @var{filename}
@@ -27,6 +27,8 @@ forces the output image to be created as a copy on write
 image of the specified base image; @code{output_base_image} should have the same
 content as the input's base image, however the path, image format, etc may
 differ
+@item base_fmt
+is the disk image format of @var{base_image}. for more information look at @var{fmt}
 @item fmt
 is the disk image format. It is guessed automatically in most cases. The following formats are supported:
 
Signature · 1 line
1.6.0.6
#3 lore raw
In reply to Uri Lublin
Quoted 13 lines · Uri Lublin

Qcow2 extensions are build of magic (id) len (in bytes) and data. They reside between the end of the header and the filename.

We can keep the backing file format in a such a qcow2 extension, to
1. Provide a way to know the backing file format without probing it (setting the format at creation time).
2. Enable using qcow2 format over host block devices.
(only if the user specifically asks for it, by providing the format at creation time).

I've added bdrv_create2 and drv->bdrv_create2 (implemented only by block-qcow2 currently) to pass the backing-format to create.

Quoted 9 lines

Based on a work done by Shahar Frank.

Also fixes a security flaw found by Daniel P. Berrange on [1] which summarizes: "Autoprobing: just say no."

[1] http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg01083.html

This made it through my regression testing but...

block.c int bdrv_open2(BlockDriverState *bs, const char *filename, int flags, · line 368

Quoted 11 lines

int bdrv_create(BlockDriver *drv,
const char *filename, int64_t size_in_sectors, const char *backing_file, int flags)

block.c +3 -0
@@ -348,6 +362,9 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
 
         /* if there is a backing file, use it */
         bs1 = bdrv_new("");
+/*         if (drv) */
+/*             pstrcpy(bs1->backing_format, sizeof(bs->backing_format), */
+/*                     drv->format_name); */
 

I don't know if I missed this before or it was added since v3 but I'm curious why this is here and why it's commented out.

Regards,

Anthony Liguori

author
#4 lore raw
Quoted 32 lines · Anthony Liguori
Quoted 13 lines · Uri Lublin

Qcow2 extensions are build of magic (id) len (in bytes) and data. They reside between the end of the header and the filename.

We can keep the backing file format in a such a qcow2 extension, to
1. Provide a way to know the backing file format without probing it (setting the format at creation time).
2. Enable using qcow2 format over host block devices.
(only if the user specifically asks for it, by providing the format at creation time).

I've added bdrv_create2 and drv->bdrv_create2 (implemented only by block-qcow2 currently) to pass the backing-format to create.

Quoted 9 lines

Based on a work done by Shahar Frank.

Also fixes a security flaw found by Daniel P. Berrange on [1] which summarizes: "Autoprobing: just say no."

[1] http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg01083.html

This made it through my regression testing but...

Quoted 6 lines

int bdrv_create(BlockDriver *drv,
const char *filename, int64_t size_in_sectors, const char *backing_file, int flags)

(quoted diff) +0 -0
@@ -348,6 +362,9 @@ int bdrv_open2(BlockDriverState *bs, const char 
 *filename, int flags,
 
Quoted 10 lines
Quoted 7 lines
(quoted diff) +2 -0
         /* if there is a backing file, use it */
         bs1 = bdrv_new("");
+/*         if (drv) */
+/*             pstrcpy(bs1->backing_format, 

sizeof(bs->backing_format), */
+/* drv->format_name); */

I don't know if I missed this before or it was added since v3 but I'm curious why this is here and why it's commented out.

It's an old code, which should have been deleted.

The purpose was to make sure the temporary-snapshot-file backing-file-format is known.
This is now done with bdrv_create2, that I've added.

You can delete it yourself or apply the attached patch (or squash patches) Alternatively I can send it to you again as a single patch (v5)

Also I've noticed that a small fix is needed for BDRV_O_SNAPSHOT. e.g we should use bdrv_open2 (so we would not probe the image if the format is known), so I'm attaching a patch to fix that too. Again if you want I can resend it as a single patch (squashed with v4) or you can apply(squash) this patch yourself on top of my previous patches.

Thanks,
Uri.

Quoted 1 line

From 5003cc362793c10f276ab186026b3f25b6b3ac9e Mon Sep 17 00:00:00 2001

From: Uri Lublin <[email protected]>
Date: Sun, 1 Mar 2009 12:43:45 +0200
Subject: [PATCH] block.c: remove dead commented out code

---
 block.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
block.c +0 -3
index 31d6d5b..1626e0c 100644
@@ -362,9 +362,6 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
 
         /* if there is a backing file, use it */
         bs1 = bdrv_new("");
-/*         if (drv) */
-/*             pstrcpy(bs1->backing_format, sizeof(bs->backing_format), */
-/*                     drv->format_name); */
         if (!bs1) {
             return -ENOMEM;
         }
Signature · 1 line
1.6.0.6
Quoted 1 line

From bc79eae6c3226d21bd3fcbc370a86d7b24657aad Mon Sep 17 00:00:00 2001

From: Uri Lublin <[email protected]>
Date: Sun, 1 Mar 2009 12:44:38 +0200
Subject: [PATCH] block.c: If image format is known, use it for BDRV_O_SNAPSHOT flag as well

Otherwise we'd
1. Unnecessarily probe the image to find out its format 2 Get it wrong for host-devices and non-raw images, which may affect created image size.

Also, force the temporary snapshot image to be of format bdrv_qcow2

---
 block.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
block.c +2 -1
index 1626e0c..0939cad 100644
@@ -365,7 +365,7 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
         if (!bs1) {
             return -ENOMEM;
         }
-        if (bdrv_open(bs1, filename, 0) < 0) {
+        if (bdrv_open2(bs1, filename, 0, drv) < 0) {
             bdrv_delete(bs1);
             return -1;
         }
@@ -392,6 +392,7 @@ int bdrv_open2(BlockDriverState *bs, const char *filename, int flags,
             return -1;
         }
         filename = tmp_filename;
+        drv = &bdrv_qcow2;
         bs->is_temporary = 1;
     }
 
Signature · 1 line
1.6.0.6