Skip to contentSkip to conversation

Initial VHDX support

PATCH Superseded Jeff Cody 5 participants 30 messages Apr 23, 2013 – Apr 29, 2013 Block layer

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jeff Cody

author
#0 lore raw

v2 cover Initial VHDX support

This adds the initial support for VHDX image files.

It currently only supports read operations of VHDX, for fixed and dynamic files.

Notably, the following is not yet supported:
* Differencing files
* Log replay (so we will refuse to open any images that are not 'clean')
* .bdrv_create()
* write operations other than to the header

Jeff Cody (5):
qemu: add castagnoli crc32c checksum algorithm
block: vhdx header for the QEMU support of VHDX images
block: initial VHDX driver support framework - supports open and probe
block: add read-only support to VHDX image format.
block: add header update capability for VHDX images.

 block/Makefile.objs   |    1 +
 block/vhdx.c          | 1041 +++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h          |  352 +++++++++++++++++
 configure             |   13 +
 include/qemu/crc32c.h |   35 ++
 util/Makefile.objs    |    1 +
 util/crc32c.c         |  115 ++++++
 7 files changed, 1558 insertions(+)
 create mode 100644 block/vhdx.c
 create mode 100644 block/vhdx.h
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
Signature · 1 line
1.8.1.4
author
#1 lore raw

PATCH v2 1/5 qemu: add castagnoli crc32c checksum algorithm

This adds the Castagnoli CRC32C algorithm, using the 0x11EDC6F41 polynomial.

This is extracted from the linux kernel cryptographic crc32.c module.

The algorithm is based on:

Castagnoli93: Guy Castagnoli and Stefan Braeuer and Martin Herrman
"Optimization of Cyclic Redundancy-Check Codes with 24
and 32 Parity Bits", IEEE Transactions on Communication, Volume 41, Number 6, June 1993

---
 include/qemu/crc32c.h |  35 +++++++++++++++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 151 insertions(+)
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
include/qemu/crc32c.h new file +35 -0
new file mode 100644
index 0000000..56d1c3b
@@ -0,0 +1,35 @@
+/*
+ *  Castagnoli CRC32C Checksum Algorithm
+ *
+ *  Polynomial: 0x11EDC6F41
+ *
+ *  Castagnoli93: Guy Castagnoli and Stefan Braeuer and Martin Herrman
+ *               "Optimization of Cyclic Redundancy-Check Codes with 24
+ *                 and 32 Parity Bits",IEEE Transactions on Communication,
+ *                Volume 41, Number 6, June 1993
+ *
+ *  Copyright (c) 2013 Red Hat, Inc.,
+ *
+ *  Authors:
+ *   Jeff Cody <[email protected]>
+ *
+ *  Based on the Linux kernel cryptographic crc32c module,
+ *
+ *  Copyright (c) 2004 Cisco Systems, Inc.
+ *  Copyright (c) 2008 Herbert Xu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+
+#ifndef QEMU_CRC32_H
+#define QEMU_CRC32_H
+
+#include "qemu-common.h"
+
+uint32_t crc32c(uint32_t crc, const uint8_t *data, unsigned int length);
+
+#endif
util/Makefile.objs +1 -0
index c5652f5..4a1bd4e 100644
@@ -10,3 +10,4 @@ util-obj-$(CONFIG_POSIX) += compatfd.o
 util-obj-y += iov.o aes.o qemu-config.o qemu-sockets.o uri.o notify.o
 util-obj-y += qemu-option.o qemu-progress.o
 util-obj-y += hexdump.o
+util-obj-y += crc32c.o
util/crc32c.c new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
1.8.1.4
author
#2 lore raw

PATCH v2 2/5 block: vhdx header for the QEMU support of VHDX images

In reply to Jeff Cody

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 327 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 block/vhdx.h
block/vhdx.h new file +327 -0
new file mode 100644
index 0000000..f5cf1ed
@@ -0,0 +1,327 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_VHDX_H
+#define BLOCK_VHDX_H
+
+/* Structures and fields present in the VHDX file */
+
+/* The header section has the following blocks,
+ * each block is 64KB:
+ *
+ * _____________________________________________________________________________
+ * | File Id. |   Header 1    | Header 2   | Region Table |  Reserved (768KB)  |
+ * |----------|---------------|------------|--------------|--------------------|
+ * |          |               |            |              |                    |
+ * 0.........64KB...........128KB........192KB..........256KB................1MB
+ */
+
+#define VHDX_HEADER_BLOCK_SIZE      (64*1024)
+
+#define VHDX_FILE_ID_OFFSET         0
+#define VHDX_HEADER1_OFFSET         (VHDX_HEADER_BLOCK_SIZE*1)
+#define VHDX_HEADER2_OFFSET         (VHDX_HEADER_BLOCK_SIZE*2)
+#define VHDX_REGION_TABLE_OFFSET    (VHDX_HEADER_BLOCK_SIZE*3)
+
+
+/*
+ * A note on the use of MS-GUID fields.  For more details on the GUID,
+ * please see: https://en.wikipedia.org/wiki/Globally_unique_identifier.
+ *
+ * The VHDX specification only states that these are MS GUIDs, and which
+ * bytes are data1-data4. It makes no mention of what algorithm should be used
+ * to generate the GUID, nor what standard.  However, looking at the specified
+ * known GUID fields, it appears the GUIDs are:
+ *  Standard/DCE GUID type  (noted by 10b in the MSB of byte 0 of .data4)
+ *  Random algorithm        (noted by 0x4XXX for .data3)
+ */
+
+/* ---- HEADER SECTION STRUCTURES ---- */
+
+/* Important note: these structures are as defined in the VHDX specification,
+ * including byte order and size.  However, without being packed structures,
+ * they will not match 1:1 data read from disk.  Rather than use potentially
+ * non-portable packed structures, data is copied from read buffers into
+ * the structures below.  However, for reference, please refrain from
+ * modifying these structures to something that does not represent the spec */
+
+#define VHDX_FILE_ID_MAGIC 0x656C696678646876  /* 'vhdxfile' */
+typedef struct vhdx_file_identifier {
+    uint64_t    signature;              /* "vhdxfile" in ASCII */
+    uint16_t    creator[256];           /* optional; utf-16 string to identify
+                                           the vhdx file creator.  Diagnotistic
+                                           only */
+} vhdx_file_identifier;
+
+
+/* the guid is a 16 byte unique ID - the definition for this used by
+ * Microsoft is not just 16 bytes though - it is a structure that is defined,
+ * so we need to follow it here so that endianness does not trip us up */
+
+typedef struct ms_guid {
+    uint32_t    data1;
+    uint16_t    data2;
+    uint16_t    data3;
+    uint8_t     data4[8];
+} ms_guid;
+
+#define guid_eq(a, b) \
+    (memcmp(&(a), &(b), sizeof(ms_guid)) == 0)
+
+#define VHDX_HEADER_SIZE (4*1024)   /* although the vhdx_header struct in disk
+                                       is only 582 bytes, for purposes of crc
+                                       the header is the first 4KB of the 64KB
+                                       block */
+
+#define VHDX_HDR_MAGIC 0x64616568   /* 'head' */
+typedef struct QEMU_PACKED vhdx_header {
+    uint32_t    signature;              /* "head" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the whole header */
+    uint64_t    sequence_number;        /* Seq number of this header.  Each
+                                           VHDX file has 2 of these headers,
+                                           and only the header with the highest
+                                           sequence number is valid */
+    ms_guid     file_write_guid;       /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           file */
+    ms_guid     data_write_guid;        /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           visible data.   Visbile data is
+                                           defined as:
+                                                    - system & user metadata
+                                                    - raw block data
+                                                    - disk size
+                                                    - any change that will
+                                                      cause the virtual disk
+                                                      sector read to differ
+
+                                           This does not need to change if
+                                           blocks are re-arranged */
+    ms_guid     log_guid;               /* 128 bit unique identifier. If zero,
+                                           there is no valid log. If non-zero,
+                                           log entries with this guid are
+                                           valid. */
+    uint16_t    log_version;            /* version of the log format. Mustn't be
+                                           zero, unless log_guid is also zero */
+    uint16_t    version;                /* version of th evhdx file.  Currently,
+                                           only supported version is "1" */
+    uint32_t    log_length;             /* length of the log.  Must be multiple
+                                           of 1MB */
+    uint64_t    log_offset;             /* byte offset in the file of the log.
+                                           Must also be a multiple of 1MB */
+} vhdx_header;
+
+/* 4KB in packed data size, not to be used except for initial data read */
+typedef struct QEMU_PACKED vhdx_header_padded {
+    vhdx_header header;
+    uint8_t     reserved[502];          /* per the VHDX spec */
+    uint8_t     reserved_[3514];        /* for the initial packed struct read */
+} vhdx_header_padded;
+
+/* Header for the region table block */
+#define VHDX_RT_MAGIC 0x69676572  /* 'regi ' */
+typedef struct QEMU_PACKED vhdx_region_table_header {
+    uint32_t    signature;              /* "regi" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_count;            /* number of valid entries */
+    uint32_t    reserved;
+} vhdx_region_table_header;
+
+/* Individual region table entry.  There may be a maximum of 2047 of these
+ *
+ *  There are two known region table properties.  Both are required.
+ *  BAT (block allocation table):  2DC27766F62342009D64115E9BFD4A08
+ *  Metadata:                      8B7CA20647904B9AB8FE575F050F886E
+ */
+#define VHDX_REGION_ENTRY_REQUIRED  0x01    /* if set, parser must understand
+                                               this entry in order to open
+                                               file */
+typedef struct QEMU_PACKED vhdx_region_table_entry {
+    ms_guid     guid;                   /* 128-bit unique identifier */
+    uint64_t    file_offset;            /* offset of the object in the file.
+                                           Must be multiple of 1MB */
+    uint32_t    length;                 /* length, in bytes, of the object */
+    uint32_t    data_bits;
+} vhdx_region_table_entry;
+
+
+/* ---- LOG ENTRY STRUCTURES ---- */
+#define VHDX_LOG_HDR_SIZE 64
+#define VHDX_LOGE_MAGIC 0x65676F6C /* 'loge' */
+typedef struct QEMU_PACKED vhdx_log_entry_header {
+    uint32_t    signature;              /* "loge" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_length;           /* length in bytes, multiple of 1MB */
+    uint32_t    tail;                   /* byte offset of first log entry of a
+                                           seq, where this entry is the last
+                                           entry */
+    uint64_t    sequence_number;        /* incremented with each log entry.
+                                           May not be zero. */
+    uint32_t    descriptor_count;       /* number of descriptors in this log
+                                           entry, must be >= 0 */
+    uint32_t    reserved;
+    ms_guid     log_guid;               /* value of the log_guid from
+                                           vhdx_header.  If not found in
+                                           vhdx_header, it is invalid */
+    uint64_t    flushed_file_offset;    /* see spec for full details - this
+                                           sould be vhdx file size in bytes */
+    uint64_t    last_file_offset;       /* size in bytes that all allocated
+                                           file structures fit into */
+} vhdx_log_entry_header;
+
+#define VHDX_LOG_DESC_SIZE 32
+
+#define VHDX_ZERO_MAGIC 0x6F72657A /* 'zero' */
+#define VHDX_DATA_MAGIC 0x63736564 /* 'desc' */
+typedef struct QEMU_PACKED vhdx_log_descriptor {
+    uint32_t    signature;              /* "zero" or "desc" in ASCII */
+    union  {
+    uint32_t    reserved;               /* zero desc */
+    uint32_t    trailing_bytes;         /* data desc: bytes 4092-4096 of the
+                                           data sector */
+    };
+    union {
+    uint64_t    zero_length;            /* zero desc: length of the section to
+                                           zero */
+    uint64_t    leading_bytes;          /* data desc: bytes 0-7 of the data
+                                           sector */
+    };
+    uint64_t    file_offset;            /* file offset to write zeros - multiple
+                                           of 4kB */
+    uint64_t    sequence_number;        /* must match same field in
+                                           vhdx_log_entry_header */
+} vhdx_log_descriptor;
+
+#define VHDX_DATAS_MAGIC 0x61746164 /* 'data' */
+typedef struct QEMU_PACKED vhdx_log_data_sector {
+    uint32_t    data_signature;         /* "data" in ASCII */
+    uint32_t    sequence_high;          /* 4 MSB of 8 byte sequence_number */
+    uint8_t     data[4084];             /* raw data, bytes 8-4091 (inclusive).
+                                           see the data descriptor field for the
+                                           other mising bytes */
+    uint32_t    sequence_low;           /* 4 LSB of 8 byte sequence_number */
+} vhdx_log_data_sector;
+
+
+
+/* block states - different state values depending on whether it is a
+ * payload block, or a sector block. */
+
+#define PAYLOAD_BLOCK_NOT_PRESENT       0
+#define PAYLOAD_BLOCK_UNDEFINED         1
+#define PAYLOAD_BLOCK_ZERO              2
+#define PAYLOAD_BLOCK_UNMAPPED          5
+#define PAYLOAD_BLOCK_FULL_PRESENT      6
+#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7
+
+#define SB_BLOCK_NOT_PRESENT    0
+#define SB_BLOCK_PRESENT        6
+
+/* per the spec */
+#define VHDX_MAX_SECTORS_PER_BLOCK  (1<<23)
+
+/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state
+   other bits are reserved */
+#define VHDX_BAT_STATE_BIT_MASK 0x07
+#define VHDX_BAT_FILE_OFF_BITS (64-44)
+typedef uint64_t vhdx_bat_entry;
+
+/* ---- METADATA REGION STRUCTURES ---- */
+
+#define VHDX_METADATA_ENTRY_SIZE 32
+#define VHDX_METADATA_MAX_ENTRIES 2047  /* not including the header */
+#define VHDX_METADATA_TABLE_MAX_SIZE \
+    (VHDX_METADATA_ENTRY_SIZE * (VHDX_METADATA_MAX_ENTRIES+1))
+#define VHDX_METADATA_MAGIC 0x617461646174656D /* 'metadata' */
+typedef struct QEMU_PACKED vhdx_metadata_table_header {
+    uint64_t    signature;              /* "metadata" in ASCII */
+    uint16_t    reserved;
+    uint16_t    entry_count;            /* number table entries. <= 2047 */
+    uint32_t    reserved2[5];
+} vhdx_metadata_table_header;
+
+#define VHDX_META_FLAGS_IS_USER         0x01    /* max 1024 entries */
+#define VHDX_META_FLAGS_IS_VIRTUAL_DISK 0x02    /* virtual disk metadata if set,
+                                                   otherwise file metdata */
+#define VHDX_META_FLAGS_IS_REQUIRED     0x04    /* parse must understand this
+                                                   entry to open the file */
+typedef struct QEMU_PACKED vhdx_metadata_table_entry {
+    ms_guid     item_id;                /* 128-bit identifier for metadata */
+    uint32_t    offset;                 /* byte offset of the metadata.  At
+                                           least 64kB.  Relative to start of
+                                           metadata region */
+                                        /* note: if length = 0, so is offset */
+    uint32_t    length;                 /* length of metadata. <= 1MB. */
+    uint32_t    data_bits;      /* least-significant 3 bits are flags, the
+                                   rest are reserved (see above) */
+    uint32_t    reserved2;
+} vhdx_metadata_table_entry;
+
+#define VHDX_PARAMS_LEAVE_BLOCKS_ALLOCED 0x01   /* Do not change any blocks to
+                                                   be BLOCK_NOT_PRESENT.
+                                                   If set indicates a fixed
+                                                   size VHDX file */
+#define VHDX_PARAMS_HAS_PARENT           0x02    /* has parent / backing file */
+typedef struct QEMU_PACKED vhdx_file_parameters {
+    uint32_t    block_size;             /* size of each payload block, always
+                                           power of 2, <= 256MB and >= 1MB. */
+    uint32_t data_bits;     /* least-significant 2 bits are flags, the rest
+                               are reserved (see above) */
+} vhdx_file_parameters;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_size {
+    uint64_t    virtual_disk_size;      /* Size of the virtual disk, in bytes.
+                                           Must be multiple of the sector size,
+                                           max of 64TB */
+} vhdx_virtual_disk_size;
+
+typedef struct QEMU_PACKED vhdx_page83_data {
+    uint8_t     page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} vhdx_page83_data;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_logical_sector_size {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_logical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_physical_sector_size {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_physical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_parent_locator_header {
+    uint8_t     locator_type[16];       /* type of the parent virtual disk. */
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} vhdx_parent_locator_header;
+
+/* key and value strings are UNICODE strings, UTF-16 LE encoding, no NULs */
+typedef struct QEMU_PACKED vhdx_parent_locator_entry {
+    uint32_t    key_offset;             /* offset in metadata for key, > 0 */
+    uint32_t    value_offset;           /* offset in metadata for value, >0 */
+    uint16_t    key_length;             /* length of entry key, > 0 */
+    uint16_t    value_length;           /* length of entry value, > 0 */
+} vhdx_parent_locator_entry;
+
+
+/* ----- END VHDX SPECIFICATION STRUCTURES ---- */
+
+#endif
Signature · 1 line
1.8.1.4
#3 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 72 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 327 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 block/vhdx.h
block/vhdx.h new file +53 -0
new file mode 100644
index 0000000..f5cf1ed
@@ -0,0 +1,327 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_VHDX_H
+#define BLOCK_VHDX_H
+
+/* Structures and fields present in the VHDX file */
+
+/* The header section has the following blocks,
+ * each block is 64KB:
+ *
+ * _____________________________________________________________________________
+ * | File Id. |   Header 1    | Header 2   | Region Table |  Reserved (768KB)  |
+ * |----------|---------------|------------|--------------|--------------------|
+ * |          |               |            |              |                    |
+ * 0.........64KB...........128KB........192KB..........256KB................1MB
+ */
+
+#define VHDX_HEADER_BLOCK_SIZE      (64*1024)
+
+#define VHDX_FILE_ID_OFFSET         0
+#define VHDX_HEADER1_OFFSET         (VHDX_HEADER_BLOCK_SIZE*1)
+#define VHDX_HEADER2_OFFSET         (VHDX_HEADER_BLOCK_SIZE*2)
+#define VHDX_REGION_TABLE_OFFSET    (VHDX_HEADER_BLOCK_SIZE*3)
+
+
+/*
+ * A note on the use of MS-GUID fields.  For more details on the GUID,
+ * please see: https://en.wikipedia.org/wiki/Globally_unique_identifier.
+ *
+ * The VHDX specification only states that these are MS GUIDs, and which
+ * bytes are data1-data4. It makes no mention of what algorithm should be used
+ * to generate the GUID, nor what standard.  However, looking at the specified
+ * known GUID fields, it appears the GUIDs are:
+ *  Standard/DCE GUID type  (noted by 10b in the MSB of byte 0 of .data4)
+ *  Random algorithm        (noted by 0x4XXX for .data3)
+ */
+
+/* ---- HEADER SECTION STRUCTURES ---- */

block/vhdx.h line 63

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+/* Important note: these structures are as defined in the VHDX specification,
+ * including byte order and size.  However, without being packed structures,
+ * they will not match 1:1 data read from disk.  Rather than use potentially
+ * non-portable packed structures, data is copied from read buffers into
+ * the structures below.  However, for reference, please refrain from
+ * modifying these structures to something that does not represent the spec */
+
+#define VHDX_FILE_ID_MAGIC 0x656C696678646876  /* 'vhdxfile' */
+typedef struct vhdx_file_identifier {

According to the qemu coding style, this is not a valid name for a struct. I think you can choose between VHDXFileIdentifier and VhdxFileIdentifier, where existing code tends towards the former.

More instances of this follow, but I won't comment on each.

Quoted 125 lines
block/vhdx.h new file +125 -0
+    uint64_t    signature;              /* "vhdxfile" in ASCII */
+    uint16_t    creator[256];           /* optional; utf-16 string to identify
+                                           the vhdx file creator.  Diagnotistic
+                                           only */
+} vhdx_file_identifier;
+
+
+/* the guid is a 16 byte unique ID - the definition for this used by
+ * Microsoft is not just 16 bytes though - it is a structure that is defined,
+ * so we need to follow it here so that endianness does not trip us up */
+
+typedef struct ms_guid {
+    uint32_t    data1;
+    uint16_t    data2;
+    uint16_t    data3;
+    uint8_t     data4[8];
+} ms_guid;
+
+#define guid_eq(a, b) \
+    (memcmp(&(a), &(b), sizeof(ms_guid)) == 0)
+
+#define VHDX_HEADER_SIZE (4*1024)   /* although the vhdx_header struct in disk
+                                       is only 582 bytes, for purposes of crc
+                                       the header is the first 4KB of the 64KB
+                                       block */
+
+#define VHDX_HDR_MAGIC 0x64616568   /* 'head' */
+typedef struct QEMU_PACKED vhdx_header {
+    uint32_t    signature;              /* "head" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the whole header */
+    uint64_t    sequence_number;        /* Seq number of this header.  Each
+                                           VHDX file has 2 of these headers,
+                                           and only the header with the highest
+                                           sequence number is valid */
+    ms_guid     file_write_guid;       /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           file */
+    ms_guid     data_write_guid;        /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           visible data.   Visbile data is
+                                           defined as:
+                                                    - system & user metadata
+                                                    - raw block data
+                                                    - disk size
+                                                    - any change that will
+                                                      cause the virtual disk
+                                                      sector read to differ
+
+                                           This does not need to change if
+                                           blocks are re-arranged */
+    ms_guid     log_guid;               /* 128 bit unique identifier. If zero,
+                                           there is no valid log. If non-zero,
+                                           log entries with this guid are
+                                           valid. */
+    uint16_t    log_version;            /* version of the log format. Mustn't be
+                                           zero, unless log_guid is also zero */
+    uint16_t    version;                /* version of th evhdx file.  Currently,
+                                           only supported version is "1" */
+    uint32_t    log_length;             /* length of the log.  Must be multiple
+                                           of 1MB */
+    uint64_t    log_offset;             /* byte offset in the file of the log.
+                                           Must also be a multiple of 1MB */
+} vhdx_header;
+
+/* 4KB in packed data size, not to be used except for initial data read */
+typedef struct QEMU_PACKED vhdx_header_padded {
+    vhdx_header header;
+    uint8_t     reserved[502];          /* per the VHDX spec */
+    uint8_t     reserved_[3514];        /* for the initial packed struct read */
+} vhdx_header_padded;
+
+/* Header for the region table block */
+#define VHDX_RT_MAGIC 0x69676572  /* 'regi ' */
+typedef struct QEMU_PACKED vhdx_region_table_header {
+    uint32_t    signature;              /* "regi" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_count;            /* number of valid entries */
+    uint32_t    reserved;
+} vhdx_region_table_header;
+
+/* Individual region table entry.  There may be a maximum of 2047 of these
+ *
+ *  There are two known region table properties.  Both are required.
+ *  BAT (block allocation table):  2DC27766F62342009D64115E9BFD4A08
+ *  Metadata:                      8B7CA20647904B9AB8FE575F050F886E
+ */
+#define VHDX_REGION_ENTRY_REQUIRED  0x01    /* if set, parser must understand
+                                               this entry in order to open
+                                               file */
+typedef struct QEMU_PACKED vhdx_region_table_entry {
+    ms_guid     guid;                   /* 128-bit unique identifier */
+    uint64_t    file_offset;            /* offset of the object in the file.
+                                           Must be multiple of 1MB */
+    uint32_t    length;                 /* length, in bytes, of the object */
+    uint32_t    data_bits;
+} vhdx_region_table_entry;
+
+
+/* ---- LOG ENTRY STRUCTURES ---- */
+#define VHDX_LOG_HDR_SIZE 64
+#define VHDX_LOGE_MAGIC 0x65676F6C /* 'loge' */
+typedef struct QEMU_PACKED vhdx_log_entry_header {
+    uint32_t    signature;              /* "loge" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_length;           /* length in bytes, multiple of 1MB */
+    uint32_t    tail;                   /* byte offset of first log entry of a
+                                           seq, where this entry is the last
+                                           entry */
+    uint64_t    sequence_number;        /* incremented with each log entry.
+                                           May not be zero. */
+    uint32_t    descriptor_count;       /* number of descriptors in this log
+                                           entry, must be >= 0 */
+    uint32_t    reserved;
+    ms_guid     log_guid;               /* value of the log_guid from
+                                           vhdx_header.  If not found in
+                                           vhdx_header, it is invalid */
+    uint64_t    flushed_file_offset;    /* see spec for full details - this
+                                           sould be vhdx file size in bytes */
+    uint64_t    last_file_offset;       /* size in bytes that all allocated
+                                           file structures fit into */
+} vhdx_log_entry_header;
+
+#define VHDX_LOG_DESC_SIZE 32

block/vhdx.h line 198

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+#define VHDX_ZERO_MAGIC 0x6F72657A /* 'zero' */
+#define VHDX_DATA_MAGIC 0x63736564 /* 'desc' */
+typedef struct QEMU_PACKED vhdx_log_descriptor {
+    uint32_t    signature;              /* "zero" or "desc" in ASCII */
+    union  {
+    uint32_t    reserved;               /* zero desc */
+    uint32_t    trailing_bytes;         /* data desc: bytes 4092-4096 of the
+                                           data sector */
+    };

Indentation inside the union is off.

block/vhdx.h line 204

Quoted 6 lines
block/vhdx.h new file +6 -0
+    union {
+    uint64_t    zero_length;            /* zero desc: length of the section to
+                                           zero */
+    uint64_t    leading_bytes;          /* data desc: bytes 0-7 of the data
+                                           sector */
+    };

Here as well.

Quoted 17 lines
block/vhdx.h new file +17 -0
+    uint64_t    file_offset;            /* file offset to write zeros - multiple
+                                           of 4kB */
+    uint64_t    sequence_number;        /* must match same field in
+                                           vhdx_log_entry_header */
+} vhdx_log_descriptor;
+
+#define VHDX_DATAS_MAGIC 0x61746164 /* 'data' */
+typedef struct QEMU_PACKED vhdx_log_data_sector {
+    uint32_t    data_signature;         /* "data" in ASCII */
+    uint32_t    sequence_high;          /* 4 MSB of 8 byte sequence_number */
+    uint8_t     data[4084];             /* raw data, bytes 8-4091 (inclusive).
+                                           see the data descriptor field for the
+                                           other mising bytes */
+    uint32_t    sequence_low;           /* 4 LSB of 8 byte sequence_number */
+} vhdx_log_data_sector;
+
+

block/vhdx.h line 231

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+/* block states - different state values depending on whether it is a
+ * payload block, or a sector block. */
+
+#define PAYLOAD_BLOCK_NOT_PRESENT       0
+#define PAYLOAD_BLOCK_UNDEFINED         1
+#define PAYLOAD_BLOCK_ZERO              2
+#define PAYLOAD_BLOCK_UNMAPPED          5
+#define PAYLOAD_BLOCK_FULL_PRESENT      6
+#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7

Looks like an enum?

Quoted 65 lines
block/vhdx.h new file +65 -0
+#define SB_BLOCK_NOT_PRESENT    0
+#define SB_BLOCK_PRESENT        6
+
+/* per the spec */
+#define VHDX_MAX_SECTORS_PER_BLOCK  (1<<23)
+
+/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state
+   other bits are reserved */
+#define VHDX_BAT_STATE_BIT_MASK 0x07
+#define VHDX_BAT_FILE_OFF_BITS (64-44)
+typedef uint64_t vhdx_bat_entry;
+
+/* ---- METADATA REGION STRUCTURES ---- */
+
+#define VHDX_METADATA_ENTRY_SIZE 32
+#define VHDX_METADATA_MAX_ENTRIES 2047  /* not including the header */
+#define VHDX_METADATA_TABLE_MAX_SIZE \
+    (VHDX_METADATA_ENTRY_SIZE * (VHDX_METADATA_MAX_ENTRIES+1))
+#define VHDX_METADATA_MAGIC 0x617461646174656D /* 'metadata' */
+typedef struct QEMU_PACKED vhdx_metadata_table_header {
+    uint64_t    signature;              /* "metadata" in ASCII */
+    uint16_t    reserved;
+    uint16_t    entry_count;            /* number table entries. <= 2047 */
+    uint32_t    reserved2[5];
+} vhdx_metadata_table_header;
+
+#define VHDX_META_FLAGS_IS_USER         0x01    /* max 1024 entries */
+#define VHDX_META_FLAGS_IS_VIRTUAL_DISK 0x02    /* virtual disk metadata if set,
+                                                   otherwise file metdata */
+#define VHDX_META_FLAGS_IS_REQUIRED     0x04    /* parse must understand this
+                                                   entry to open the file */
+typedef struct QEMU_PACKED vhdx_metadata_table_entry {
+    ms_guid     item_id;                /* 128-bit identifier for metadata */
+    uint32_t    offset;                 /* byte offset of the metadata.  At
+                                           least 64kB.  Relative to start of
+                                           metadata region */
+                                        /* note: if length = 0, so is offset */
+    uint32_t    length;                 /* length of metadata. <= 1MB. */
+    uint32_t    data_bits;      /* least-significant 3 bits are flags, the
+                                   rest are reserved (see above) */
+    uint32_t    reserved2;
+} vhdx_metadata_table_entry;
+
+#define VHDX_PARAMS_LEAVE_BLOCKS_ALLOCED 0x01   /* Do not change any blocks to
+                                                   be BLOCK_NOT_PRESENT.
+                                                   If set indicates a fixed
+                                                   size VHDX file */
+#define VHDX_PARAMS_HAS_PARENT           0x02    /* has parent / backing file */
+typedef struct QEMU_PACKED vhdx_file_parameters {
+    uint32_t    block_size;             /* size of each payload block, always
+                                           power of 2, <= 256MB and >= 1MB. */
+    uint32_t data_bits;     /* least-significant 2 bits are flags, the rest
+                               are reserved (see above) */
+} vhdx_file_parameters;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_size {
+    uint64_t    virtual_disk_size;      /* Size of the virtual disk, in bytes.
+                                           Must be multiple of the sector size,
+                                           max of 64TB */
+} vhdx_virtual_disk_size;
+
+typedef struct QEMU_PACKED vhdx_page83_data {
+    uint8_t     page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} vhdx_page83_data;

block/vhdx.h line 307

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_logical_sector_size {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_logical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_physical_sector_size {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_physical_sector_size;

What's the point with all the single-field structs?

Quoted 9 lines
block/vhdx.h new file +9 -0
+typedef struct QEMU_PACKED vhdx_parent_locator_header {
+    uint8_t     locator_type[16];       /* type of the parent virtual disk. */
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} vhdx_parent_locator_header;
+
+/* key and value strings are UNICODE strings, UTF-16 LE encoding, no NULs */
+typedef struct QEMU_PACKED vhdx_parent_locator_entry {

block/vhdx.h line 327

Quoted 10 lines
block/vhdx.h new file +10 -0
+    uint32_t    key_offset;             /* offset in metadata for key, > 0 */
+    uint32_t    value_offset;           /* offset in metadata for value, >0 */
+    uint16_t    key_length;             /* length of entry key, > 0 */
+    uint16_t    value_length;           /* length of entry value, > 0 */
+} vhdx_parent_locator_entry;
+
+
+/* ----- END VHDX SPECIFICATION STRUCTURES ---- */
+
+#endif

Kevin

author
#4 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

Quoted 88 lines · Kevin Wolf

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 72 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 327 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 block/vhdx.h
block/vhdx.h new file +53 -0
new file mode 100644
index 0000000..f5cf1ed
@@ -0,0 +1,327 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_VHDX_H
+#define BLOCK_VHDX_H
+
+/* Structures and fields present in the VHDX file */
+
+/* The header section has the following blocks,
+ * each block is 64KB:
+ *
+ * _____________________________________________________________________________
+ * | File Id. |   Header 1    | Header 2   | Region Table |  Reserved (768KB)  |
+ * |----------|---------------|------------|--------------|--------------------|
+ * |          |               |            |              |                    |
+ * 0.........64KB...........128KB........192KB..........256KB................1MB
+ */
+
+#define VHDX_HEADER_BLOCK_SIZE      (64*1024)
+
+#define VHDX_FILE_ID_OFFSET         0
+#define VHDX_HEADER1_OFFSET         (VHDX_HEADER_BLOCK_SIZE*1)
+#define VHDX_HEADER2_OFFSET         (VHDX_HEADER_BLOCK_SIZE*2)
+#define VHDX_REGION_TABLE_OFFSET    (VHDX_HEADER_BLOCK_SIZE*3)
+
+
+/*
+ * A note on the use of MS-GUID fields.  For more details on the GUID,
+ * please see: https://en.wikipedia.org/wiki/Globally_unique_identifier.
+ *
+ * The VHDX specification only states that these are MS GUIDs, and which
+ * bytes are data1-data4. It makes no mention of what algorithm should be used
+ * to generate the GUID, nor what standard.  However, looking at the specified
+ * known GUID fields, it appears the GUIDs are:
+ *  Standard/DCE GUID type  (noted by 10b in the MSB of byte 0 of .data4)
+ *  Random algorithm        (noted by 0x4XXX for .data3)
+ */
+
+/* ---- HEADER SECTION STRUCTURES ---- */

block/vhdx.h line 63

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+/* Important note: these structures are as defined in the VHDX specification,
+ * including byte order and size.  However, without being packed structures,
+ * they will not match 1:1 data read from disk.  Rather than use potentially
+ * non-portable packed structures, data is copied from read buffers into
+ * the structures below.  However, for reference, please refrain from
+ * modifying these structures to something that does not represent the spec */
+
+#define VHDX_FILE_ID_MAGIC 0x656C696678646876  /* 'vhdxfile' */
+typedef struct vhdx_file_identifier {

According to the qemu coding style, this is not a valid name for a struct. I think you can choose between VHDXFileIdentifier and VhdxFileIdentifier, where existing code tends towards the former.

Quoted 2 lines

More instances of this follow, but I won't comment on each.

OK. I'll go through and change them all; almost all of them will need to be changed.

Quoted 138 lines
Quoted 125 lines
block/vhdx.h new file +125 -0
+    uint64_t    signature;              /* "vhdxfile" in ASCII */
+    uint16_t    creator[256];           /* optional; utf-16 string to identify
+                                           the vhdx file creator.  Diagnotistic
+                                           only */
+} vhdx_file_identifier;
+
+
+/* the guid is a 16 byte unique ID - the definition for this used by
+ * Microsoft is not just 16 bytes though - it is a structure that is defined,
+ * so we need to follow it here so that endianness does not trip us up */
+
+typedef struct ms_guid {
+    uint32_t    data1;
+    uint16_t    data2;
+    uint16_t    data3;
+    uint8_t     data4[8];
+} ms_guid;
+
+#define guid_eq(a, b) \
+    (memcmp(&(a), &(b), sizeof(ms_guid)) == 0)
+
+#define VHDX_HEADER_SIZE (4*1024)   /* although the vhdx_header struct in disk
+                                       is only 582 bytes, for purposes of crc
+                                       the header is the first 4KB of the 64KB
+                                       block */
+
+#define VHDX_HDR_MAGIC 0x64616568   /* 'head' */
+typedef struct QEMU_PACKED vhdx_header {
+    uint32_t    signature;              /* "head" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the whole header */
+    uint64_t    sequence_number;        /* Seq number of this header.  Each
+                                           VHDX file has 2 of these headers,
+                                           and only the header with the highest
+                                           sequence number is valid */
+    ms_guid     file_write_guid;       /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           file */
+    ms_guid     data_write_guid;        /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           visible data.   Visbile data is
+                                           defined as:
+                                                    - system & user metadata
+                                                    - raw block data
+                                                    - disk size
+                                                    - any change that will
+                                                      cause the virtual disk
+                                                      sector read to differ
+
+                                           This does not need to change if
+                                           blocks are re-arranged */
+    ms_guid     log_guid;               /* 128 bit unique identifier. If zero,
+                                           there is no valid log. If non-zero,
+                                           log entries with this guid are
+                                           valid. */
+    uint16_t    log_version;            /* version of the log format. Mustn't be
+                                           zero, unless log_guid is also zero */
+    uint16_t    version;                /* version of th evhdx file.  Currently,
+                                           only supported version is "1" */
+    uint32_t    log_length;             /* length of the log.  Must be multiple
+                                           of 1MB */
+    uint64_t    log_offset;             /* byte offset in the file of the log.
+                                           Must also be a multiple of 1MB */
+} vhdx_header;
+
+/* 4KB in packed data size, not to be used except for initial data read */
+typedef struct QEMU_PACKED vhdx_header_padded {
+    vhdx_header header;
+    uint8_t     reserved[502];          /* per the VHDX spec */
+    uint8_t     reserved_[3514];        /* for the initial packed struct read */
+} vhdx_header_padded;
+
+/* Header for the region table block */
+#define VHDX_RT_MAGIC 0x69676572  /* 'regi ' */
+typedef struct QEMU_PACKED vhdx_region_table_header {
+    uint32_t    signature;              /* "regi" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_count;            /* number of valid entries */
+    uint32_t    reserved;
+} vhdx_region_table_header;
+
+/* Individual region table entry.  There may be a maximum of 2047 of these
+ *
+ *  There are two known region table properties.  Both are required.
+ *  BAT (block allocation table):  2DC27766F62342009D64115E9BFD4A08
+ *  Metadata:                      8B7CA20647904B9AB8FE575F050F886E
+ */
+#define VHDX_REGION_ENTRY_REQUIRED  0x01    /* if set, parser must understand
+                                               this entry in order to open
+                                               file */
+typedef struct QEMU_PACKED vhdx_region_table_entry {
+    ms_guid     guid;                   /* 128-bit unique identifier */
+    uint64_t    file_offset;            /* offset of the object in the file.
+                                           Must be multiple of 1MB */
+    uint32_t    length;                 /* length, in bytes, of the object */
+    uint32_t    data_bits;
+} vhdx_region_table_entry;
+
+
+/* ---- LOG ENTRY STRUCTURES ---- */
+#define VHDX_LOG_HDR_SIZE 64
+#define VHDX_LOGE_MAGIC 0x65676F6C /* 'loge' */
+typedef struct QEMU_PACKED vhdx_log_entry_header {
+    uint32_t    signature;              /* "loge" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_length;           /* length in bytes, multiple of 1MB */
+    uint32_t    tail;                   /* byte offset of first log entry of a
+                                           seq, where this entry is the last
+                                           entry */
+    uint64_t    sequence_number;        /* incremented with each log entry.
+                                           May not be zero. */
+    uint32_t    descriptor_count;       /* number of descriptors in this log
+                                           entry, must be >= 0 */
+    uint32_t    reserved;
+    ms_guid     log_guid;               /* value of the log_guid from
+                                           vhdx_header.  If not found in
+                                           vhdx_header, it is invalid */
+    uint64_t    flushed_file_offset;    /* see spec for full details - this
+                                           sould be vhdx file size in bytes */
+    uint64_t    last_file_offset;       /* size in bytes that all allocated
+                                           file structures fit into */
+} vhdx_log_entry_header;
+
+#define VHDX_LOG_DESC_SIZE 32

block/vhdx.h line 198

Quoted 10 lines
block/vhdx.h new file +10 -0
+
+#define VHDX_ZERO_MAGIC 0x6F72657A /* 'zero' */
+#define VHDX_DATA_MAGIC 0x63736564 /* 'desc' */
+typedef struct QEMU_PACKED vhdx_log_descriptor {
+    uint32_t    signature;              /* "zero" or "desc" in ASCII */
+    union  {
+    uint32_t    reserved;               /* zero desc */
+    uint32_t    trailing_bytes;         /* data desc: bytes 4092-4096 of the
+                                           data sector */
+    };

Indentation inside the union is off.

Quoted 9 lines

block/vhdx.h line 204

Quoted 6 lines
block/vhdx.h new file +6 -0
+    union {
+    uint64_t    zero_length;            /* zero desc: length of the section to
+                                           zero */
+    uint64_t    leading_bytes;          /* data desc: bytes 0-7 of the data
+                                           sector */
+    };

Here as well.

OK, thanks.

Quoted 21 lines
Quoted 21 lines
block/vhdx.h new file +21 -0
+    uint64_t    file_offset;            /* file offset to write zeros - multiple
+                                           of 4kB */
+    uint64_t    sequence_number;        /* must match same field in
+                                           vhdx_log_entry_header */
+} vhdx_log_descriptor;
+
+#define VHDX_DATAS_MAGIC 0x61746164 /* 'data' */
+typedef struct QEMU_PACKED vhdx_log_data_sector {
+    uint32_t    data_signature;         /* "data" in ASCII */
+    uint32_t    sequence_high;          /* 4 MSB of 8 byte sequence_number */
+    uint8_t     data[4084];             /* raw data, bytes 8-4091 (inclusive).
+                                           see the data descriptor field for the
+                                           other mising bytes */
+    uint32_t    sequence_low;           /* 4 LSB of 8 byte sequence_number */
+} vhdx_log_data_sector;
+
+
+
+/* block states - different state values depending on whether it is a
+ * payload block, or a sector block. */
+
Quoted 9 lines

block/vhdx.h line 231

Quoted 6 lines
block/vhdx.h new file +6 -0
+#define PAYLOAD_BLOCK_NOT_PRESENT       0
+#define PAYLOAD_BLOCK_UNDEFINED         1
+#define PAYLOAD_BLOCK_ZERO              2
+#define PAYLOAD_BLOCK_UNMAPPED          5
+#define PAYLOAD_BLOCK_FULL_PRESENT      6
+#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7

Looks like an enum?

The spec lays them out as #define's, so this is a case where I figured it helped with a reader comparing against the spec. But I can make this (and the corresponding ones below) into enums, if you are worried about namespace pollution.

Quoted 69 lines
Quoted 69 lines
block/vhdx.h new file +69 -0
+#define SB_BLOCK_NOT_PRESENT    0
+#define SB_BLOCK_PRESENT        6
+
+/* per the spec */
+#define VHDX_MAX_SECTORS_PER_BLOCK  (1<<23)
+
+/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state
+   other bits are reserved */
+#define VHDX_BAT_STATE_BIT_MASK 0x07
+#define VHDX_BAT_FILE_OFF_BITS (64-44)
+typedef uint64_t vhdx_bat_entry;
+
+/* ---- METADATA REGION STRUCTURES ---- */
+
+#define VHDX_METADATA_ENTRY_SIZE 32
+#define VHDX_METADATA_MAX_ENTRIES 2047  /* not including the header */
+#define VHDX_METADATA_TABLE_MAX_SIZE \
+    (VHDX_METADATA_ENTRY_SIZE * (VHDX_METADATA_MAX_ENTRIES+1))
+#define VHDX_METADATA_MAGIC 0x617461646174656D /* 'metadata' */
+typedef struct QEMU_PACKED vhdx_metadata_table_header {
+    uint64_t    signature;              /* "metadata" in ASCII */
+    uint16_t    reserved;
+    uint16_t    entry_count;            /* number table entries. <= 2047 */
+    uint32_t    reserved2[5];
+} vhdx_metadata_table_header;
+
+#define VHDX_META_FLAGS_IS_USER         0x01    /* max 1024 entries */
+#define VHDX_META_FLAGS_IS_VIRTUAL_DISK 0x02    /* virtual disk metadata if set,
+                                                   otherwise file metdata */
+#define VHDX_META_FLAGS_IS_REQUIRED     0x04    /* parse must understand this
+                                                   entry to open the file */
+typedef struct QEMU_PACKED vhdx_metadata_table_entry {
+    ms_guid     item_id;                /* 128-bit identifier for metadata */
+    uint32_t    offset;                 /* byte offset of the metadata.  At
+                                           least 64kB.  Relative to start of
+                                           metadata region */
+                                        /* note: if length = 0, so is offset */
+    uint32_t    length;                 /* length of metadata. <= 1MB. */
+    uint32_t    data_bits;      /* least-significant 3 bits are flags, the
+                                   rest are reserved (see above) */
+    uint32_t    reserved2;
+} vhdx_metadata_table_entry;
+
+#define VHDX_PARAMS_LEAVE_BLOCKS_ALLOCED 0x01   /* Do not change any blocks to
+                                                   be BLOCK_NOT_PRESENT.
+                                                   If set indicates a fixed
+                                                   size VHDX file */
+#define VHDX_PARAMS_HAS_PARENT           0x02    /* has parent / backing file */
+typedef struct QEMU_PACKED vhdx_file_parameters {
+    uint32_t    block_size;             /* size of each payload block, always
+                                           power of 2, <= 256MB and >= 1MB. */
+    uint32_t data_bits;     /* least-significant 2 bits are flags, the rest
+                               are reserved (see above) */
+} vhdx_file_parameters;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_size {
+    uint64_t    virtual_disk_size;      /* Size of the virtual disk, in bytes.
+                                           Must be multiple of the sector size,
+                                           max of 64TB */
+} vhdx_virtual_disk_size;
+
+typedef struct QEMU_PACKED vhdx_page83_data {
+    uint8_t     page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} vhdx_page83_data;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_logical_sector_size {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
Quoted 9 lines

block/vhdx.h line 307

Quoted 6 lines
block/vhdx.h new file +6 -0
+} vhdx_virtual_disk_logical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_physical_sector_size {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_physical_sector_size;

What's the point with all the single-field structs?

Another case of laying out as it was in the spec, for readability. I tried to keep the header as faithful to the spec as possible; if you have a strong preference, I can drop these and just use the single field types inside vhdx.c.

Quoted 21 lines
Quoted 9 lines
block/vhdx.h new file +9 -0
+typedef struct QEMU_PACKED vhdx_parent_locator_header {
+    uint8_t     locator_type[16];       /* type of the parent virtual disk. */
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} vhdx_parent_locator_header;
+
+/* key and value strings are UNICODE strings, UTF-16 LE encoding, no NULs */
+typedef struct QEMU_PACKED vhdx_parent_locator_entry {

block/vhdx.h line 327

Quoted 10 lines
block/vhdx.h new file +10 -0
+    uint32_t    key_offset;             /* offset in metadata for key, > 0 */
+    uint32_t    value_offset;           /* offset in metadata for value, >0 */
+    uint16_t    key_length;             /* length of entry key, > 0 */
+    uint16_t    value_length;           /* length of entry value, > 0 */
+} vhdx_parent_locator_entry;
+
+
+/* ----- END VHDX SPECIFICATION STRUCTURES ---- */
+
+#endif

Kevin

#5 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

In reply to Jeff Cody

block/vhdx.h line 60

Quoted 8 lines · Jeff Cody
block/vhdx.h new file +8 -0
+/* ---- HEADER SECTION STRUCTURES ---- */
+
+/* Important note: these structures are as defined in the VHDX specification,
+ * including byte order and size.  However, without being packed structures,
+ * they will not match 1:1 data read from disk.  Rather than use potentially
+ * non-portable packed structures, data is copied from read buffers into
+ * the structures below.  However, for reference, please refrain from
+ * modifying these structures to something that does not represent the spec */

Outdated comment? The patch uses QEMU_PACKED.

block/vhdx.h line 138

Quoted 2 lines
block/vhdx.h new file +2 -0
+/* Header for the region table block */
+#define VHDX_RT_MAGIC 0x69676572  /* 'regi ' */

Not worth respinnning, but should be 'regi' (4 bytes).

author
#6 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

Quoted 12 lines · Stefan Hajnoczi

block/vhdx.h line 60

Quoted 8 lines · Jeff Cody
block/vhdx.h new file +8 -0
+/* ---- HEADER SECTION STRUCTURES ---- */
+
+/* Important note: these structures are as defined in the VHDX specification,
+ * including byte order and size.  However, without being packed structures,
+ * they will not match 1:1 data read from disk.  Rather than use potentially
+ * non-portable packed structures, data is copied from read buffers into
+ * the structures below.  However, for reference, please refrain from
+ * modifying these structures to something that does not represent the spec */

Outdated comment? The patch uses QEMU_PACKED.

Yes, outdated comment, thanks.

Quoted 4 lines

block/vhdx.h line 138

Quoted 2 lines
block/vhdx.h new file +2 -0
+/* Header for the region table block */
+#define VHDX_RT_MAGIC 0x69676572  /* 'regi ' */

Not worth respinnning, but should be 'regi' (4 bytes).

Per Kevin's comments, I am removing all the _MAGIC's from the headers anyway, and using memcmp against the strings in the source file. So I'll take care of the outdated comment as well then.

#7 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

In reply to Jeff Cody

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 13 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 327 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 block/vhdx.h

block/vhdx.h line 297

Quoted 4 lines
block/vhdx.h new file +4 -0
+typedef struct QEMU_PACKED vhdx_page83_data {
+    uint8_t     page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} vhdx_page83_data;

Why uint8_t[16] instead of ms_guid?

block/vhdx.h line 310

Quoted 13 lines
block/vhdx.h new file +13 -0
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_logical_sector_size {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_logical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_physical_sector_size {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_physical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_parent_locator_header {
+    uint8_t     locator_type[16];       /* type of the parent virtual disk. */

Same question here.

block/vhdx.h line 314

Quoted 4 lines
block/vhdx.h new file +4 -0
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} vhdx_parent_locator_header;

Kevin

author
#8 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] block: vhdx header for the QEMU support of VHDX images

Quoted 21 lines · Kevin Wolf

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 13 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 327 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 block/vhdx.h

block/vhdx.h line 297

Quoted 4 lines
block/vhdx.h new file +4 -0
+typedef struct QEMU_PACKED vhdx_page83_data {
+    uint8_t     page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} vhdx_page83_data;

Why uint8_t[16] instead of ms_guid?

Quoted 16 lines

block/vhdx.h line 310

Quoted 13 lines
block/vhdx.h new file +13 -0
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_logical_sector_size {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_logical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_virtual_disk_physical_sector_size {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} vhdx_virtual_disk_physical_sector_size;
+
+typedef struct QEMU_PACKED vhdx_parent_locator_header {
+    uint8_t     locator_type[16];       /* type of the parent virtual disk. */

Same question here.

You are right, those should both be ms_guid.

Quoted 6 lines

block/vhdx.h line 314

Quoted 4 lines
block/vhdx.h new file +4 -0
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} vhdx_parent_locator_header;

Kevin

author
#9 lore raw

PATCH v2 3/5 block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody

This is the initial block driver framework for VHDX image support ( i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681
---
 block/Makefile.objs |   1 +
 block/vhdx.c        | 769 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  25 ++
 3 files changed, 795 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +769 -0

769 lines not on this page · raw message

block/vhdx.h +25 -0
index f5cf1ed..d71efac 100644
@@ -324,4 +324,29 @@ typedef struct QEMU_PACKED vhdx_parent_locator_entry {
 
 /* ----- END VHDX SPECIFICATION STRUCTURES ---- */
 
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset);
+
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset);
+
+
+static void leguid_to_cpus(ms_guid *guid)
+{
+    le32_to_cpus(&guid->data1);
+    le16_to_cpus(&guid->data2);
+    le16_to_cpus(&guid->data3);
+}
+
+static void cpu_to_leguids(ms_guid *guid)
+{
+    cpu_to_le32s(&guid->data1);
+    cpu_to_le16s(&guid->data2);
+    cpu_to_le16s(&guid->data3);
+}
+
+
+
+
+
 #endif
Signature · 1 line
1.8.1.4
#10 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 11 lines

This is the initial block driver framework for VHDX image support ( i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

Quoted 5 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

Okay, I need more time actually reading the spec and comparing this code to it before I can really say anything about the logic (I hope I'll get to it tomorrow), but for a start, let me point out the obvious things.

Quoted 129 lines
---
 block/Makefile.objs |   1 +
 block/vhdx.c        | 769 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  25 ++
 3 files changed, 795 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +104 -0
new file mode 100644
index 0000000..b0ea2ba
@@ -0,0 +1,769 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block/block_int.h"
+#include "qemu/module.h"
+#include "qemu/crc32c.h"
+#include "block/vhdx.h"
+
+
+/* Several metadata and region table data entries are identified by
+ * guids in  a MS-specific GUID format. */
+
+
+/* ------- Known Region Table GUIDs ---------------------- */
+static const ms_guid bat_guid =      { .data1 = 0x2dc27766,
+                                       .data2 = 0xf623,
+                                       .data3 = 0x4200,
+                                       .data4 = { 0x9d, 0x64, 0x11, 0x5e,
+                                                  0x9b, 0xfd, 0x4a, 0x08} };
+
+static const ms_guid metadata_guid = { .data1 = 0x8b7ca206,
+                                       .data2 = 0x4790,
+                                       .data3 = 0x4b9a,
+                                       .data4 = { 0xb8, 0xfe, 0x57, 0x5f,
+                                                  0x05, 0x0f, 0x88, 0x6e} };
+
+
+
+/* ------- Known Metadata Entry GUIDs ---------------------- */
+static const ms_guid file_param_guid =   { .data1 = 0xcaa16737,
+                                           .data2 = 0xfa36,
+                                           .data3 = 0x4d43,
+                                           .data4 = { 0xb3, 0xb6, 0x33, 0xf0,
+                                                      0xaa, 0x44, 0xe7, 0x6b} };
+
+static const ms_guid virtual_size_guid = { .data1 = 0x2FA54224,
+                                           .data2 = 0xcd1b,
+                                           .data3 = 0x4876,
+                                           .data4 = { 0xb2, 0x11, 0x5d, 0xbe,
+                                                      0xd8, 0x3b, 0xf4, 0xb8} };
+
+static const ms_guid page83_guid =       { .data1 = 0xbeca12ab,
+                                           .data2 = 0xb2e6,
+                                           .data3 = 0x4523,
+                                           .data4 = { 0x93, 0xef, 0xc3, 0x09,
+                                                      0xe0, 0x00, 0xc7, 0x46} };
+
+
+static const ms_guid phys_sector_guid =  { .data1 = 0xcda348c7,
+                                           .data2 = 0x445d,
+                                           .data3 = 0x4471,
+                                           .data4 = { 0x9c, 0xc9, 0xe9, 0x88,
+                                                      0x52, 0x51, 0xc5, 0x56} };
+
+static const ms_guid parent_locator_guid = { .data1 = 0xa8d35f2d,
+                                             .data2 = 0xb30b,
+                                             .data3 = 0x454d,
+                                             .data4 = { 0xab, 0xf7, 0xd3,
+                                                        0xd8, 0x48, 0x34,
+                                                        0xab, 0x0c} };
+
+static const ms_guid logical_sector_guid = { .data1 = 0x8141bf1d,
+                                             .data2 = 0xa96f,
+                                             .data3 = 0x4709,
+                                             .data4 = { 0xba, 0x47, 0xf2,
+                                                        0x33, 0xa8, 0xfa,
+                                                        0xab, 0x5f} };
+
+/* Each parent type must have a valid GUID; this is for parent images
+ * of type 'VHDX'.  If we were to allow e.g. a QCOW2 parent, we would
+ * need to make up our own QCOW2 GUID type */
+static const ms_guid parent_vhdx_guid = { .data1 = 0xb04aefb7,
+                                          .data2 = 0xd19e,
+                                          .data3 = 0x4a81,
+                                          .data4 = { 0xb7, 0x89, 0x25, 0xb8,
+                                                     0xe9, 0x44, 0x59, 0x13} };
+
+
+#define META_FILE_PARAMETER_PRESENT      0x01
+#define META_VIRTUAL_DISK_SIZE_PRESENT   0x02
+#define META_PAGE_83_PRESENT             0x04
+#define META_LOGICAL_SECTOR_SIZE_PRESENT 0x08
+#define META_PHYS_SECTOR_SIZE_PRESENT    0x10
+#define META_PARENT_LOCATOR_PRESENT      0x20
+
+#define META_ALL_PRESENT    \
+    (META_FILE_PARAMETER_PRESENT | META_VIRTUAL_DISK_SIZE_PRESENT | \
+     META_PAGE_83_PRESENT | META_LOGICAL_SECTOR_SIZE_PRESENT | \
+     META_PHYS_SECTOR_SIZE_PRESENT)

block/vhdx.c line 114

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+typedef struct vhdx_metadata_entries {
+    vhdx_metadata_table_entry file_parameters_entry;
+    vhdx_metadata_table_entry virtual_disk_size_entry;
+    vhdx_metadata_table_entry page83_data_entry;
+    vhdx_metadata_table_entry logical_sector_size_entry;
+    vhdx_metadata_table_entry phys_sector_size_entry;
+    vhdx_metadata_table_entry parent_locator_entry;
+    uint16_t present;
+} vhdx_metadata_entries;

Should everything before this line actually be in the header?

Quoted 36 lines
block/vhdx.c new file +36 -0
+
+typedef struct BDRVVHDXState {
+    CoMutex lock;
+
+    int curr_header;
+    vhdx_header *headers[2];
+
+    vhdx_region_table_header rt;
+    vhdx_region_table_entry bat_rt;         /* region table for the BAT */
+    vhdx_region_table_entry metadata_rt;    /* region table for the metadata */
+
+    vhdx_metadata_table_header  metadata_hdr;
+    vhdx_metadata_entries metadata_entries;
+
+    vhdx_file_parameters params;
+    uint32_t block_size;
+    uint32_t block_size_bits;
+    uint32_t sectors_per_block;
+    uint32_t sectors_per_block_bits;
+
+    uint64_t virtual_disk_size;
+    uint32_t logical_sector_size;
+    uint32_t physical_sector_size;
+
+    uint64_t chunk_ratio;
+    uint32_t chunk_ratio_bits;
+    uint32_t logical_sector_size_bits;
+
+    uint32_t bat_entries;
+    vhdx_bat_entry *bat;
+    uint64_t bat_offset;
+
+    vhdx_parent_locator_header parent_header;
+    vhdx_parent_locator_entry *parent_entries;
+
+} BDRVVHDXState;

block/vhdx.c line 161

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset)
+{
+    uint32_t crc_new;
+    uint32_t crc_orig;
+    assert(buf != NULL);
+
+    if (crc_offset > 0) {
+        memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));

Please add spaces before and after operators. I saw this a lot in this series. I'm relatively sure that you can use checkpatch.pl to get all instances pointed out.

Quoted 45 lines
block/vhdx.c new file +45 -0
+        memset(buf+crc_offset, 0, sizeof(crc_orig));
+    }
+
+    crc_new = crc32c(crc, buf, size);
+    if (crc_offset > 0) {
+        memcpy(buf+crc_offset, &crc_orig, sizeof(crc_orig));
+    }
+
+    return crc_new;
+}
+
+/* Validates the checksum of the buffer, with an in-place CRC.
+ *
+ * Zero is substituted during crc calculation for the original crc field,
+ * and the crc field is restored afterwards.  But the buffer will be modifed
+ * during the calculation, so this may not be not suitable for multi-threaded
+ * use.
+ *
+ * crc_offset: byte offset in buf of the buffer crc
+ * buf: buffer pointer
+ * size: size of buffer (must be > crc_offset+4)
+ *
+ * returns true if checksum is valid, false otherwise
+ */
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset)
+{
+    uint32_t crc_orig;
+    uint32_t crc;
+
+    assert(buf != NULL);
+    assert(size > (crc_offset+4));
+
+    memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+    crc_orig = le32_to_cpu(crc_orig);
+
+    crc = vhdx_checksum_calc(0xffffffff, buf, size, crc_offset);
+
+    return crc == crc_orig;
+}
+
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *      - The header section is fixed size - 1 MB
+ *      - The header section is always the first "object"

block/vhdx.c line 216

Quoted 10 lines
block/vhdx.c new file +10 -0
+ *      - The first 64KB of the header is the File Identifier
+ *      - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *      - The following 512 bytes constitute a UTF-16 string identifiying the
+ *        software that created the file, and is optional and diagnostic only.
+ *
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+    if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {

There's a VHDX_FILE_ID_MAGIC constant in the header. Don't you want to use it?

Quoted 68 lines
block/vhdx.c new file +68 -0
+        return 100;
+    }
+    return 0;
+}
+
+/* All VHDX structures on disk are little endian */
+static void vhdx_header_le_import(vhdx_header *h)
+{
+    assert(h != NULL);
+
+    le32_to_cpus(&h->signature);
+    le32_to_cpus(&h->checksum);
+    le64_to_cpus(&h->sequence_number);
+
+    leguid_to_cpus(&h->file_write_guid);
+    leguid_to_cpus(&h->data_write_guid);
+    leguid_to_cpus(&h->log_guid);
+
+    le16_to_cpus(&h->log_version);
+    le16_to_cpus(&h->version);
+    le32_to_cpus(&h->log_length);
+    le64_to_cpus(&h->log_offset);
+}
+
+
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;
+    uint8_t *buffer;
+
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;
+
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&

block/vhdx.c line 294

Quoted 10 lines
block/vhdx.c new file +10 -0
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }
+
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
+        printf("NO VALID HEADER\n");

Make that an qerror_report() at least.

Quoted 397 lines
block/vhdx.c new file +397 -0
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    vhdx_region_table_entry rt_entry;
+    int i;
+
+    /* We have to read the whole 64KB block, because the crc32 is over the
+     * whole block */
+    buffer = qemu_blockalign(bs, VHDX_HEADER_BLOCK_SIZE);
+
+    ret = bdrv_pread(bs->file, VHDX_REGION_TABLE_OFFSET, buffer,
+                    VHDX_HEADER_BLOCK_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    memcpy(&s->rt, buffer, sizeof(s->rt));
+    le32_to_cpus(&s->rt.signature);
+    le32_to_cpus(&s->rt.checksum);
+    le32_to_cpus(&s->rt.entry_count);
+    le32_to_cpus(&s->rt.reserved);
+    offset += sizeof(s->rt);
+
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);
+
+        leguid_to_cpus(&rt_entry.guid);
+        le64_to_cpus(&rt_entry.file_offset);
+        le32_to_cpus(&rt_entry.length);
+        le32_to_cpus(&rt_entry.data_bits);
+
+        /* see if we recognize the entry */
+        if (guid_eq(rt_entry.guid, bat_guid)) {
+            s->bat_rt = rt_entry;
+            continue;
+        }
+
+        if (guid_eq(rt_entry.guid, metadata_guid)) {
+            s->metadata_rt = rt_entry;
+            continue;
+        }
+
+        if (rt_entry.data_bits & VHDX_REGION_ENTRY_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto fail;
+        }
+    }
+    ret = 0;
+
+fail:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+
+/* Metadata initial parser
+ *
+ * This loads all the metadata entry fields.  This may cause additional
+ * fields to be processed (e.g. parent locator, etc..).
+ *
+ * There are 5 Metadata items that are always required:
+ *      - File Parameters (block size, has a parent)
+ *      - Virtual Disk Size (size, in bytes, of the virtual drive)
+ *      - Page 83 Data (scsi page 83 guid)
+ *      - Logical Sector Size (logical sector size in bytes, either 512 or
+ *                             4096.  We only support 512 currently)
+ *      - Physical Sector Size (512 or 4096)
+ *
+ * Also, if the File Parameters indicate this is a differencing file,
+ * we must also look for the Parent Locator metadata item.
+ */
+static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    int i = 0;
+    uint32_t block_size, sectors_per_block, logical_sector_size;
+    uint64_t chunk_ratio;
+    vhdx_metadata_table_entry md_entry;
+
+    buffer = qemu_blockalign(bs, VHDX_METADATA_TABLE_MAX_SIZE);
+
+    ret = bdrv_pread(bs->file, s->metadata_rt.file_offset, buffer,
+                     VHDX_METADATA_TABLE_MAX_SIZE);
+    if (ret < 0) {
+        goto exit;
+    }
+    memcpy(&s->metadata_hdr, buffer, sizeof(s->metadata_hdr));
+    offset += sizeof(s->metadata_hdr);
+
+    le64_to_cpus(&s->metadata_hdr.signature);
+    le16_to_cpus(&s->metadata_hdr.reserved);
+    le16_to_cpus(&s->metadata_hdr.entry_count);
+
+    if (s->metadata_hdr.signature != VHDX_METADATA_MAGIC) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    s->metadata_entries.present = 0;
+
+    if ((s->metadata_hdr.entry_count * sizeof(md_entry)) >
+        (VHDX_METADATA_TABLE_MAX_SIZE - offset)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    for (i = 0; i < s->metadata_hdr.entry_count; i++) {
+        memcpy(&md_entry, buffer+offset, sizeof(md_entry));
+        offset += sizeof(md_entry);
+
+        leguid_to_cpus(&md_entry.item_id);
+        le32_to_cpus(&md_entry.offset);
+        le32_to_cpus(&md_entry.length);
+        le32_to_cpus(&md_entry.data_bits);
+        le32_to_cpus(&md_entry.reserved2);
+
+        if (guid_eq(md_entry.item_id, file_param_guid)) {
+            s->metadata_entries.file_parameters_entry = md_entry;
+            s->metadata_entries.present |= META_FILE_PARAMETER_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, virtual_size_guid)) {
+            s->metadata_entries.virtual_disk_size_entry = md_entry;
+            s->metadata_entries.present |= META_VIRTUAL_DISK_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, page83_guid)) {
+            s->metadata_entries.page83_data_entry = md_entry;
+            s->metadata_entries.present |= META_PAGE_83_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, logical_sector_guid)) {
+            s->metadata_entries.logical_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_LOGICAL_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, phys_sector_guid)) {
+            s->metadata_entries.phys_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_PHYS_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, parent_locator_guid)) {
+            s->metadata_entries.parent_locator_entry = md_entry;
+            s->metadata_entries.present |= META_PARENT_LOCATOR_PRESENT;
+            continue;
+        }
+
+        if (md_entry.data_bits & VHDX_META_FLAGS_IS_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto exit;
+        }
+    }
+
+    if (s->metadata_entries.present != META_ALL_PRESENT) {
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.file_parameters_entry.offset
+                                         + s->metadata_rt.file_offset,
+                     &s->params,
+                     sizeof(s->params));
+
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le32_to_cpus(&s->params.block_size);
+    le32_to_cpus(&s->params.data_bits);
+
+
+    /* We now have the file parameters, so we can tell if this is a
+     * differencing file (i.e.. has_parent), is dynamic or fixed
+     * sized (leave_blocks_allocated), and the block size */
+
+    /* The parent locator required iff the file parameters has_parent set */
+    if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+        if (s->metadata_entries.present & ~META_PARENT_LOCATOR_PRESENT) {
+            /* TODO: parse  parent locator fields */
+            ret = -ENOTSUP; /* temp, until differencing files are supported */
+            goto exit;
+        } else {
+            /* if has_parent is set, but there is not parent locator present,
+             * then that is an invalid combination */
+            ret = -EINVAL;
+            goto exit;
+        }
+    }
+
+    /* determine virtual disk size, logical sector size,
+     * and phys sector size */
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.virtual_disk_size_entry.offset
+                                           + s->metadata_rt.file_offset,
+                     &s->virtual_disk_size,
+                     sizeof(uint64_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.logical_sector_size_entry.offset
+                                             + s->metadata_rt.file_offset,
+                     &s->logical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.phys_sector_size_entry.offset
+                                          + s->metadata_rt.file_offset,
+                     &s->physical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le64_to_cpus(&s->virtual_disk_size);
+    le32_to_cpus(&s->logical_sector_size);
+    le32_to_cpus(&s->physical_sector_size);
+
+    if (s->logical_sector_size == 0 || s->params.block_size == 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    /* both block_size and sector_size are guaranteed powers of 2 */
+    s->sectors_per_block = s->params.block_size / s->logical_sector_size;
+    s->chunk_ratio = (VHDX_MAX_SECTORS_PER_BLOCK) *
+                     (uint64_t)s->logical_sector_size /
+                     (uint64_t)s->params.block_size;
+
+    /* These values are ones we will want to use for division / multiplication
+     * later on, and they are all guaranteed (per the spec) to be powers of 2,
+     * so we can take advantage of that for shift operations during
+     * reads/writes */
+    logical_sector_size = s->logical_sector_size;
+    if (logical_sector_size & (logical_sector_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    sectors_per_block = s->sectors_per_block;
+    if (sectors_per_block & (sectors_per_block - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    chunk_ratio = s->chunk_ratio;
+    if (chunk_ratio & (chunk_ratio - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    block_size = s->params.block_size;
+    if (block_size & (block_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }
+
+    if (s->logical_sector_size != BDRV_SECTOR_SIZE) {
+        printf("VHDX error - QEMU only supports 512 byte sector sizes\n");
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = 0;
+
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+/* Parse the replay log.  Per the VHDX spec, if the log is present
+ * it must be replayed prior to opening the file, even read-only.
+ *
+ * If read-only, we must replay the log in RAM (or refuse to open
+ * a dirty VHDX file read-only */
+static int vhdx_parse_log(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    int i;
+    vhdx_header *hdr;
+
+    hdr = s->headers[s->curr_header];
+
+    /* either either the log guid, or log length is zero,
+     * then a replay log is present */
+    for (i = 0; i < sizeof(hdr->log_guid.data4); i++) {
+        ret |= hdr->log_guid.data4[i];
+    }
+    if (hdr->log_guid.data1 == 0 &&
+        hdr->log_guid.data2 == 0 &&
+        hdr->log_guid.data3 == 0 &&
+        ret == 0) {
+        goto exit;
+    }
+
+    /* per spec, only log version of 0 is supported */
+    if (hdr->log_version != 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    if (hdr->log_length == 0) {
+        goto exit;
+    }
+
+    /* We currently do not support images with logs to replay */
+    ret = -ENOTSUP;
+
+exit:
+    return ret;
+}
+
+
+static int vhdx_open(BlockDriverState *bs, QDict *options, int flags)
+{
+    BDRVVHDXState *s = bs->opaque;
+    int ret = 0;
+    int i;
+
+    s->bat = NULL;
+
+    qemu_co_mutex_init(&s->lock);
+
+    ret = vhdx_parse_header(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_parse_log(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_open_region_tables(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_parse_metadata(bs, s);
+    if (ret) {
+        goto fail;
+    }
+    s->block_size = s->params.block_size;

block/vhdx.c line 701

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+    /* the VHDX spec dictates that virtual_disk_size is always a multiple of
+     * logical_sector_size */
+    bs->total_sectors = s->virtual_disk_size / s->logical_sector_size;
+
+    s->bat_offset = s->bat_rt.file_offset;
+    s->bat_entries = s->bat_rt.length / sizeof(vhdx_bat_entry);
+    s->bat = qemu_blockalign(bs, s->bat_rt.length);
+
+    ret = bdrv_pread(bs->file, s->bat_offset, s->bat, s->bat_rt.length);

No error checking?

Quoted 4 lines
block/vhdx.c new file +4 -0
+    for (i = 0; i < s->bat_entries; i++) {
+        le64_to_cpus(&s->bat[i]);
+    }
+

block/vhdx.c line 716

Quoted 10 lines
block/vhdx.c new file +10 -0
+    if (flags & BDRV_O_RDWR) {
+        ret = -ENOTSUP;
+        goto fail;
+    }
+
+    /* TODO: differencing files, read, write */
+
+    return 0;
+fail:
+    qemu_vfree(s->bat);

This doesn't consider all the structs that were allocated by functions called in vhdx_open(). Here the same things should be freed as in vhdx_close().

Quoted 28 lines
block/vhdx.c new file +28 -0
+    return ret;
+}
+
+static int vhdx_reopen_prepare(BDRVReopenState *state,
+                               BlockReopenQueue *queue, Error **errp)
+{
+    return 0;
+}
+
+
+static coroutine_fn int vhdx_co_readv(BlockDriverState *bs, int64_t sector_num,
+                                      int nb_sectors, QEMUIOVector *qiov)
+{
+    return -ENOTSUP;
+}
+
+
+
+static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num,
+                                      int nb_sectors, QEMUIOVector *qiov)
+{
+    return -ENOTSUP;
+}
+
+
+static void vhdx_close(BlockDriverState *bs)
+{
+    BDRVVHDXState *s = bs->opaque;

block/vhdx.c line 754

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+    qemu_vfree(s->headers[0]);
+    qemu_vfree(s->headers[1]);
+    qemu_vfree(s->bat);
+    qemu_vfree(s->parent_entries);
+}
+
+static BlockDriver bdrv_vhdx = {
+    .format_name    = "vhdx",
+    .instance_size  = sizeof(BDRVVHDXState),

Use the same alignment for = as below?

block/vhdx.c line 762

Quoted 7 lines
block/vhdx.c new file +7 -0
+    .bdrv_probe             = vhdx_probe,
+    .bdrv_open              = vhdx_open,
+    .bdrv_close             = vhdx_close,
+    .bdrv_reopen_prepare    = vhdx_reopen_prepare,
+    .bdrv_co_readv          = vhdx_co_readv,
+    .bdrv_co_writev         = vhdx_co_writev,
+};

Kevin

author
#11 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Quoted 206 lines · Kevin Wolf

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 11 lines

This is the initial block driver framework for VHDX image support ( i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

Quoted 5 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

Okay, I need more time actually reading the spec and comparing this code to it before I can really say anything about the logic (I hope I'll get to it tomorrow), but for a start, let me point out the obvious things.

Quoted 129 lines
---
 block/Makefile.objs |   1 +
 block/vhdx.c        | 769 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  25 ++
 3 files changed, 795 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +104 -0
new file mode 100644
index 0000000..b0ea2ba
@@ -0,0 +1,769 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block/block_int.h"
+#include "qemu/module.h"
+#include "qemu/crc32c.h"
+#include "block/vhdx.h"
+
+
+/* Several metadata and region table data entries are identified by
+ * guids in  a MS-specific GUID format. */
+
+
+/* ------- Known Region Table GUIDs ---------------------- */
+static const ms_guid bat_guid =      { .data1 = 0x2dc27766,
+                                       .data2 = 0xf623,
+                                       .data3 = 0x4200,
+                                       .data4 = { 0x9d, 0x64, 0x11, 0x5e,
+                                                  0x9b, 0xfd, 0x4a, 0x08} };
+
+static const ms_guid metadata_guid = { .data1 = 0x8b7ca206,
+                                       .data2 = 0x4790,
+                                       .data3 = 0x4b9a,
+                                       .data4 = { 0xb8, 0xfe, 0x57, 0x5f,
+                                                  0x05, 0x0f, 0x88, 0x6e} };
+
+
+
+/* ------- Known Metadata Entry GUIDs ---------------------- */
+static const ms_guid file_param_guid =   { .data1 = 0xcaa16737,
+                                           .data2 = 0xfa36,
+                                           .data3 = 0x4d43,
+                                           .data4 = { 0xb3, 0xb6, 0x33, 0xf0,
+                                                      0xaa, 0x44, 0xe7, 0x6b} };
+
+static const ms_guid virtual_size_guid = { .data1 = 0x2FA54224,
+                                           .data2 = 0xcd1b,
+                                           .data3 = 0x4876,
+                                           .data4 = { 0xb2, 0x11, 0x5d, 0xbe,
+                                                      0xd8, 0x3b, 0xf4, 0xb8} };
+
+static const ms_guid page83_guid =       { .data1 = 0xbeca12ab,
+                                           .data2 = 0xb2e6,
+                                           .data3 = 0x4523,
+                                           .data4 = { 0x93, 0xef, 0xc3, 0x09,
+                                                      0xe0, 0x00, 0xc7, 0x46} };
+
+
+static const ms_guid phys_sector_guid =  { .data1 = 0xcda348c7,
+                                           .data2 = 0x445d,
+                                           .data3 = 0x4471,
+                                           .data4 = { 0x9c, 0xc9, 0xe9, 0x88,
+                                                      0x52, 0x51, 0xc5, 0x56} };
+
+static const ms_guid parent_locator_guid = { .data1 = 0xa8d35f2d,
+                                             .data2 = 0xb30b,
+                                             .data3 = 0x454d,
+                                             .data4 = { 0xab, 0xf7, 0xd3,
+                                                        0xd8, 0x48, 0x34,
+                                                        0xab, 0x0c} };
+
+static const ms_guid logical_sector_guid = { .data1 = 0x8141bf1d,
+                                             .data2 = 0xa96f,
+                                             .data3 = 0x4709,
+                                             .data4 = { 0xba, 0x47, 0xf2,
+                                                        0x33, 0xa8, 0xfa,
+                                                        0xab, 0x5f} };
+
+/* Each parent type must have a valid GUID; this is for parent images
+ * of type 'VHDX'.  If we were to allow e.g. a QCOW2 parent, we would
+ * need to make up our own QCOW2 GUID type */
+static const ms_guid parent_vhdx_guid = { .data1 = 0xb04aefb7,
+                                          .data2 = 0xd19e,
+                                          .data3 = 0x4a81,
+                                          .data4 = { 0xb7, 0x89, 0x25, 0xb8,
+                                                     0xe9, 0x44, 0x59, 0x13} };
+
+
+#define META_FILE_PARAMETER_PRESENT      0x01
+#define META_VIRTUAL_DISK_SIZE_PRESENT   0x02
+#define META_PAGE_83_PRESENT             0x04
+#define META_LOGICAL_SECTOR_SIZE_PRESENT 0x08
+#define META_PHYS_SECTOR_SIZE_PRESENT    0x10
+#define META_PARENT_LOCATOR_PRESENT      0x20
+
+#define META_ALL_PRESENT    \
+    (META_FILE_PARAMETER_PRESENT | META_VIRTUAL_DISK_SIZE_PRESENT | \
+     META_PAGE_83_PRESENT | META_LOGICAL_SECTOR_SIZE_PRESENT | \
+     META_PHYS_SECTOR_SIZE_PRESENT)

block/vhdx.c line 114

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+typedef struct vhdx_metadata_entries {
+    vhdx_metadata_table_entry file_parameters_entry;
+    vhdx_metadata_table_entry virtual_disk_size_entry;
+    vhdx_metadata_table_entry page83_data_entry;
+    vhdx_metadata_table_entry logical_sector_size_entry;
+    vhdx_metadata_table_entry phys_sector_size_entry;
+    vhdx_metadata_table_entry parent_locator_entry;
+    uint16_t present;
+} vhdx_metadata_entries;

Should everything before this line actually be in the header?

Quoted 42 lines
block/vhdx.c new file +42 -0
+
+typedef struct BDRVVHDXState {
+    CoMutex lock;
+
+    int curr_header;
+    vhdx_header *headers[2];
+
+    vhdx_region_table_header rt;
+    vhdx_region_table_entry bat_rt;         /* region table for the BAT */
+    vhdx_region_table_entry metadata_rt;    /* region table for the metadata */
+
+    vhdx_metadata_table_header  metadata_hdr;
+    vhdx_metadata_entries metadata_entries;
+
+    vhdx_file_parameters params;
+    uint32_t block_size;
+    uint32_t block_size_bits;
+    uint32_t sectors_per_block;
+    uint32_t sectors_per_block_bits;
+
+    uint64_t virtual_disk_size;
+    uint32_t logical_sector_size;
+    uint32_t physical_sector_size;
+
+    uint64_t chunk_ratio;
+    uint32_t chunk_ratio_bits;
+    uint32_t logical_sector_size_bits;
+
+    uint32_t bat_entries;
+    vhdx_bat_entry *bat;
+    uint64_t bat_offset;
+
+    vhdx_parent_locator_header parent_header;
+    vhdx_parent_locator_entry *parent_entries;
+
+} BDRVVHDXState;
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset)
+{
+    uint32_t crc_new;
+    uint32_t crc_orig;
Quoted 9 lines

block/vhdx.c line 161

Quoted 4 lines
block/vhdx.c new file +4 -0
+    assert(buf != NULL);
+
+    if (crc_offset > 0) {
+        memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));

Please add spaces before and after operators. I saw this a lot in this series. I'm relatively sure that you can use checkpatch.pl to get all instances pointed out.

Sure, I will seek those out and change them. BTW, I ran it checkpatch.pl, and it apparently missed the one above & below.

Quoted 50 lines
Quoted 50 lines
block/vhdx.c new file +50 -0
+        memset(buf+crc_offset, 0, sizeof(crc_orig));
+    }
+
+    crc_new = crc32c(crc, buf, size);
+    if (crc_offset > 0) {
+        memcpy(buf+crc_offset, &crc_orig, sizeof(crc_orig));
+    }
+
+    return crc_new;
+}
+
+/* Validates the checksum of the buffer, with an in-place CRC.
+ *
+ * Zero is substituted during crc calculation for the original crc field,
+ * and the crc field is restored afterwards.  But the buffer will be modifed
+ * during the calculation, so this may not be not suitable for multi-threaded
+ * use.
+ *
+ * crc_offset: byte offset in buf of the buffer crc
+ * buf: buffer pointer
+ * size: size of buffer (must be > crc_offset+4)
+ *
+ * returns true if checksum is valid, false otherwise
+ */
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset)
+{
+    uint32_t crc_orig;
+    uint32_t crc;
+
+    assert(buf != NULL);
+    assert(size > (crc_offset+4));
+
+    memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+    crc_orig = le32_to_cpu(crc_orig);
+
+    crc = vhdx_checksum_calc(0xffffffff, buf, size, crc_offset);
+
+    return crc == crc_orig;
+}
+
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *      - The header section is fixed size - 1 MB
+ *      - The header section is always the first "object"
+ *      - The first 64KB of the header is the File Identifier
+ *      - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *      - The following 512 bytes constitute a UTF-16 string identifiying the
+ *        software that created the file, and is optional and diagnostic only.
+ *
Quoted 9 lines

block/vhdx.c line 216

Quoted 5 lines
block/vhdx.c new file +5 -0
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+    if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {

There's a VHDX_FILE_ID_MAGIC constant in the header. Don't you want to use it?

Maybe I should remove all the magics from the header. I could use it, but I think this is more readable and perhaps easier to follow against the spec. For the upcoming log patches, there are other magic items that are compared against strings rather than the magic defines as well.

If you prefer I compared against the _MAGIC defines in the header, I have no problem with that.

Quoted 72 lines
Quoted 72 lines
block/vhdx.c new file +72 -0
+        return 100;
+    }
+    return 0;
+}
+
+/* All VHDX structures on disk are little endian */
+static void vhdx_header_le_import(vhdx_header *h)
+{
+    assert(h != NULL);
+
+    le32_to_cpus(&h->signature);
+    le32_to_cpus(&h->checksum);
+    le64_to_cpus(&h->sequence_number);
+
+    leguid_to_cpus(&h->file_write_guid);
+    leguid_to_cpus(&h->data_write_guid);
+    leguid_to_cpus(&h->log_guid);
+
+    le16_to_cpus(&h->log_version);
+    le16_to_cpus(&h->version);
+    le32_to_cpus(&h->log_length);
+    le64_to_cpus(&h->log_offset);
+}
+
+
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;
+    uint8_t *buffer;
+
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;
+
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }
+
Quoted 9 lines

block/vhdx.c line 294

Quoted 6 lines
block/vhdx.c new file +6 -0
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
+        printf("NO VALID HEADER\n");

Make that an qerror_report() at least.

OK

Quoted 401 lines
Quoted 401 lines
block/vhdx.c new file +401 -0
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    vhdx_region_table_entry rt_entry;
+    int i;
+
+    /* We have to read the whole 64KB block, because the crc32 is over the
+     * whole block */
+    buffer = qemu_blockalign(bs, VHDX_HEADER_BLOCK_SIZE);
+
+    ret = bdrv_pread(bs->file, VHDX_REGION_TABLE_OFFSET, buffer,
+                    VHDX_HEADER_BLOCK_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    memcpy(&s->rt, buffer, sizeof(s->rt));
+    le32_to_cpus(&s->rt.signature);
+    le32_to_cpus(&s->rt.checksum);
+    le32_to_cpus(&s->rt.entry_count);
+    le32_to_cpus(&s->rt.reserved);
+    offset += sizeof(s->rt);
+
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);
+
+        leguid_to_cpus(&rt_entry.guid);
+        le64_to_cpus(&rt_entry.file_offset);
+        le32_to_cpus(&rt_entry.length);
+        le32_to_cpus(&rt_entry.data_bits);
+
+        /* see if we recognize the entry */
+        if (guid_eq(rt_entry.guid, bat_guid)) {
+            s->bat_rt = rt_entry;
+            continue;
+        }
+
+        if (guid_eq(rt_entry.guid, metadata_guid)) {
+            s->metadata_rt = rt_entry;
+            continue;
+        }
+
+        if (rt_entry.data_bits & VHDX_REGION_ENTRY_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto fail;
+        }
+    }
+    ret = 0;
+
+fail:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+
+/* Metadata initial parser
+ *
+ * This loads all the metadata entry fields.  This may cause additional
+ * fields to be processed (e.g. parent locator, etc..).
+ *
+ * There are 5 Metadata items that are always required:
+ *      - File Parameters (block size, has a parent)
+ *      - Virtual Disk Size (size, in bytes, of the virtual drive)
+ *      - Page 83 Data (scsi page 83 guid)
+ *      - Logical Sector Size (logical sector size in bytes, either 512 or
+ *                             4096.  We only support 512 currently)
+ *      - Physical Sector Size (512 or 4096)
+ *
+ * Also, if the File Parameters indicate this is a differencing file,
+ * we must also look for the Parent Locator metadata item.
+ */
+static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    int i = 0;
+    uint32_t block_size, sectors_per_block, logical_sector_size;
+    uint64_t chunk_ratio;
+    vhdx_metadata_table_entry md_entry;
+
+    buffer = qemu_blockalign(bs, VHDX_METADATA_TABLE_MAX_SIZE);
+
+    ret = bdrv_pread(bs->file, s->metadata_rt.file_offset, buffer,
+                     VHDX_METADATA_TABLE_MAX_SIZE);
+    if (ret < 0) {
+        goto exit;
+    }
+    memcpy(&s->metadata_hdr, buffer, sizeof(s->metadata_hdr));
+    offset += sizeof(s->metadata_hdr);
+
+    le64_to_cpus(&s->metadata_hdr.signature);
+    le16_to_cpus(&s->metadata_hdr.reserved);
+    le16_to_cpus(&s->metadata_hdr.entry_count);
+
+    if (s->metadata_hdr.signature != VHDX_METADATA_MAGIC) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    s->metadata_entries.present = 0;
+
+    if ((s->metadata_hdr.entry_count * sizeof(md_entry)) >
+        (VHDX_METADATA_TABLE_MAX_SIZE - offset)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    for (i = 0; i < s->metadata_hdr.entry_count; i++) {
+        memcpy(&md_entry, buffer+offset, sizeof(md_entry));
+        offset += sizeof(md_entry);
+
+        leguid_to_cpus(&md_entry.item_id);
+        le32_to_cpus(&md_entry.offset);
+        le32_to_cpus(&md_entry.length);
+        le32_to_cpus(&md_entry.data_bits);
+        le32_to_cpus(&md_entry.reserved2);
+
+        if (guid_eq(md_entry.item_id, file_param_guid)) {
+            s->metadata_entries.file_parameters_entry = md_entry;
+            s->metadata_entries.present |= META_FILE_PARAMETER_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, virtual_size_guid)) {
+            s->metadata_entries.virtual_disk_size_entry = md_entry;
+            s->metadata_entries.present |= META_VIRTUAL_DISK_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, page83_guid)) {
+            s->metadata_entries.page83_data_entry = md_entry;
+            s->metadata_entries.present |= META_PAGE_83_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, logical_sector_guid)) {
+            s->metadata_entries.logical_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_LOGICAL_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, phys_sector_guid)) {
+            s->metadata_entries.phys_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_PHYS_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, parent_locator_guid)) {
+            s->metadata_entries.parent_locator_entry = md_entry;
+            s->metadata_entries.present |= META_PARENT_LOCATOR_PRESENT;
+            continue;
+        }
+
+        if (md_entry.data_bits & VHDX_META_FLAGS_IS_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto exit;
+        }
+    }
+
+    if (s->metadata_entries.present != META_ALL_PRESENT) {
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.file_parameters_entry.offset
+                                         + s->metadata_rt.file_offset,
+                     &s->params,
+                     sizeof(s->params));
+
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le32_to_cpus(&s->params.block_size);
+    le32_to_cpus(&s->params.data_bits);
+
+
+    /* We now have the file parameters, so we can tell if this is a
+     * differencing file (i.e.. has_parent), is dynamic or fixed
+     * sized (leave_blocks_allocated), and the block size */
+
+    /* The parent locator required iff the file parameters has_parent set */
+    if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+        if (s->metadata_entries.present & ~META_PARENT_LOCATOR_PRESENT) {
+            /* TODO: parse  parent locator fields */
+            ret = -ENOTSUP; /* temp, until differencing files are supported */
+            goto exit;
+        } else {
+            /* if has_parent is set, but there is not parent locator present,
+             * then that is an invalid combination */
+            ret = -EINVAL;
+            goto exit;
+        }
+    }
+
+    /* determine virtual disk size, logical sector size,
+     * and phys sector size */
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.virtual_disk_size_entry.offset
+                                           + s->metadata_rt.file_offset,
+                     &s->virtual_disk_size,
+                     sizeof(uint64_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.logical_sector_size_entry.offset
+                                             + s->metadata_rt.file_offset,
+                     &s->logical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.phys_sector_size_entry.offset
+                                          + s->metadata_rt.file_offset,
+                     &s->physical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le64_to_cpus(&s->virtual_disk_size);
+    le32_to_cpus(&s->logical_sector_size);
+    le32_to_cpus(&s->physical_sector_size);
+
+    if (s->logical_sector_size == 0 || s->params.block_size == 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    /* both block_size and sector_size are guaranteed powers of 2 */
+    s->sectors_per_block = s->params.block_size / s->logical_sector_size;
+    s->chunk_ratio = (VHDX_MAX_SECTORS_PER_BLOCK) *
+                     (uint64_t)s->logical_sector_size /
+                     (uint64_t)s->params.block_size;
+
+    /* These values are ones we will want to use for division / multiplication
+     * later on, and they are all guaranteed (per the spec) to be powers of 2,
+     * so we can take advantage of that for shift operations during
+     * reads/writes */
+    logical_sector_size = s->logical_sector_size;
+    if (logical_sector_size & (logical_sector_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    sectors_per_block = s->sectors_per_block;
+    if (sectors_per_block & (sectors_per_block - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    chunk_ratio = s->chunk_ratio;
+    if (chunk_ratio & (chunk_ratio - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    block_size = s->params.block_size;
+    if (block_size & (block_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }
+
+    if (s->logical_sector_size != BDRV_SECTOR_SIZE) {
+        printf("VHDX error - QEMU only supports 512 byte sector sizes\n");
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = 0;
+
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+/* Parse the replay log.  Per the VHDX spec, if the log is present
+ * it must be replayed prior to opening the file, even read-only.
+ *
+ * If read-only, we must replay the log in RAM (or refuse to open
+ * a dirty VHDX file read-only */
+static int vhdx_parse_log(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    int i;
+    vhdx_header *hdr;
+
+    hdr = s->headers[s->curr_header];
+
+    /* either either the log guid, or log length is zero,
+     * then a replay log is present */
+    for (i = 0; i < sizeof(hdr->log_guid.data4); i++) {
+        ret |= hdr->log_guid.data4[i];
+    }
+    if (hdr->log_guid.data1 == 0 &&
+        hdr->log_guid.data2 == 0 &&
+        hdr->log_guid.data3 == 0 &&
+        ret == 0) {
+        goto exit;
+    }
+
+    /* per spec, only log version of 0 is supported */
+    if (hdr->log_version != 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    if (hdr->log_length == 0) {
+        goto exit;
+    }
+
+    /* We currently do not support images with logs to replay */
+    ret = -ENOTSUP;
+
+exit:
+    return ret;
+}
+
+
+static int vhdx_open(BlockDriverState *bs, QDict *options, int flags)
+{
+    BDRVVHDXState *s = bs->opaque;
+    int ret = 0;
+    int i;
+
+    s->bat = NULL;
+
+    qemu_co_mutex_init(&s->lock);
+
+    ret = vhdx_parse_header(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_parse_log(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_open_region_tables(bs, s);
+    if (ret) {
+        goto fail;
+    }
+
+    ret = vhdx_parse_metadata(bs, s);
+    if (ret) {
+        goto fail;
+    }
+    s->block_size = s->params.block_size;
+
+    /* the VHDX spec dictates that virtual_disk_size is always a multiple of
+     * logical_sector_size */
+    bs->total_sectors = s->virtual_disk_size / s->logical_sector_size;
Quoted 9 lines

block/vhdx.c line 701

Quoted 6 lines
block/vhdx.c new file +6 -0
+
+    s->bat_offset = s->bat_rt.file_offset;
+    s->bat_entries = s->bat_rt.length / sizeof(vhdx_bat_entry);
+    s->bat = qemu_blockalign(bs, s->bat_rt.length);
+
+    ret = bdrv_pread(bs->file, s->bat_offset, s->bat, s->bat_rt.length);

No error checking?

Thanks, missed that, adding it in.

Quoted 10 lines
Quoted 10 lines
block/vhdx.c new file +10 -0
+    for (i = 0; i < s->bat_entries; i++) {
+        le64_to_cpus(&s->bat[i]);
+    }
+
+    if (flags & BDRV_O_RDWR) {
+        ret = -ENOTSUP;
+        goto fail;
+    }
+
+    /* TODO: differencing files, read, write */
Quoted 9 lines

block/vhdx.c line 716

Quoted 4 lines
block/vhdx.c new file +4 -0
+
+    return 0;
+fail:
+    qemu_vfree(s->bat);

This doesn't consider all the structs that were allocated by functions called in vhdx_open(). Here the same things should be freed as in vhdx_close().

OK. Those functions clean up after themselves, but you are right, if a failure happens later they should be cleaned up here as well.

Quoted 32 lines
Quoted 32 lines
block/vhdx.c new file +32 -0
+    return ret;
+}
+
+static int vhdx_reopen_prepare(BDRVReopenState *state,
+                               BlockReopenQueue *queue, Error **errp)
+{
+    return 0;
+}
+
+
+static coroutine_fn int vhdx_co_readv(BlockDriverState *bs, int64_t sector_num,
+                                      int nb_sectors, QEMUIOVector *qiov)
+{
+    return -ENOTSUP;
+}
+
+
+
+static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num,
+                                      int nb_sectors, QEMUIOVector *qiov)
+{
+    return -ENOTSUP;
+}
+
+
+static void vhdx_close(BlockDriverState *bs)
+{
+    BDRVVHDXState *s = bs->opaque;
+
+    qemu_vfree(s->headers[0]);
+    qemu_vfree(s->headers[1]);
+    qemu_vfree(s->bat);
Quoted 9 lines

block/vhdx.c line 754

Quoted 6 lines
block/vhdx.c new file +6 -0
+    qemu_vfree(s->parent_entries);
+}
+
+static BlockDriver bdrv_vhdx = {
+    .format_name    = "vhdx",
+    .instance_size  = sizeof(BDRVVHDXState),

Use the same alignment for = as below?

OK

Quoted 9 lines

block/vhdx.c line 762

Quoted 7 lines
block/vhdx.c new file +7 -0
+    .bdrv_probe             = vhdx_probe,
+    .bdrv_open              = vhdx_open,
+    .bdrv_close             = vhdx_close,
+    .bdrv_reopen_prepare    = vhdx_reopen_prepare,
+    .bdrv_co_readv          = vhdx_co_readv,
+    .bdrv_co_writev         = vhdx_co_writev,
+};

Kevin

#12 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Am 23.04.2013 um 18:11 hat Jeff Cody geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 10 lines

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 9 lines

+/*
+ * Per the MS VHDX Specification, for every VHDX file:

+ *      - The header section is fixed size - 1 MB
+ *      - The header section is always the first "object"
+ *      - The first 64KB of the header is the File Identifier
+ *      - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *      - The following 512 bytes constitute a UTF-16 string identifiying the

+ * software that created the file, and is optional and diagnostic only.
+ *

Quoted 9 lines

block/vhdx.c line 216

Quoted 5 lines

+ * Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */

block/vhdx.c new file +3 -0
+static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+    if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {

There's a VHDX_FILE_ID_MAGIC constant in the header. Don't you want to use it?

Quoted 8 lines

Maybe I should remove all the magics from the header. I could use it, but I think this is more readable and perhaps easier to follow against the spec. For the upcoming log patches, there are other magic items that are compared against strings rather than the magic defines as well.

If you prefer I compared against the _MAGIC defines in the header, I have no problem with that.

I don't mind using the strings, but then I think removing the numeric magic from the header is the right thing indeed.

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
block/vhdx.c new file +10 -0
+    for (i = 0; i < s->bat_entries; i++) {
+        le64_to_cpus(&s->bat[i]);
+    }
+
+    if (flags & BDRV_O_RDWR) {
+        ret = -ENOTSUP;
+        goto fail;
+    }
+
+    /* TODO: differencing files, read, write */
Quoted 9 lines

block/vhdx.c line 716

Quoted 4 lines
block/vhdx.c new file +4 -0
+
+    return 0;
+fail:
+    qemu_vfree(s->bat);

This doesn't consider all the structs that were allocated by functions called in vhdx_open(). Here the same things should be freed as in vhdx_close().

Quoted 2 lines

OK. Those functions clean up after themselves, but you are right, if a failure happens later they should be cleaned up here as well.

Yes, you can probably get away with not freeing whatever the last function call could have allocated if it cleans up after itself, but doing the full thing here is more obviously correct. (You may have to move the cleanup here instead of duplicating it, or make the functions' cleanup reset the pointers to NULL)

Kevin

#13 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody

block/vhdx.c line 344

Quoted 7 lines · Jeff Cody
block/vhdx.c new file +7 -0
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {

It's nice to avoid signed/unsigned comparisons. i should be uint32_t just like entry_count.

block/vhdx.c line 346

Quoted 2 lines
block/vhdx.c new file +2 -0
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);

Looks like we're trusting rt.entry_count to be a sane value? Need to prevent offset from exceeding buffer size.

block/vhdx.c line 604

Quoted 12 lines
block/vhdx.c new file +12 -0
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }

ctz()/clo() do this.

block/vhdx.c line 632

Quoted 9 lines
block/vhdx.c new file +9 -0
+static int vhdx_parse_log(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    int i;
+    vhdx_header *hdr;
+
+    hdr = s->headers[s->curr_header];
+
+    /* either either the log guid, or log length is zero,

either either

block/vhdx.c line 699

Quoted 3 lines
block/vhdx.c new file +3 -0
+    s->bat_offset = s->bat_rt.file_offset;
+    s->bat_entries = s->bat_rt.length / sizeof(vhdx_bat_entry);
+    s->bat = qemu_blockalign(bs, s->bat_rt.length);

No sanity check was done on bat_rt.length. If this allocation fails QEMU will exit. Could be used as a DoS if you can get someone to attach a malicious VHDX to their VM?

author
#14 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Quoted 11 lines · Stefan Hajnoczi

block/vhdx.c line 344

Quoted 7 lines · Jeff Cody
block/vhdx.c new file +7 -0
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {

It's nice to avoid signed/unsigned comparisons. i should be uint32_t just like entry_count.

I agree. I will also double check the other parsing routines (e.g. vhdx_parse_metadata()).

Quoted 7 lines

block/vhdx.c line 346

Quoted 3 lines
block/vhdx.c new file +2 -0
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);

Looks like we're trusting rt.entry_count to be a sane value? Need to prevent offset from exceeding buffer size.

Agree again, and I will also check the other parsers as well.

Quoted 13 lines

block/vhdx.c line 604

Quoted 12 lines
block/vhdx.c new file +12 -0
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }
Quoted 2 lines

ctz()/clo() do this.

Ah, yes! I will switch over to using those.

Quoted 12 lines

block/vhdx.c line 632

Quoted 9 lines
block/vhdx.c new file +9 -0
+static int vhdx_parse_log(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    int i;
+    vhdx_header *hdr;
+
+    hdr = s->headers[s->curr_header];
+
+    /* either either the log guid, or log length is zero,

either either

Thanks

Quoted 7 lines

block/vhdx.c line 699

Quoted 3 lines
block/vhdx.c new file +3 -0
+    s->bat_offset = s->bat_rt.file_offset;
+    s->bat_entries = s->bat_rt.length / sizeof(vhdx_bat_entry);
+    s->bat = qemu_blockalign(bs, s->bat_rt.length);

No sanity check was done on bat_rt.length. If this allocation fails QEMU will exit. Could be used as a DoS if you can get someone to attach a malicious VHDX to their VM?

Yes, bat_rt.length needs to be verified here as well. I will add that in.

#15 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 11 lines

This is the initial block driver framework for VHDX image support ( i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

Quoted 5 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

Okay, now I'm starting to actually compare the spec to your implementation. Comments inline.

Quoted 264 lines
 block/Makefile.objs |   1 +
 block/vhdx.c        | 769 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  25 ++
 3 files changed, 795 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +240 -0
new file mode 100644
index 0000000..b0ea2ba
@@ -0,0 +1,769 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block/block_int.h"
+#include "qemu/module.h"
+#include "qemu/crc32c.h"
+#include "block/vhdx.h"
+
+
+/* Several metadata and region table data entries are identified by
+ * guids in  a MS-specific GUID format. */
+
+
+/* ------- Known Region Table GUIDs ---------------------- */
+static const ms_guid bat_guid =      { .data1 = 0x2dc27766,
+                                       .data2 = 0xf623,
+                                       .data3 = 0x4200,
+                                       .data4 = { 0x9d, 0x64, 0x11, 0x5e,
+                                                  0x9b, 0xfd, 0x4a, 0x08} };
+
+static const ms_guid metadata_guid = { .data1 = 0x8b7ca206,
+                                       .data2 = 0x4790,
+                                       .data3 = 0x4b9a,
+                                       .data4 = { 0xb8, 0xfe, 0x57, 0x5f,
+                                                  0x05, 0x0f, 0x88, 0x6e} };
+
+
+
+/* ------- Known Metadata Entry GUIDs ---------------------- */
+static const ms_guid file_param_guid =   { .data1 = 0xcaa16737,
+                                           .data2 = 0xfa36,
+                                           .data3 = 0x4d43,
+                                           .data4 = { 0xb3, 0xb6, 0x33, 0xf0,
+                                                      0xaa, 0x44, 0xe7, 0x6b} };
+
+static const ms_guid virtual_size_guid = { .data1 = 0x2FA54224,
+                                           .data2 = 0xcd1b,
+                                           .data3 = 0x4876,
+                                           .data4 = { 0xb2, 0x11, 0x5d, 0xbe,
+                                                      0xd8, 0x3b, 0xf4, 0xb8} };
+
+static const ms_guid page83_guid =       { .data1 = 0xbeca12ab,
+                                           .data2 = 0xb2e6,
+                                           .data3 = 0x4523,
+                                           .data4 = { 0x93, 0xef, 0xc3, 0x09,
+                                                      0xe0, 0x00, 0xc7, 0x46} };
+
+
+static const ms_guid phys_sector_guid =  { .data1 = 0xcda348c7,
+                                           .data2 = 0x445d,
+                                           .data3 = 0x4471,
+                                           .data4 = { 0x9c, 0xc9, 0xe9, 0x88,
+                                                      0x52, 0x51, 0xc5, 0x56} };
+
+static const ms_guid parent_locator_guid = { .data1 = 0xa8d35f2d,
+                                             .data2 = 0xb30b,
+                                             .data3 = 0x454d,
+                                             .data4 = { 0xab, 0xf7, 0xd3,
+                                                        0xd8, 0x48, 0x34,
+                                                        0xab, 0x0c} };
+
+static const ms_guid logical_sector_guid = { .data1 = 0x8141bf1d,
+                                             .data2 = 0xa96f,
+                                             .data3 = 0x4709,
+                                             .data4 = { 0xba, 0x47, 0xf2,
+                                                        0x33, 0xa8, 0xfa,
+                                                        0xab, 0x5f} };
+
+/* Each parent type must have a valid GUID; this is for parent images
+ * of type 'VHDX'.  If we were to allow e.g. a QCOW2 parent, we would
+ * need to make up our own QCOW2 GUID type */
+static const ms_guid parent_vhdx_guid = { .data1 = 0xb04aefb7,
+                                          .data2 = 0xd19e,
+                                          .data3 = 0x4a81,
+                                          .data4 = { 0xb7, 0x89, 0x25, 0xb8,
+                                                     0xe9, 0x44, 0x59, 0x13} };
+
+
+#define META_FILE_PARAMETER_PRESENT      0x01
+#define META_VIRTUAL_DISK_SIZE_PRESENT   0x02
+#define META_PAGE_83_PRESENT             0x04
+#define META_LOGICAL_SECTOR_SIZE_PRESENT 0x08
+#define META_PHYS_SECTOR_SIZE_PRESENT    0x10
+#define META_PARENT_LOCATOR_PRESENT      0x20
+
+#define META_ALL_PRESENT    \
+    (META_FILE_PARAMETER_PRESENT | META_VIRTUAL_DISK_SIZE_PRESENT | \
+     META_PAGE_83_PRESENT | META_LOGICAL_SECTOR_SIZE_PRESENT | \
+     META_PHYS_SECTOR_SIZE_PRESENT)
+
+typedef struct vhdx_metadata_entries {
+    vhdx_metadata_table_entry file_parameters_entry;
+    vhdx_metadata_table_entry virtual_disk_size_entry;
+    vhdx_metadata_table_entry page83_data_entry;
+    vhdx_metadata_table_entry logical_sector_size_entry;
+    vhdx_metadata_table_entry phys_sector_size_entry;
+    vhdx_metadata_table_entry parent_locator_entry;
+    uint16_t present;
+} vhdx_metadata_entries;
+
+
+typedef struct BDRVVHDXState {
+    CoMutex lock;
+
+    int curr_header;
+    vhdx_header *headers[2];
+
+    vhdx_region_table_header rt;
+    vhdx_region_table_entry bat_rt;         /* region table for the BAT */
+    vhdx_region_table_entry metadata_rt;    /* region table for the metadata */
+
+    vhdx_metadata_table_header  metadata_hdr;
+    vhdx_metadata_entries metadata_entries;
+
+    vhdx_file_parameters params;
+    uint32_t block_size;
+    uint32_t block_size_bits;
+    uint32_t sectors_per_block;
+    uint32_t sectors_per_block_bits;
+
+    uint64_t virtual_disk_size;
+    uint32_t logical_sector_size;
+    uint32_t physical_sector_size;
+
+    uint64_t chunk_ratio;
+    uint32_t chunk_ratio_bits;
+    uint32_t logical_sector_size_bits;
+
+    uint32_t bat_entries;
+    vhdx_bat_entry *bat;
+    uint64_t bat_offset;
+
+    vhdx_parent_locator_header parent_header;
+    vhdx_parent_locator_entry *parent_entries;
+
+} BDRVVHDXState;
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset)
+{
+    uint32_t crc_new;
+    uint32_t crc_orig;
+    assert(buf != NULL);
+
+    if (crc_offset > 0) {
+        memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+        memset(buf+crc_offset, 0, sizeof(crc_orig));
+    }
+
+    crc_new = crc32c(crc, buf, size);
+    if (crc_offset > 0) {
+        memcpy(buf+crc_offset, &crc_orig, sizeof(crc_orig));
+    }
+
+    return crc_new;
+}
+
+/* Validates the checksum of the buffer, with an in-place CRC.
+ *
+ * Zero is substituted during crc calculation for the original crc field,
+ * and the crc field is restored afterwards.  But the buffer will be modifed
+ * during the calculation, so this may not be not suitable for multi-threaded
+ * use.
+ *
+ * crc_offset: byte offset in buf of the buffer crc
+ * buf: buffer pointer
+ * size: size of buffer (must be > crc_offset+4)
+ *
+ * returns true if checksum is valid, false otherwise
+ */
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset)
+{
+    uint32_t crc_orig;
+    uint32_t crc;
+
+    assert(buf != NULL);
+    assert(size > (crc_offset+4));
+
+    memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+    crc_orig = le32_to_cpu(crc_orig);
+
+    crc = vhdx_checksum_calc(0xffffffff, buf, size, crc_offset);
+
+    return crc == crc_orig;
+}
+
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *      - The header section is fixed size - 1 MB
+ *      - The header section is always the first "object"
+ *      - The first 64KB of the header is the File Identifier
+ *      - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *      - The following 512 bytes constitute a UTF-16 string identifiying the
+ *        software that created the file, and is optional and diagnostic only.
+ *
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+    if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
+        return 100;
+    }
+    return 0;
+}
+
+/* All VHDX structures on disk are little endian */
+static void vhdx_header_le_import(vhdx_header *h)
+{
+    assert(h != NULL);
+
+    le32_to_cpus(&h->signature);
+    le32_to_cpus(&h->checksum);
+    le64_to_cpus(&h->sequence_number);
+
+    leguid_to_cpus(&h->file_write_guid);
+    leguid_to_cpus(&h->data_write_guid);
+    leguid_to_cpus(&h->log_guid);
+
+    le16_to_cpus(&h->log_version);
+    le16_to_cpus(&h->version);
+    le32_to_cpus(&h->log_length);
+    le64_to_cpus(&h->log_offset);
+}
+

block/vhdx.c line 250

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;
+    uint8_t *buffer;

Makes sense to use uint8_t* (or possible rather void*) here, but it means that struct vhdx_header_padded is unused and could be dropped from the header file.

block/vhdx.c line 258

Quoted 7 lines
block/vhdx.c new file +7 -0
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;

The spec requires that the signature field of the File Type Identifier must be validated when loading a VHDX file. We only have the check in vhdx_probe(), but not in vhdx_open().

(This means that our struct vhdx_file_identifier is unused.)

Quoted 28 lines
block/vhdx.c new file +28 -0
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }

block/vhdx.c line 297

Quoted 10 lines
block/vhdx.c new file +10 -0
+
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
+        printf("NO VALID HEADER\n");
+        ret = -EINVAL;
+        goto fail;
+    }

This may be nit-picking, but according to the spec, I think a header with a sequence number of 0 can be valid (if the other header is invalid). This code would get 0 for both and error out.

What does a new VHDX file look like in practice? Sequence numbers 1 and 2?

Quoted 29 lines
block/vhdx.c new file +29 -0
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    vhdx_region_table_entry rt_entry;
+    int i;
+
+    /* We have to read the whole 64KB block, because the crc32 is over the
+     * whole block */
+    buffer = qemu_blockalign(bs, VHDX_HEADER_BLOCK_SIZE);
+
+    ret = bdrv_pread(bs->file, VHDX_REGION_TABLE_OFFSET, buffer,

block/vhdx.c line 336

Quoted 10 lines
block/vhdx.c new file +10 -0
+                    VHDX_HEADER_BLOCK_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    memcpy(&s->rt, buffer, sizeof(s->rt));
+    le32_to_cpus(&s->rt.signature);
+    le32_to_cpus(&s->rt.checksum);
+    le32_to_cpus(&s->rt.entry_count);
+    le32_to_cpus(&s->rt.reserved);
+    offset += sizeof(s->rt);

Maybe it would safer with respect to forgotting endianness conversion if we didn't copy and then convert fields, but only copy converted values. Like:

vhdx_region_table_header *rt = buffer;

s->rt = (vhdx_region_table_header) {
    .signature  = le32_to_cpu(rt->signature),
    .checksum   = le32_to_cpu(rt->checksum),
    ...
};

Matter of taste, I guess, so this is just a suggestion, not a request. You could do this is quite a few places, I think.

Quoted 16 lines
block/vhdx.c new file +16 -0
+
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);
+
+        leguid_to_cpus(&rt_entry.guid);
+        le64_to_cpus(&rt_entry.file_offset);
+        le32_to_cpus(&rt_entry.length);
+        le32_to_cpus(&rt_entry.data_bits);
+

block/vhdx.c line 362

Quoted 10 lines
block/vhdx.c new file +10 -0
+        /* see if we recognize the entry */
+        if (guid_eq(rt_entry.guid, bat_guid)) {
+            s->bat_rt = rt_entry;
+            continue;
+        }
+
+        if (guid_eq(rt_entry.guid, metadata_guid)) {
+            s->metadata_rt = rt_entry;
+            continue;
+        }

If the same regions occurs multiple times, the latest wins. Such images aren't valid, but what should we do with such cases? Is this good enough or should we detect it?

Quoted 139 lines
block/vhdx.c new file +139 -0
+        if (rt_entry.data_bits & VHDX_REGION_ENTRY_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto fail;
+        }
+    }
+    ret = 0;
+
+fail:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+
+/* Metadata initial parser
+ *
+ * This loads all the metadata entry fields.  This may cause additional
+ * fields to be processed (e.g. parent locator, etc..).
+ *
+ * There are 5 Metadata items that are always required:
+ *      - File Parameters (block size, has a parent)
+ *      - Virtual Disk Size (size, in bytes, of the virtual drive)
+ *      - Page 83 Data (scsi page 83 guid)
+ *      - Logical Sector Size (logical sector size in bytes, either 512 or
+ *                             4096.  We only support 512 currently)
+ *      - Physical Sector Size (512 or 4096)
+ *
+ * Also, if the File Parameters indicate this is a differencing file,
+ * we must also look for the Parent Locator metadata item.
+ */
+static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    int i = 0;
+    uint32_t block_size, sectors_per_block, logical_sector_size;
+    uint64_t chunk_ratio;
+    vhdx_metadata_table_entry md_entry;
+
+    buffer = qemu_blockalign(bs, VHDX_METADATA_TABLE_MAX_SIZE);
+
+    ret = bdrv_pread(bs->file, s->metadata_rt.file_offset, buffer,
+                     VHDX_METADATA_TABLE_MAX_SIZE);
+    if (ret < 0) {
+        goto exit;
+    }
+    memcpy(&s->metadata_hdr, buffer, sizeof(s->metadata_hdr));
+    offset += sizeof(s->metadata_hdr);
+
+    le64_to_cpus(&s->metadata_hdr.signature);
+    le16_to_cpus(&s->metadata_hdr.reserved);
+    le16_to_cpus(&s->metadata_hdr.entry_count);
+
+    if (s->metadata_hdr.signature != VHDX_METADATA_MAGIC) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    s->metadata_entries.present = 0;
+
+    if ((s->metadata_hdr.entry_count * sizeof(md_entry)) >
+        (VHDX_METADATA_TABLE_MAX_SIZE - offset)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    for (i = 0; i < s->metadata_hdr.entry_count; i++) {
+        memcpy(&md_entry, buffer+offset, sizeof(md_entry));
+        offset += sizeof(md_entry);
+
+        leguid_to_cpus(&md_entry.item_id);
+        le32_to_cpus(&md_entry.offset);
+        le32_to_cpus(&md_entry.length);
+        le32_to_cpus(&md_entry.data_bits);
+        le32_to_cpus(&md_entry.reserved2);
+
+        if (guid_eq(md_entry.item_id, file_param_guid)) {
+            s->metadata_entries.file_parameters_entry = md_entry;
+            s->metadata_entries.present |= META_FILE_PARAMETER_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, virtual_size_guid)) {
+            s->metadata_entries.virtual_disk_size_entry = md_entry;
+            s->metadata_entries.present |= META_VIRTUAL_DISK_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, page83_guid)) {
+            s->metadata_entries.page83_data_entry = md_entry;
+            s->metadata_entries.present |= META_PAGE_83_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, logical_sector_guid)) {
+            s->metadata_entries.logical_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_LOGICAL_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, phys_sector_guid)) {
+            s->metadata_entries.phys_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_PHYS_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, parent_locator_guid)) {
+            s->metadata_entries.parent_locator_entry = md_entry;
+            s->metadata_entries.present |= META_PARENT_LOCATOR_PRESENT;
+            continue;
+        }
+
+        if (md_entry.data_bits & VHDX_META_FLAGS_IS_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto exit;
+        }
+    }
+
+    if (s->metadata_entries.present != META_ALL_PRESENT) {
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.file_parameters_entry.offset
+                                         + s->metadata_rt.file_offset,
+                     &s->params,
+                     sizeof(s->params));
+
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le32_to_cpus(&s->params.block_size);

block/vhdx.c line 512

Quoted 10 lines
block/vhdx.c new file +10 -0
+    le32_to_cpus(&s->params.data_bits);
+
+
+    /* We now have the file parameters, so we can tell if this is a
+     * differencing file (i.e.. has_parent), is dynamic or fixed
+     * sized (leave_blocks_allocated), and the block size */
+
+    /* The parent locator required iff the file parameters has_parent set */
+    if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+        if (s->metadata_entries.present & ~META_PARENT_LOCATOR_PRESENT) {

Not sure what you're trying to achieve here. We get -ENOTSUP if any metadata entry except the parent locator is present, and -EINVAL if there is none?

Of course, this doesn't matter currently, because above we've verified that s->metadata_entries.present == META_ALL_PRESENT, so we'll always get the -ENOTSUP case.

Quoted 85 lines
block/vhdx.c new file +85 -0
+            /* TODO: parse  parent locator fields */
+            ret = -ENOTSUP; /* temp, until differencing files are supported */
+            goto exit;
+        } else {
+            /* if has_parent is set, but there is not parent locator present,
+             * then that is an invalid combination */
+            ret = -EINVAL;
+            goto exit;
+        }
+    }
+
+    /* determine virtual disk size, logical sector size,
+     * and phys sector size */
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.virtual_disk_size_entry.offset
+                                           + s->metadata_rt.file_offset,
+                     &s->virtual_disk_size,
+                     sizeof(uint64_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.logical_sector_size_entry.offset
+                                             + s->metadata_rt.file_offset,
+                     &s->logical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.phys_sector_size_entry.offset
+                                          + s->metadata_rt.file_offset,
+                     &s->physical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le64_to_cpus(&s->virtual_disk_size);
+    le32_to_cpus(&s->logical_sector_size);
+    le32_to_cpus(&s->physical_sector_size);
+
+    if (s->logical_sector_size == 0 || s->params.block_size == 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    /* both block_size and sector_size are guaranteed powers of 2 */
+    s->sectors_per_block = s->params.block_size / s->logical_sector_size;
+    s->chunk_ratio = (VHDX_MAX_SECTORS_PER_BLOCK) *
+                     (uint64_t)s->logical_sector_size /
+                     (uint64_t)s->params.block_size;
+
+    /* These values are ones we will want to use for division / multiplication
+     * later on, and they are all guaranteed (per the spec) to be powers of 2,
+     * so we can take advantage of that for shift operations during
+     * reads/writes */
+    logical_sector_size = s->logical_sector_size;
+    if (logical_sector_size & (logical_sector_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    sectors_per_block = s->sectors_per_block;
+    if (sectors_per_block & (sectors_per_block - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    chunk_ratio = s->chunk_ratio;
+    if (chunk_ratio & (chunk_ratio - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    block_size = s->params.block_size;
+    if (block_size & (block_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;

block/vhdx.c line 607

Quoted 10 lines
block/vhdx.c new file +10 -0
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }
+
+    if (s->logical_sector_size != BDRV_SECTOR_SIZE) {
+        printf("VHDX error - QEMU only supports 512 byte sector sizes\n");

This is not true. It depends on the device model, and in particular the qdev properties of it. The block layer doesn't have access to them, so checking that it matches the image file isn't trivial. It would have to be the device model that checks that the BlockDriverState is valid for the device it emulates.

I suggest dropping this check; otherwise make it (q)error_report at least.

block/vhdx.c line 617

Quoted 10 lines
block/vhdx.c new file +10 -0
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = 0;
+
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}

Kevin

author
#16 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Quoted 291 lines · Kevin Wolf

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 11 lines

This is the initial block driver framework for VHDX image support ( i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

Quoted 5 lines

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

Okay, now I'm starting to actually compare the spec to your implementation. Comments inline.

Quoted 270 lines
 block/Makefile.objs |   1 +
 block/vhdx.c        | 769 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  25 ++
 3 files changed, 795 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +246 -0
new file mode 100644
index 0000000..b0ea2ba
@@ -0,0 +1,769 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block/block_int.h"
+#include "qemu/module.h"
+#include "qemu/crc32c.h"
+#include "block/vhdx.h"
+
+
+/* Several metadata and region table data entries are identified by
+ * guids in  a MS-specific GUID format. */
+
+
+/* ------- Known Region Table GUIDs ---------------------- */
+static const ms_guid bat_guid =      { .data1 = 0x2dc27766,
+                                       .data2 = 0xf623,
+                                       .data3 = 0x4200,
+                                       .data4 = { 0x9d, 0x64, 0x11, 0x5e,
+                                                  0x9b, 0xfd, 0x4a, 0x08} };
+
+static const ms_guid metadata_guid = { .data1 = 0x8b7ca206,
+                                       .data2 = 0x4790,
+                                       .data3 = 0x4b9a,
+                                       .data4 = { 0xb8, 0xfe, 0x57, 0x5f,
+                                                  0x05, 0x0f, 0x88, 0x6e} };
+
+
+
+/* ------- Known Metadata Entry GUIDs ---------------------- */
+static const ms_guid file_param_guid =   { .data1 = 0xcaa16737,
+                                           .data2 = 0xfa36,
+                                           .data3 = 0x4d43,
+                                           .data4 = { 0xb3, 0xb6, 0x33, 0xf0,
+                                                      0xaa, 0x44, 0xe7, 0x6b} };
+
+static const ms_guid virtual_size_guid = { .data1 = 0x2FA54224,
+                                           .data2 = 0xcd1b,
+                                           .data3 = 0x4876,
+                                           .data4 = { 0xb2, 0x11, 0x5d, 0xbe,
+                                                      0xd8, 0x3b, 0xf4, 0xb8} };
+
+static const ms_guid page83_guid =       { .data1 = 0xbeca12ab,
+                                           .data2 = 0xb2e6,
+                                           .data3 = 0x4523,
+                                           .data4 = { 0x93, 0xef, 0xc3, 0x09,
+                                                      0xe0, 0x00, 0xc7, 0x46} };
+
+
+static const ms_guid phys_sector_guid =  { .data1 = 0xcda348c7,
+                                           .data2 = 0x445d,
+                                           .data3 = 0x4471,
+                                           .data4 = { 0x9c, 0xc9, 0xe9, 0x88,
+                                                      0x52, 0x51, 0xc5, 0x56} };
+
+static const ms_guid parent_locator_guid = { .data1 = 0xa8d35f2d,
+                                             .data2 = 0xb30b,
+                                             .data3 = 0x454d,
+                                             .data4 = { 0xab, 0xf7, 0xd3,
+                                                        0xd8, 0x48, 0x34,
+                                                        0xab, 0x0c} };
+
+static const ms_guid logical_sector_guid = { .data1 = 0x8141bf1d,
+                                             .data2 = 0xa96f,
+                                             .data3 = 0x4709,
+                                             .data4 = { 0xba, 0x47, 0xf2,
+                                                        0x33, 0xa8, 0xfa,
+                                                        0xab, 0x5f} };
+
+/* Each parent type must have a valid GUID; this is for parent images
+ * of type 'VHDX'.  If we were to allow e.g. a QCOW2 parent, we would
+ * need to make up our own QCOW2 GUID type */
+static const ms_guid parent_vhdx_guid = { .data1 = 0xb04aefb7,
+                                          .data2 = 0xd19e,
+                                          .data3 = 0x4a81,
+                                          .data4 = { 0xb7, 0x89, 0x25, 0xb8,
+                                                     0xe9, 0x44, 0x59, 0x13} };
+
+
+#define META_FILE_PARAMETER_PRESENT      0x01
+#define META_VIRTUAL_DISK_SIZE_PRESENT   0x02
+#define META_PAGE_83_PRESENT             0x04
+#define META_LOGICAL_SECTOR_SIZE_PRESENT 0x08
+#define META_PHYS_SECTOR_SIZE_PRESENT    0x10
+#define META_PARENT_LOCATOR_PRESENT      0x20
+
+#define META_ALL_PRESENT    \
+    (META_FILE_PARAMETER_PRESENT | META_VIRTUAL_DISK_SIZE_PRESENT | \
+     META_PAGE_83_PRESENT | META_LOGICAL_SECTOR_SIZE_PRESENT | \
+     META_PHYS_SECTOR_SIZE_PRESENT)
+
+typedef struct vhdx_metadata_entries {
+    vhdx_metadata_table_entry file_parameters_entry;
+    vhdx_metadata_table_entry virtual_disk_size_entry;
+    vhdx_metadata_table_entry page83_data_entry;
+    vhdx_metadata_table_entry logical_sector_size_entry;
+    vhdx_metadata_table_entry phys_sector_size_entry;
+    vhdx_metadata_table_entry parent_locator_entry;
+    uint16_t present;
+} vhdx_metadata_entries;
+
+
+typedef struct BDRVVHDXState {
+    CoMutex lock;
+
+    int curr_header;
+    vhdx_header *headers[2];
+
+    vhdx_region_table_header rt;
+    vhdx_region_table_entry bat_rt;         /* region table for the BAT */
+    vhdx_region_table_entry metadata_rt;    /* region table for the metadata */
+
+    vhdx_metadata_table_header  metadata_hdr;
+    vhdx_metadata_entries metadata_entries;
+
+    vhdx_file_parameters params;
+    uint32_t block_size;
+    uint32_t block_size_bits;
+    uint32_t sectors_per_block;
+    uint32_t sectors_per_block_bits;
+
+    uint64_t virtual_disk_size;
+    uint32_t logical_sector_size;
+    uint32_t physical_sector_size;
+
+    uint64_t chunk_ratio;
+    uint32_t chunk_ratio_bits;
+    uint32_t logical_sector_size_bits;
+
+    uint32_t bat_entries;
+    vhdx_bat_entry *bat;
+    uint64_t bat_offset;
+
+    vhdx_parent_locator_header parent_header;
+    vhdx_parent_locator_entry *parent_entries;
+
+} BDRVVHDXState;
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset)
+{
+    uint32_t crc_new;
+    uint32_t crc_orig;
+    assert(buf != NULL);
+
+    if (crc_offset > 0) {
+        memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+        memset(buf+crc_offset, 0, sizeof(crc_orig));
+    }
+
+    crc_new = crc32c(crc, buf, size);
+    if (crc_offset > 0) {
+        memcpy(buf+crc_offset, &crc_orig, sizeof(crc_orig));
+    }
+
+    return crc_new;
+}
+
+/* Validates the checksum of the buffer, with an in-place CRC.
+ *
+ * Zero is substituted during crc calculation for the original crc field,
+ * and the crc field is restored afterwards.  But the buffer will be modifed
+ * during the calculation, so this may not be not suitable for multi-threaded
+ * use.
+ *
+ * crc_offset: byte offset in buf of the buffer crc
+ * buf: buffer pointer
+ * size: size of buffer (must be > crc_offset+4)
+ *
+ * returns true if checksum is valid, false otherwise
+ */
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset)
+{
+    uint32_t crc_orig;
+    uint32_t crc;
+
+    assert(buf != NULL);
+    assert(size > (crc_offset+4));
+
+    memcpy(&crc_orig, buf+crc_offset, sizeof(crc_orig));
+    crc_orig = le32_to_cpu(crc_orig);
+
+    crc = vhdx_checksum_calc(0xffffffff, buf, size, crc_offset);
+
+    return crc == crc_orig;
+}
+
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *      - The header section is fixed size - 1 MB
+ *      - The header section is always the first "object"
+ *      - The first 64KB of the header is the File Identifier
+ *      - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *      - The following 512 bytes constitute a UTF-16 string identifiying the
+ *        software that created the file, and is optional and diagnostic only.
+ *
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+    if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
+        return 100;
+    }
+    return 0;
+}
+
+/* All VHDX structures on disk are little endian */
+static void vhdx_header_le_import(vhdx_header *h)
+{
+    assert(h != NULL);
+
+    le32_to_cpus(&h->signature);
+    le32_to_cpus(&h->checksum);
+    le64_to_cpus(&h->sequence_number);
+
+    leguid_to_cpus(&h->file_write_guid);
+    leguid_to_cpus(&h->data_write_guid);
+    leguid_to_cpus(&h->log_guid);
+
+    le16_to_cpus(&h->log_version);
+    le16_to_cpus(&h->version);
+    le32_to_cpus(&h->log_length);
+    le64_to_cpus(&h->log_offset);
+}
+
+
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
Quoted 9 lines

block/vhdx.c line 250

Quoted 4 lines
block/vhdx.c new file +4 -0
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;
+    uint8_t *buffer;

Makes sense to use uint8_t* (or possible rather void*) here, but it means that struct vhdx_header_padded is unused and could be dropped from the header file.

OK, I'll drop it from the header file (or maybe convert the meaning of it into a comment in the header, to preserve the info).

Quoted 12 lines

block/vhdx.c line 258

Quoted 7 lines
block/vhdx.c new file +7 -0
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;

The spec requires that the signature field of the File Type Identifier must be validated when loading a VHDX file. We only have the check in vhdx_probe(), but not in vhdx_open().

Yes, true - I'll add that in first.

Quoted 36 lines

(This means that our struct vhdx_file_identifier is unused.)

Quoted 34 lines
block/vhdx.c new file +34 -0
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }
+
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
Quoted 9 lines

block/vhdx.c line 297

Quoted 4 lines
block/vhdx.c new file +4 -0
+        printf("NO VALID HEADER\n");
+        ret = -EINVAL;
+        goto fail;
+    }

This may be nit-picking, but according to the spec, I think a header with a sequence number of 0 can be valid (if the other header is invalid). This code would get 0 for both and error out.

I think you are right. The other header would need to have either an invalid signature or checksum, but if that was the case technically 0 could indicate the active header.

Quoted 3 lines

What does a new VHDX file look like in practice? Sequence numbers 1 and 2?

I'll check that out - my Hyper-V test server is at home, and I am working at a co-working place today. I'll check it this evening - it'll have to be on a freshly created image before even opening it up with Hyper-V.

Quoted 76 lines
Quoted 29 lines
block/vhdx.c new file +29 -0
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    vhdx_region_table_entry rt_entry;
+    int i;
+
+    /* We have to read the whole 64KB block, because the crc32 is over the
+     * whole block */
+    buffer = qemu_blockalign(bs, VHDX_HEADER_BLOCK_SIZE);
+
+    ret = bdrv_pread(bs->file, VHDX_REGION_TABLE_OFFSET, buffer,

block/vhdx.c line 336

Quoted 10 lines
block/vhdx.c new file +10 -0
+                    VHDX_HEADER_BLOCK_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    memcpy(&s->rt, buffer, sizeof(s->rt));
+    le32_to_cpus(&s->rt.signature);
+    le32_to_cpus(&s->rt.checksum);
+    le32_to_cpus(&s->rt.entry_count);
+    le32_to_cpus(&s->rt.reserved);
+    offset += sizeof(s->rt);

Maybe it would safer with respect to forgotting endianness conversion if we didn't copy and then convert fields, but only copy converted values. Like:

vhdx_region_table_header *rt = buffer;

s->rt = (vhdx_region_table_header) {
    .signature  = le32_to_cpu(rt->signature),
    .checksum   = le32_to_cpu(rt->checksum),
    ...
};

Matter of taste, I guess, so this is just a suggestion, not a request. You could do this is quite a few places, I think.

Quoted 22 lines
block/vhdx.c new file +22 -0
+
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);
+
+        leguid_to_cpus(&rt_entry.guid);
+        le64_to_cpus(&rt_entry.file_offset);
+        le32_to_cpus(&rt_entry.length);
+        le32_to_cpus(&rt_entry.data_bits);
+
+        /* see if we recognize the entry */
+        if (guid_eq(rt_entry.guid, bat_guid)) {
+            s->bat_rt = rt_entry;
+            continue;
+        }
+
Quoted 8 lines

block/vhdx.c line 362

Quoted 4 lines
block/vhdx.c new file +4 -0
+        if (guid_eq(rt_entry.guid, metadata_guid)) {
+            s->metadata_rt = rt_entry;
+            continue;
+        }

If the same regions occurs multiple times, the latest wins. Such images aren't valid, but what should we do with such cases? Is this good enough or should we detect it?

I think such images are more undefined rather than explicitly invalid. I don't think the spec touches on the idea of multiple regions of the same type. That said, I don't what to make of an image file with multiple BAT regions, for instance - I think error is the only sane option.

Maybe keep a list of all entries, and if there are duplicates error out with -EINVAL?

Quoted 146 lines
Quoted 146 lines
block/vhdx.c new file +145 -0
+        if (rt_entry.data_bits & VHDX_REGION_ENTRY_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto fail;
+        }
+    }
+    ret = 0;
+
+fail:
+    qemu_vfree(buffer);
+    return ret;
+}
+
+
+
+/* Metadata initial parser
+ *
+ * This loads all the metadata entry fields.  This may cause additional
+ * fields to be processed (e.g. parent locator, etc..).
+ *
+ * There are 5 Metadata items that are always required:
+ *      - File Parameters (block size, has a parent)
+ *      - Virtual Disk Size (size, in bytes, of the virtual drive)
+ *      - Page 83 Data (scsi page 83 guid)
+ *      - Logical Sector Size (logical sector size in bytes, either 512 or
+ *                             4096.  We only support 512 currently)
+ *      - Physical Sector Size (512 or 4096)
+ *
+ * Also, if the File Parameters indicate this is a differencing file,
+ * we must also look for the Parent Locator metadata item.
+ */
+static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    uint8_t *buffer;
+    int offset = 0;
+    int i = 0;
+    uint32_t block_size, sectors_per_block, logical_sector_size;
+    uint64_t chunk_ratio;
+    vhdx_metadata_table_entry md_entry;
+
+    buffer = qemu_blockalign(bs, VHDX_METADATA_TABLE_MAX_SIZE);
+
+    ret = bdrv_pread(bs->file, s->metadata_rt.file_offset, buffer,
+                     VHDX_METADATA_TABLE_MAX_SIZE);
+    if (ret < 0) {
+        goto exit;
+    }
+    memcpy(&s->metadata_hdr, buffer, sizeof(s->metadata_hdr));
+    offset += sizeof(s->metadata_hdr);
+
+    le64_to_cpus(&s->metadata_hdr.signature);
+    le16_to_cpus(&s->metadata_hdr.reserved);
+    le16_to_cpus(&s->metadata_hdr.entry_count);
+
+    if (s->metadata_hdr.signature != VHDX_METADATA_MAGIC) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    s->metadata_entries.present = 0;
+
+    if ((s->metadata_hdr.entry_count * sizeof(md_entry)) >
+        (VHDX_METADATA_TABLE_MAX_SIZE - offset)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    for (i = 0; i < s->metadata_hdr.entry_count; i++) {
+        memcpy(&md_entry, buffer+offset, sizeof(md_entry));
+        offset += sizeof(md_entry);
+
+        leguid_to_cpus(&md_entry.item_id);
+        le32_to_cpus(&md_entry.offset);
+        le32_to_cpus(&md_entry.length);
+        le32_to_cpus(&md_entry.data_bits);
+        le32_to_cpus(&md_entry.reserved2);
+
+        if (guid_eq(md_entry.item_id, file_param_guid)) {
+            s->metadata_entries.file_parameters_entry = md_entry;
+            s->metadata_entries.present |= META_FILE_PARAMETER_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, virtual_size_guid)) {
+            s->metadata_entries.virtual_disk_size_entry = md_entry;
+            s->metadata_entries.present |= META_VIRTUAL_DISK_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, page83_guid)) {
+            s->metadata_entries.page83_data_entry = md_entry;
+            s->metadata_entries.present |= META_PAGE_83_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, logical_sector_guid)) {
+            s->metadata_entries.logical_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_LOGICAL_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, phys_sector_guid)) {
+            s->metadata_entries.phys_sector_size_entry = md_entry;
+            s->metadata_entries.present |= META_PHYS_SECTOR_SIZE_PRESENT;
+            continue;
+        }
+
+        if (guid_eq(md_entry.item_id, parent_locator_guid)) {
+            s->metadata_entries.parent_locator_entry = md_entry;
+            s->metadata_entries.present |= META_PARENT_LOCATOR_PRESENT;
+            continue;
+        }
+
+        if (md_entry.data_bits & VHDX_META_FLAGS_IS_REQUIRED) {
+            /* cannot read vhdx file - required region table entry that
+             * we do not understand.  per spec, we must fail to open */
+            ret = -ENOTSUP;
+            goto exit;
+        }
+    }
+
+    if (s->metadata_entries.present != META_ALL_PRESENT) {
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.file_parameters_entry.offset
+                                         + s->metadata_rt.file_offset,
+                     &s->params,
+                     sizeof(s->params));
+
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le32_to_cpus(&s->params.block_size);
+    le32_to_cpus(&s->params.data_bits);
+
+
+    /* We now have the file parameters, so we can tell if this is a
+     * differencing file (i.e.. has_parent), is dynamic or fixed
+     * sized (leave_blocks_allocated), and the block size */
Quoted 9 lines

block/vhdx.c line 512

Quoted 4 lines
block/vhdx.c new file +4 -0
+
+    /* The parent locator required iff the file parameters has_parent set */
+    if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+        if (s->metadata_entries.present & ~META_PARENT_LOCATOR_PRESENT) {

Not sure what you're trying to achieve here. We get -ENOTSUP if any metadata entry except the parent locator is present, and -EINVAL if there is none?

That is a mistake - that check should be sans tilde:
if (s->metadata_entries.present & META_PARENT_LOCATOR_PRESENT)

There are currently 2 failure cases (while parent locators are unsupported):

1.) There is a parent present, and a parent locator field.
- This is proper per the spec. Normally, we would parse the parent locator field. But this is not supported, so we return -ENOTSUP.

2.) There is a parent present, but not parent locator field.
- This is invalid, because if the parent is present the parent locator field per spec is a required field. So, return -EINVAL to indicate invalid field.

Quoted 4 lines

Of course, this doesn't matter currently, because above we've verified that s->metadata_entries.present == META_ALL_PRESENT, so we'll always get the -ENOTSUP case.

Once we support parent locator fields, that check should be masked against META_ALL_PRESENT, so that we are only checking for the base required fields. It wouldn't hurt to got ahead and do it now.

Quoted 102 lines
Quoted 85 lines
block/vhdx.c new file +85 -0
+            /* TODO: parse  parent locator fields */
+            ret = -ENOTSUP; /* temp, until differencing files are supported */
+            goto exit;
+        } else {
+            /* if has_parent is set, but there is not parent locator present,
+             * then that is an invalid combination */
+            ret = -EINVAL;
+            goto exit;
+        }
+    }
+
+    /* determine virtual disk size, logical sector size,
+     * and phys sector size */
+
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.virtual_disk_size_entry.offset
+                                           + s->metadata_rt.file_offset,
+                     &s->virtual_disk_size,
+                     sizeof(uint64_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.logical_sector_size_entry.offset
+                                             + s->metadata_rt.file_offset,
+                     &s->logical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+    ret = bdrv_pread(bs->file,
+                     s->metadata_entries.phys_sector_size_entry.offset
+                                          + s->metadata_rt.file_offset,
+                     &s->physical_sector_size,
+                     sizeof(uint32_t));
+    if (ret < 0) {
+        goto exit;
+    }
+
+    le64_to_cpus(&s->virtual_disk_size);
+    le32_to_cpus(&s->logical_sector_size);
+    le32_to_cpus(&s->physical_sector_size);
+
+    if (s->logical_sector_size == 0 || s->params.block_size == 0) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    /* both block_size and sector_size are guaranteed powers of 2 */
+    s->sectors_per_block = s->params.block_size / s->logical_sector_size;
+    s->chunk_ratio = (VHDX_MAX_SECTORS_PER_BLOCK) *
+                     (uint64_t)s->logical_sector_size /
+                     (uint64_t)s->params.block_size;
+
+    /* These values are ones we will want to use for division / multiplication
+     * later on, and they are all guaranteed (per the spec) to be powers of 2,
+     * so we can take advantage of that for shift operations during
+     * reads/writes */
+    logical_sector_size = s->logical_sector_size;
+    if (logical_sector_size & (logical_sector_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    sectors_per_block = s->sectors_per_block;
+    if (sectors_per_block & (sectors_per_block - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    chunk_ratio = s->chunk_ratio;
+    if (chunk_ratio & (chunk_ratio - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+    block_size = s->params.block_size;
+    if (block_size & (block_size - 1)) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    while (logical_sector_size >>= 1) {
+        s->logical_sector_size_bits++;
+    }
+    while (sectors_per_block >>= 1) {
+        s->sectors_per_block_bits++;

block/vhdx.c line 607

Quoted 10 lines
block/vhdx.c new file +10 -0
+    }
+    while (chunk_ratio >>= 1) {
+        s->chunk_ratio_bits++;
+    }
+    while (block_size >>= 1) {
+        s->block_size_bits++;
+    }
+
+    if (s->logical_sector_size != BDRV_SECTOR_SIZE) {
+        printf("VHDX error - QEMU only supports 512 byte sector sizes\n");

This is not true. It depends on the device model, and in particular the qdev properties of it. The block layer doesn't have access to them, so checking that it matches the image file isn't trivial. It would have to be the device model that checks that the BlockDriverState is valid for the device it emulates.

Quoted 3 lines

I suggest dropping this check; otherwise make it (q)error_report at least.

OK.

Quoted 12 lines

block/vhdx.c line 617

Quoted 10 lines
block/vhdx.c new file +10 -0
+        ret = -ENOTSUP;
+        goto exit;
+    }
+
+    ret = 0;
+
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}

Kevin

#17 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Am 25.04.2013 um 17:03 hat Jeff Cody geschrieben:

Quoted 32 lines · Kevin Wolf
Quoted 22 lines

Am 23.04.2013 um 16:24 hat Jeff Cody geschrieben:

Quoted 21 lines
block/vhdx.c new file +21 -0
+    if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
+        s->rt.signature != VHDX_RT_MAGIC) {
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    for (i = 0; i < s->rt.entry_count; i++) {
+        memcpy(&rt_entry, buffer+offset, sizeof(rt_entry));
+        offset += sizeof(rt_entry);
+
+        leguid_to_cpus(&rt_entry.guid);
+        le64_to_cpus(&rt_entry.file_offset);
+        le32_to_cpus(&rt_entry.length);
+        le32_to_cpus(&rt_entry.data_bits);
+
+        /* see if we recognize the entry */
+        if (guid_eq(rt_entry.guid, bat_guid)) {
+            s->bat_rt = rt_entry;
+            continue;
+        }
+
Quoted 8 lines

block/vhdx.c line 362

Quoted 4 lines
block/vhdx.c new file +4 -0
+        if (guid_eq(rt_entry.guid, metadata_guid)) {
+            s->metadata_rt = rt_entry;
+            continue;
+        }

If the same regions occurs multiple times, the latest wins. Such images aren't valid, but what should we do with such cases? Is this good enough or should we detect it?

Quoted 3 lines

I think such images are more undefined rather than explicitly invalid. I don't think the spec touches on the idea of multiple regions of the same type.

It says "The Guid field specifies a 128-bit identifier for the object and must be unique within the table.", so it's explicitly forbidden to have two regions of the same type.

Quoted 6 lines

That said, I don't what to make of an image file with multiple BAT regions, for instance - I think error is the only sane option.

Maybe keep a list of all entries, and if there are duplicates error out with -EINVAL?

We could do that if we really care. Or just check for the ones that we recognise that they aren't set yet when we assign them.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
block/vhdx.c new file +7 -0
+    /* We now have the file parameters, so we can tell if this is a
+     * differencing file (i.e.. has_parent), is dynamic or fixed
+     * sized (leave_blocks_allocated), and the block size */
+
+    /* The parent locator required iff the file parameters has_parent set */
+    if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+        if (s->metadata_entries.present & ~META_PARENT_LOCATOR_PRESENT) {
Quoted 7 lines
Quoted 4 lines

Not sure what you're trying to achieve here. We get -ENOTSUP if any metadata entry except the parent locator is present, and -EINVAL if there is none?

That is a mistake - that check should be sans tilde:
if (s->metadata_entries.present & META_PARENT_LOCATOR_PRESENT)

Right, that makes more sense.

Kevin

#18 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody

block/vhdx.h typedef struct QEMU_PACKED vhdx_parent_locator_entry { · line 346

Quoted 6 lines
block/vhdx.h +6 -0
+static void cpu_to_leguids(ms_guid *guid)
+{
+    cpu_to_le32s(&guid->data1);
+    cpu_to_le16s(&guid->data2);
+    cpu_to_le16s(&guid->data3);
+}

This one seems used in 5/5 only, so this patch fails compiling (defined but not used with -Werror=unused-function). Maybe move to 5/5?

Signature · 1 line
Fam
author
#19 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Quoted 10 lines · Fam Zheng

block/vhdx.h typedef struct QEMU_PACKED vhdx_parent_locator_entry { · line 346

Quoted 6 lines
block/vhdx.h +6 -0
+static void cpu_to_leguids(ms_guid *guid)
+{
+    cpu_to_le32s(&guid->data1);
+    cpu_to_le16s(&guid->data2);
+    cpu_to_le16s(&guid->data3);
+}

This one seems used in 5/5 only, so this patch fails compiling (defined but not used with -Werror=unused-function). Maybe move to 5/5?

Thanks. Yes, that should be in patch 5 (git rebase user error there), which means this will go away completely for v3.

#20 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody
Quoted 8 lines · Jeff Cody
block/vhdx.c new file +8 -0
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;

block/vhdx.c line 259

Quoted 10 lines
block/vhdx.c new file +10 -0
+    uint8_t *buffer;
+
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;
+

Logic for header1 and header2 are completely the same, IMO it might be better to avoid code with a loop.

Quoted 5 lines
block/vhdx.c new file +5 -0
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {

block/vhdx.c line 274

Quoted 10 lines
block/vhdx.c new file +10 -0
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }

Do we need to check the version here? As the specification page 15 says:

The Version field specifies the version of the VHDX format used within the VHDX file. This field must be set to 1. If it is not, a parser must not attempt to parse the file using the details from this format specification.

block/vhdx.c line 287

Quoted 13 lines
block/vhdx.c new file +13 -0
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }

Same as above.

Quoted 24 lines
block/vhdx.c new file +24 -0
+
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
+        printf("NO VALID HEADER\n");
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
Signature · 1 line
Fam
author
#21 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block: initial VHDX driver support framework - supports open and probe

Quoted 37 lines · Fam Zheng
Quoted 8 lines · Jeff Cody
block/vhdx.c new file +8 -0
+/* opens the specified header block from the VHDX file header section */
+static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)
+{
+    int ret = 0;
+    vhdx_header *header1;
+    vhdx_header *header2;
+    uint64_t h1_seq = 0;
+    uint64_t h2_seq = 0;

block/vhdx.c line 259

Quoted 10 lines
block/vhdx.c new file +10 -0
+    uint8_t *buffer;
+
+    header1 = qemu_blockalign(bs, sizeof(vhdx_header));
+    header2 = qemu_blockalign(bs, sizeof(vhdx_header));
+
+    buffer = qemu_blockalign(bs, VHDX_HEADER_SIZE);
+
+    s->headers[0] = header1;
+    s->headers[1] = header2;
+

Logic for header1 and header2 are completely the same, IMO it might be better to avoid code with a loop.

Quoted 5 lines
block/vhdx.c new file +5 -0
+    /* We have to read the whole VHDX_HEADER_SIZE instead of
+     * sizeof(vhdx_header), because the checksum is over the whole
+     * region */
+    ret = bdrv_pread(bs->file, VHDX_HEADER1_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {

block/vhdx.c line 274

Quoted 10 lines
block/vhdx.c new file +10 -0
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header1, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header1);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header1->signature == VHDX_HDR_MAGIC) {
+        h1_seq = header1->sequence_number;
+    }
Quoted 7 lines

Do we need to check the version here? As the specification page 15 says:

The Version field specifies the version of the VHDX format used within the VHDX file. This field must be set to 1. If it is not, a parser must not attempt to parse the file using the details from this format specification.

Yes, you are correct - I will add that in for v3.

Quoted 41 lines

block/vhdx.c line 287

Quoted 13 lines
block/vhdx.c new file +13 -0
+
+    ret = bdrv_pread(bs->file, VHDX_HEADER2_OFFSET, buffer, VHDX_HEADER_SIZE);
+    if (ret < 0) {
+        goto fail;
+    }
+    /* copy over just the relevant portion that we need */
+    memcpy(header2, buffer, sizeof(vhdx_header));
+    vhdx_header_le_import(header2);
+
+    if (vhdx_checksum_is_valid(buffer, VHDX_HEADER_SIZE, 4) &&
+        header2->signature == VHDX_HDR_MAGIC) {
+        h2_seq = header2->sequence_number;
+    }

Same as above.

Quoted 24 lines
block/vhdx.c new file +24 -0
+
+    if (h1_seq > h2_seq) {
+        s->curr_header = 0;
+    } else if (h2_seq > h1_seq) {
+        s->curr_header = 1;
+    } else {
+        printf("NO VALID HEADER\n");
+        ret = -EINVAL;
+        goto fail;
+    }
+
+    ret = 0;
+
+    goto exit;
+
+fail:
+    qemu_vfree(header1);
+    qemu_vfree(header2);
+    s->headers[0] = NULL;
+    s->headers[1] = NULL;
+exit:
+    qemu_vfree(buffer);
+    return ret;
+}
Signature · 1 line
Fam
author
#22 lore raw

PATCH v2 4/5 block: add read-only support to VHDX image format.

In reply to Jeff Cody

This adds in read-only support to the VHDX image format. This supports reads for fixed-size, and dynamic sized VHDX images.

Differencing files are still unsupported.

The image must be opened without BDRV_O_RDWR set, because we do not yet update the headers. I.e., pass 'readonly=on' in the drive image options from the QEMU commandline.

---
 block/vhdx.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 121 insertions(+), 2 deletions(-)
block/vhdx.c +121 -2
index b0ea2ba..ca4a298 100644
@@ -114,6 +114,17 @@ typedef struct vhdx_metadata_entries {
 } vhdx_metadata_entries;
 
 
+typedef struct vhdx_sector_info {
+    uint32_t bat_idx;       /* BAT entry index */
+    uint32_t sectors_avail; /* sectors available in payload block */
+    uint32_t bytes_left;    /* bytes left in the block after data to r/w */
+    uint32_t bytes_avail;   /* bytes available in payload block */
+    uint64_t file_offset;   /* absolute offset in bytes, in file */
+    uint64_t block_offset;  /* block offset, in bytes */
+} vhdx_sector_info;
+
+
+
 typedef struct BDRVVHDXState {
     CoMutex lock;
 
@@ -709,7 +720,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags)
         goto fail;
     }
 
-    /* TODO: differencing files, read, write */
+    /* TODO: differencing files, write */
 
     return 0;
 fail:
@@ -724,10 +735,118 @@ static int vhdx_reopen_prepare(BDRVReopenState *state,
 }
 
 
+/*
+ * Perform sector to block offset translations, to get various
+ * sector and file offsets into the image.  See vhdx_sector_info
+ */
+static void vhdx_block_translate(BDRVVHDXState *s, int64_t sector_num,
+                                 int nb_sectors, vhdx_sector_info *sinfo)
+{
+    uint32_t block_offset;
+
+    sinfo->bat_idx = sector_num >> s->sectors_per_block_bits;
+    /* effectively a modulo - this gives us the offset into the block
+     * (in sector sizes) for our sector number */
+    block_offset = sector_num - (sinfo->bat_idx << s->sectors_per_block_bits);
+    /* the chunk ratio gives us the interleaving of the sector
+     * bitmaps, so we need to advance our page block index by the
+     * sector bitmaps entry number */
+    sinfo->bat_idx += sinfo->bat_idx >> s->chunk_ratio_bits;
+
+    /* the number of sectors we can read/write in this cycle */
+    sinfo->sectors_avail = s->sectors_per_block - block_offset;
+
+    sinfo->bytes_left = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    if (sinfo->sectors_avail > nb_sectors) {
+        sinfo->sectors_avail = nb_sectors;
+    }
+
+    sinfo->bytes_avail = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    sinfo->file_offset = s->bat[sinfo->bat_idx] >> VHDX_BAT_FILE_OFF_BITS;
+
+    sinfo->block_offset = block_offset << s->logical_sector_size_bits;
+
+    /* The file offset must be past the header section, so must be > 0 */
+    if (sinfo->file_offset == 0) {
+        return;
+    }
+
+    /* block offset is the offset in vhdx logical sectors, in
+     * the payload data block. Convert that to a byte offset
+     * in the block, and add in the payload data block offset
+     * in the file, in bytes, to get the final read address */
+
+    sinfo->file_offset <<= 20;  /* now in bytes, rather than 1MB units */
+    sinfo->file_offset += sinfo->block_offset;
+}
+
+
+
 static coroutine_fn int vhdx_co_readv(BlockDriverState *bs, int64_t sector_num,
                                       int nb_sectors, QEMUIOVector *qiov)
 {
-    return -ENOTSUP;
+    BDRVVHDXState *s = bs->opaque;
+    int ret = 0;
+    vhdx_sector_info sinfo;
+    uint64_t bytes_done = 0;
+    QEMUIOVector hd_qiov;
+
+    qemu_iovec_init(&hd_qiov, qiov->niov);
+
+    qemu_co_mutex_lock(&s->lock);
+
+    while (nb_sectors > 0) {
+        /* We are a differencing file, so we need to inspect the sector bitmap
+         * to see if we have the data or not */
+        if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+            /* not supported yet */
+            ret = -ENOTSUP;
+            goto exit;
+        } else {
+            vhdx_block_translate(s, sector_num, nb_sectors, &sinfo);
+
+            qemu_iovec_reset(&hd_qiov);
+            qemu_iovec_concat(&hd_qiov, qiov,  bytes_done, sinfo.bytes_avail);
+
+            /* check the payload block state */
+            switch (s->bat[sinfo.bat_idx] & VHDX_BAT_STATE_BIT_MASK) {
+            case PAYLOAD_BLOCK_NOT_PRESENT: /* fall through */
+            case PAYLOAD_BLOCK_UNDEFINED:   /* fall through */
+            case PAYLOAD_BLOCK_UNMAPPED:    /* fall through */
+            case PAYLOAD_BLOCK_ZERO:
+                /* return zero */
+                qemu_iovec_memset(&hd_qiov, 0, 0, sinfo.bytes_avail);
+                break;
+            case PAYLOAD_BLOCK_FULL_PRESENT:
+                qemu_co_mutex_unlock(&s->lock);
+                ret = bdrv_co_readv(bs->file,
+                                    sinfo.file_offset >> BDRV_SECTOR_BITS,
+                                    sinfo.sectors_avail, &hd_qiov);
+                qemu_co_mutex_lock(&s->lock);
+                if (ret < 0) {
+                    goto exit;
+                }
+                break;
+            case PAYLOAD_BLOCK_PARTIALLY_PRESENT:
+                /* we don't yet support difference files, fall through
+                 * to error */
+            default:
+                ret = -EIO;
+                goto exit;
+                break;
+            }
+            nb_sectors -= sinfo.sectors_avail;
+            sector_num += sinfo.sectors_avail;
+            bytes_done += sinfo.bytes_avail;
+        }
+    }
+    ret = 0;
+exit:
+    qemu_co_mutex_unlock(&s->lock);
+    qemu_iovec_destroy(&hd_qiov);
+    return ret;
 }
 
 
Signature · 1 line
1.8.1.4
#23 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] block: add read-only support to VHDX image format.

Quoted 16 lines · Jeff Cody
block/vhdx.c +16 -0
+static void vhdx_block_translate(BDRVVHDXState *s, int64_t sector_num,
+                                 int nb_sectors, vhdx_sector_info *sinfo)
+{
+    uint32_t block_offset;
+
+    sinfo->bat_idx = sector_num >> s->sectors_per_block_bits;
+    /* effectively a modulo - this gives us the offset into the block
+     * (in sector sizes) for our sector number */
+    block_offset = sector_num - (sinfo->bat_idx << s->sectors_per_block_bits);
+    /* the chunk ratio gives us the interleaving of the sector
+     * bitmaps, so we need to advance our page block index by the
+     * sector bitmaps entry number */
+    sinfo->bat_idx += sinfo->bat_idx >> s->chunk_ratio_bits;
+
+    /* the number of sectors we can read/write in this cycle */
+    sinfo->sectors_avail = s->sectors_per_block - block_offset;

block/vhdx.c static int vhdx_reopen_prepare(BDRVReopenState *state, · line 767

Quoted 10 lines
block/vhdx.c +10 -0
+
+    sinfo->bytes_left = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    if (sinfo->sectors_avail > nb_sectors) {
+        sinfo->sectors_avail = nb_sectors;
+    }
+
+    sinfo->bytes_avail = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    sinfo->file_offset = s->bat[sinfo->bat_idx] >> VHDX_BAT_FILE_OFF_BITS;

If my calculation is correct, a 2 TB image would result in 16 MB of BAT entries for payload blocks. The BAT entries for sector bitmap blocks would be smaller.

On a slow disk the startup time with many VHDX files could be be several seconds though since we need to read all this data into memory.

Perhaps something to consider for the future but not critical right now.

author
#24 lore raw

PATCH v2 5/5 block: add header update capability for VHDX images.

In reply to Jeff Cody

This adds the ability to update the headers in a VHDX image, including generating a new MS-compatible GUID.

As VHDX depends on uuid.h, VHDX is now a configurable build option. If VHDX support is enabled, that will also enable uuid as well.

To enable/disable VHDX: --enable-vhdx, --disable-vhdx

---
 block/Makefile.objs |   2 +-
 block/vhdx.c        | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 configure           |  13 +++++
 3 files changed, 169 insertions(+), 3 deletions(-)
block/Makefile.objs +1 -1
index 5f0358a..da0a990 100644
@@ -2,7 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
-block-obj-y += vhdx.o
+block-obj-$(CONFIG_VHDX) += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c +155 -2

198 lines not on this page · raw message

configure +13 -0
index 51a6c56..9b048ab 100755
@@ -242,6 +242,7 @@ gtk=""
 gtkabi="2.0"
 tpm="no"
 libssh2=""
+vhdx="no"
 
 # parse CC options first
 for opt do
@@ -934,6 +935,11 @@ for opt do
   ;;
   --enable-libssh2) libssh2="yes"
   ;;
+  --enable-vhdx) vhdx="yes" ;
+                 uuid="yes"
+  ;;
+  --disable-vhdx) vhdx="no"
+  ;;
   *) echo "ERROR: unknown option $opt"; show_help="yes"
   ;;
   esac
@@ -1201,6 +1207,8 @@ echo "  --gcov=GCOV              use specified gcov [$gcov_tool]"
 echo "  --enable-tpm             enable TPM support"
 echo "  --disable-libssh2        disable ssh block device support"
 echo "  --enable-libssh2         enable ssh block device support"
+echo "  --disable-vhdx           disables support for the Microsoft VHDX image format"
+echo "  --enable-vhdx            enable support for the Microsoft VHDX image format"
 echo ""
 echo "NOTE: The object files are built at the place where configure is launched"
 exit 1
@@ -3580,6 +3588,7 @@ echo "gcov enabled      $gcov"
 echo "TPM support       $tpm"
 echo "libssh2 support   $libssh2"
 echo "TPM passthrough   $tpm_passthrough"
+echo "vhdx              $vhdx"
 
 if test "$sdl_too_old" = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -3968,6 +3977,10 @@ if test "$virtio_blk_data_plane" = "yes" ; then
   echo 'CONFIG_VIRTIO_BLK_DATA_PLANE=$(CONFIG_VIRTIO)' >> $config_host_mak
 fi
 
+if test "$vhdx" = "yes" ; then
+  echo "CONFIG_VHDX=y" >> $config_host_mak
+fi
+
 # USB host support
 case "$usb" in
 linux)
Signature · 1 line
1.8.1.4
#25 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block: add header update capability for VHDX images.

Quoted 6 lines · Jeff Cody

This adds the ability to update the headers in a VHDX image, including generating a new MS-compatible GUID.

As VHDX depends on uuid.h, VHDX is now a configurable build option. If VHDX support is enabled, that will also enable uuid as well.

Quoted 8 lines

To enable/disable VHDX: --enable-vhdx, --disable-vhdx

---
 block/Makefile.objs |   2 +-
 block/vhdx.c        | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 configure           |  13 +++++
 3 files changed, 169 insertions(+), 3 deletions(-)

Why is this part of the series?

Stefan

author
#26 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block: add header update capability for VHDX images.

Quoted 16 lines · Stefan Hajnoczi
Quoted 6 lines · Jeff Cody

This adds the ability to update the headers in a VHDX image, including generating a new MS-compatible GUID.

As VHDX depends on uuid.h, VHDX is now a configurable build option. If VHDX support is enabled, that will also enable uuid as well.

Quoted 8 lines

To enable/disable VHDX: --enable-vhdx, --disable-vhdx

---
 block/Makefile.objs |   2 +-
 block/vhdx.c        | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 configure           |  13 +++++
 3 files changed, 169 insertions(+), 3 deletions(-)
Quoted 3 lines

Why is this part of the series?

Stefan

This could technically be dropped from the series, since the current series only supports r/o functions.

I included it because it is not log-dependent - This patch provides the basic mechanism to update the headers (header updates do not go through the log), so it is safe. It is not required yet, but it will be in the future.

#27 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block: add header update capability for VHDX images.

Quoted 21 lines · Jeff Cody
Quoted 16 lines · Stefan Hajnoczi
Quoted 6 lines · Jeff Cody

This adds the ability to update the headers in a VHDX image, including generating a new MS-compatible GUID.

As VHDX depends on uuid.h, VHDX is now a configurable build option. If VHDX support is enabled, that will also enable uuid as well.

Quoted 8 lines

To enable/disable VHDX: --enable-vhdx, --disable-vhdx

---
 block/Makefile.objs |   2 +-
 block/vhdx.c        | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 configure           |  13 +++++
 3 files changed, 169 insertions(+), 3 deletions(-)
Quoted 3 lines

Why is this part of the series?

Stefan

Quoted 7 lines

This could technically be dropped from the series, since the current series only supports r/o functions.

I included it because it is not log-dependent - This patch provides the basic mechanism to update the headers (header updates do not go through the log), so it is safe. It is not required yet, but it will be in the future.

I suggest putting this patch into the series that makes use of it.

Stefan

#28 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block: add header update capability for VHDX images.

In reply to Jeff Cody

block/vhdx.c static int vhdx_open(BlockDriverState *bs, QDict *options, int flags) · line 873

Quoted 4 lines · Jeff Cody
     if (flags & BDRV_O_RDWR) {
block/vhdx.c +1 -2
-        ret = -ENOTSUP;
-        goto fail;
+        vhdx_update_headers(bs, s, false);

Do we really have to update the header immediately on RW open? I assume when implementing vhdx_co_writev, this is guaranteed to get called before other write. In that case we don't need it here, which means we don't change anything if no user write, even for RW opened file.

Signature · 1 line
Fam
author
#29 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block: add header update capability for VHDX images.

Quoted 11 lines · Fam Zheng

block/vhdx.c static int vhdx_open(BlockDriverState *bs, QDict *options, int flags) · line 873

Quoted 4 lines · Jeff Cody
     if (flags & BDRV_O_RDWR) {
block/vhdx.c +1 -2
-        ret = -ENOTSUP;
-        goto fail;
+        vhdx_update_headers(bs, s, false);

Do we really have to update the header immediately on RW open? I assume when implementing vhdx_co_writev, this is guaranteed to get called before other write. In that case we don't need it here, which means we don't change anything if no user write, even for RW opened file.

There are two file modification GUIDS that need to be generated: a 'file write' guid, and a 'data write' guid. The data write GUID encompasses anything visible to the guest. The file write GUID includes any modifications to metadata.

This will update the file write GUID in the header, but not the data write GUID. According the spec, the file write GUID is for any data
modification, including metadata. So once the log support is
implemented, we could latch that this has not occurred and do it during a log write if we wanted (since all metdata updates will go through the log).

In the vhdx_co_writev patches from the RFC, it does writes the data guid if it detects it is the first write.

However, the spec says (with regard to the file write guid):

"The parser can skip updating this field if the storage media on which
the file is stored is read-only, or if the file is opened in read-only mode."

Which I think implies that we should go ahead and update the file write guid field if the image is opened r/w. This would also encompass any modifications to resizing the image, changing backing files (once differencing files are supported), which would not go through vhdx_co_writev().

All that said, I am getting ready to submit v3, and I am dropping this patch from the series for now.

Thanks,
Jeff