Skip to contentSkip to conversation

Initial VHDX support

PATCH Applied Jeff Cody 3 participants 8 messages Apr 29, 2013 – May 2, 2013 Block layer

lore
Share & tools
lore

Jeff Cody

author
#0 lore raw

v3 cover Initial VHDX support

Differences from v2:

Patch 2/4: changed 2 uint8_t[16] to MSGUID (Kevin)
renamed all strucs to proper style (Kevin) indentations / style (Kevin) Removed/fixed outdated comments (Stefan) dropped vhdx_header_padded (Kevin)

Patch 3/4: removed hunk that was meant for old patch 5/5 (Fam)
check VHDX header version field (Fam)
validate file type id in open and not just probe (Kevin) allow headers with seq # of 0 if other header is invalid (Kevin) fixed masking typo for parent meta present (Kevin) removed error check for 512-byte sectors (Kevin) verify region and metadata table guid are unique (Kevin) removed signed/unsigned comparisons (Stefan)
sanity checking on rt.entry_count (Stefan)
use clz_() instead of inline code (Stefan) bat_rt.length sanity check - compute it rather
than trust the entry length (Stefan)

Patch 5 (v2): Dropped

This adds the initial support for VHDX image files.

It currently only supports read operations of VHDX, for fixed and dynamic files.

Notably, the following is not yet supported:
* Differencing files
* Log replay (so we will refuse to open any images that are not 'clean')
* .bdrv_create()
* write operations other than to the header

Jeff Cody (4):
qemu: add castagnoli crc32c checksum algorithm
block: vhdx header for the QEMU support of VHDX images
block: initial VHDX driver support framework - supports open and probe
block: add read-only support to VHDX image format.

 block/Makefile.objs   |   1 +
 block/vhdx.c          | 972 ++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h          | 325 +++++++++++++++++
 include/qemu/crc32c.h |  35 ++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++
 6 files changed, 1449 insertions(+)
 create mode 100644 block/vhdx.c
 create mode 100644 block/vhdx.h
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
Signature · 1 line
1.8.1.4
author
#1 lore raw

PATCH v3 1/4 qemu: add castagnoli crc32c checksum algorithm

This adds the Castagnoli CRC32C algorithm, using the 0x11EDC6F41 polynomial.

This is extracted from the linux kernel cryptographic crc32.c module.

The algorithm is based on:

Castagnoli93: Guy Castagnoli and Stefan Braeuer and Martin Herrman
"Optimization of Cyclic Redundancy-Check Codes with 24
and 32 Parity Bits", IEEE Transactions on Communication, Volume 41, Number 6, June 1993

---
 include/qemu/crc32c.h |  35 +++++++++++++++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 151 insertions(+)
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
include/qemu/crc32c.h new file +35 -0
new file mode 100644
index 0000000..56d1c3b
@@ -0,0 +1,35 @@
+/*
+ *  Castagnoli CRC32C Checksum Algorithm
+ *
+ *  Polynomial: 0x11EDC6F41
+ *
+ *  Castagnoli93: Guy Castagnoli and Stefan Braeuer and Martin Herrman
+ *               "Optimization of Cyclic Redundancy-Check Codes with 24
+ *                 and 32 Parity Bits",IEEE Transactions on Communication,
+ *                Volume 41, Number 6, June 1993
+ *
+ *  Copyright (c) 2013 Red Hat, Inc.,
+ *
+ *  Authors:
+ *   Jeff Cody <[email protected]>
+ *
+ *  Based on the Linux kernel cryptographic crc32c module,
+ *
+ *  Copyright (c) 2004 Cisco Systems, Inc.
+ *  Copyright (c) 2008 Herbert Xu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+
+#ifndef QEMU_CRC32_H
+#define QEMU_CRC32_H
+
+#include "qemu-common.h"
+
+uint32_t crc32c(uint32_t crc, const uint8_t *data, unsigned int length);
+
+#endif
util/Makefile.objs +1 -0
index c5652f5..4a1bd4e 100644
@@ -10,3 +10,4 @@ util-obj-$(CONFIG_POSIX) += compatfd.o
 util-obj-y += iov.o aes.o qemu-config.o qemu-sockets.o uri.o notify.o
 util-obj-y += qemu-option.o qemu-progress.o
 util-obj-y += hexdump.o
+util-obj-y += crc32c.o
util/crc32c.c new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
1.8.1.4
author
#2 lore raw

PATCH v3 2/4 block: vhdx header for the QEMU support of VHDX images

In reply to Jeff Cody

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681

These structures define the various header, metadata, and other block structures defined in the VHDX specification.

---
 block/vhdx.h | 311 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 311 insertions(+)
 create mode 100644 block/vhdx.h
block/vhdx.h new file +311 -0
new file mode 100644
index 0000000..fcddd37
@@ -0,0 +1,311 @@
+/*
+ * Block driver for Hyper-V VHDX Images
+ *
+ * Copyright (c) 2013 Red Hat, Inc.,
+ *
+ * Authors:
+ *  Jeff Cody <[email protected]>
+ *
+ *  This is based on the "VHDX Format Specification v0.95", published 4/12/2012
+ *  by Microsoft:
+ *      https://www.microsoft.com/en-us/download/details.aspx?id=29681
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_VHDX_H
+#define BLOCK_VHDX_H
+
+/* Structures and fields present in the VHDX file */
+
+/* The header section has the following blocks,
+ * each block is 64KB:
+ *
+ * _____________________________________________________________________________
+ * | File Id. |   Header 1    | Header 2   | Region Table |  Reserved (768KB)  |
+ * |----------|---------------|------------|--------------|--------------------|
+ * |          |               |            |              |                    |
+ * 0.........64KB...........128KB........192KB..........256KB................1MB
+ */
+
+#define VHDX_HEADER_BLOCK_SIZE      (64*1024)
+
+#define VHDX_FILE_ID_OFFSET         0
+#define VHDX_HEADER1_OFFSET         (VHDX_HEADER_BLOCK_SIZE*1)
+#define VHDX_HEADER2_OFFSET         (VHDX_HEADER_BLOCK_SIZE*2)
+#define VHDX_REGION_TABLE_OFFSET    (VHDX_HEADER_BLOCK_SIZE*3)
+
+
+/*
+ * A note on the use of MS-GUID fields.  For more details on the GUID,
+ * please see: https://en.wikipedia.org/wiki/Globally_unique_identifier.
+ *
+ * The VHDX specification only states that these are MS GUIDs, and which
+ * bytes are data1-data4. It makes no mention of what algorithm should be used
+ * to generate the GUID, nor what standard.  However, looking at the specified
+ * known GUID fields, it appears the GUIDs are:
+ *  Standard/DCE GUID type  (noted by 10b in the MSB of byte 0 of .data4)
+ *  Random algorithm        (noted by 0x4XXX for .data3)
+ */
+
+/* ---- HEADER SECTION STRUCTURES ---- */
+
+/* These structures are ones that are defined in the VHDX specification
+ * document */
+
+typedef struct VHDXFileIdentifier {
+    uint64_t    signature;              /* "vhdxfile" in ASCII */
+    uint16_t    creator[256];           /* optional; utf-16 string to identify
+                                           the vhdx file creator.  Diagnotistic
+                                           only */
+} VHDXFileIdentifier;
+
+
+/* the guid is a 16 byte unique ID - the definition for this used by
+ * Microsoft is not just 16 bytes though - it is a structure that is defined,
+ * so we need to follow it here so that endianness does not trip us up */
+
+typedef struct MSGUID {
+    uint32_t  data1;
+    uint16_t  data2;
+    uint16_t  data3;
+    uint8_t   data4[8];
+} MSGUID;
+
+#define guid_eq(a, b) \
+    (memcmp(&(a), &(b), sizeof(MSGUID)) == 0)
+
+#define VHDX_HEADER_SIZE (4*1024)   /* although the vhdx_header struct in disk
+                                       is only 582 bytes, for purposes of crc
+                                       the header is the first 4KB of the 64KB
+                                       block */
+
+/* The full header is 4KB, although the actual header data is much smaller.
+ * But for the checksum calculation, it is over the entire 4KB structure,
+ * not just the defined portion of it */
+typedef struct QEMU_PACKED VHDXHeader {
+    uint32_t    signature;              /* "head" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the whole header */
+    uint64_t    sequence_number;        /* Seq number of this header.  Each
+                                           VHDX file has 2 of these headers,
+                                           and only the header with the highest
+                                           sequence number is valid */
+    MSGUID      file_write_guid;       /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           file */
+    MSGUID      data_write_guid;        /* 128 bit unique identifier. Must be
+                                           updated to new, unique value before
+                                           the first modification is made to
+                                           visible data.   Visbile data is
+                                           defined as:
+                                                    - system & user metadata
+                                                    - raw block data
+                                                    - disk size
+                                                    - any change that will
+                                                      cause the virtual disk
+                                                      sector read to differ
+
+                                           This does not need to change if
+                                           blocks are re-arranged */
+    MSGUID      log_guid;               /* 128 bit unique identifier. If zero,
+                                           there is no valid log. If non-zero,
+                                           log entries with this guid are
+                                           valid. */
+    uint16_t    log_version;            /* version of the log format. Mustn't be
+                                           zero, unless log_guid is also zero */
+    uint16_t    version;                /* version of th evhdx file.  Currently,
+                                           only supported version is "1" */
+    uint32_t    log_length;             /* length of the log.  Must be multiple
+                                           of 1MB */
+    uint64_t    log_offset;             /* byte offset in the file of the log.
+                                           Must also be a multiple of 1MB */
+} VHDXHeader;
+
+/* Header for the region table block */
+typedef struct QEMU_PACKED VHDXRegionTableHeader {
+    uint32_t    signature;              /* "regi" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_count;            /* number of valid entries */
+    uint32_t    reserved;
+} VHDXRegionTableHeader;
+
+/* Individual region table entry.  There may be a maximum of 2047 of these
+ *
+ *  There are two known region table properties.  Both are required.
+ *  BAT (block allocation table):  2DC27766F62342009D64115E9BFD4A08
+ *  Metadata:                      8B7CA20647904B9AB8FE575F050F886E
+ */
+#define VHDX_REGION_ENTRY_REQUIRED  0x01    /* if set, parser must understand
+                                               this entry in order to open
+                                               file */
+typedef struct QEMU_PACKED VHDXRegionTableEntry {
+    MSGUID      guid;                   /* 128-bit unique identifier */
+    uint64_t    file_offset;            /* offset of the object in the file.
+                                           Must be multiple of 1MB */
+    uint32_t    length;                 /* length, in bytes, of the object */
+    uint32_t    data_bits;
+} VHDXRegionTableEntry;
+
+
+/* ---- LOG ENTRY STRUCTURES ---- */
+#define VHDX_LOG_HDR_SIZE 64
+typedef struct QEMU_PACKED VHDXLogEntryHeader {
+    uint32_t    signature;              /* "loge" in ASCII */
+    uint32_t    checksum;               /* CRC-32C hash of the 64KB table */
+    uint32_t    entry_length;           /* length in bytes, multiple of 1MB */
+    uint32_t    tail;                   /* byte offset of first log entry of a
+                                           seq, where this entry is the last
+                                           entry */
+    uint64_t    sequence_number;        /* incremented with each log entry.
+                                           May not be zero. */
+    uint32_t    descriptor_count;       /* number of descriptors in this log
+                                           entry, must be >= 0 */
+    uint32_t    reserved;
+    MSGUID      log_guid;               /* value of the log_guid from
+                                           vhdx_header.  If not found in
+                                           vhdx_header, it is invalid */
+    uint64_t    flushed_file_offset;    /* see spec for full details - this
+                                           sould be vhdx file size in bytes */
+    uint64_t    last_file_offset;       /* size in bytes that all allocated
+                                           file structures fit into */
+} VHDXLogEntryHeader;
+
+#define VHDX_LOG_DESC_SIZE 32
+
+typedef struct QEMU_PACKED VHDXLogDescriptor {
+    uint32_t    signature;              /* "zero" or "desc" in ASCII */
+    union  {
+        uint32_t    reserved;           /* zero desc */
+        uint32_t    trailing_bytes;     /* data desc: bytes 4092-4096 of the
+                                           data sector */
+    };
+    union {
+        uint64_t    zero_length;        /* zero desc: length of the section to
+                                           zero */
+        uint64_t    leading_bytes;      /* data desc: bytes 0-7 of the data
+                                           sector */
+    };
+    uint64_t    file_offset;            /* file offset to write zeros - multiple
+                                           of 4kB */
+    uint64_t    sequence_number;        /* must match same field in
+                                           vhdx_log_entry_header */
+} VHDXLogDescriptor;
+
+typedef struct QEMU_PACKED VHDXLogDataSector {
+    uint32_t    data_signature;         /* "data" in ASCII */
+    uint32_t    sequence_high;          /* 4 MSB of 8 byte sequence_number */
+    uint8_t     data[4084];             /* raw data, bytes 8-4091 (inclusive).
+                                           see the data descriptor field for the
+                                           other mising bytes */
+    uint32_t    sequence_low;           /* 4 LSB of 8 byte sequence_number */
+} VHDXLogDataSector;
+
+
+
+/* block states - different state values depending on whether it is a
+ * payload block, or a sector block. */
+
+#define PAYLOAD_BLOCK_NOT_PRESENT       0
+#define PAYLOAD_BLOCK_UNDEFINED         1
+#define PAYLOAD_BLOCK_ZERO              2
+#define PAYLOAD_BLOCK_UNMAPPED          5
+#define PAYLOAD_BLOCK_FULL_PRESENT      6
+#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7
+
+#define SB_BLOCK_NOT_PRESENT    0
+#define SB_BLOCK_PRESENT        6
+
+/* per the spec */
+#define VHDX_MAX_SECTORS_PER_BLOCK  (1<<23)
+
+/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state
+   other bits are reserved */
+#define VHDX_BAT_STATE_BIT_MASK 0x07
+#define VHDX_BAT_FILE_OFF_BITS (64-44)
+typedef uint64_t VHDXBatEntry;
+
+/* ---- METADATA REGION STRUCTURES ---- */
+
+#define VHDX_METADATA_ENTRY_SIZE 32
+#define VHDX_METADATA_MAX_ENTRIES 2047  /* not including the header */
+#define VHDX_METADATA_TABLE_MAX_SIZE \
+    (VHDX_METADATA_ENTRY_SIZE * (VHDX_METADATA_MAX_ENTRIES+1))
+typedef struct QEMU_PACKED VHDXMetadataTableHeader {
+    uint64_t    signature;              /* "metadata" in ASCII */
+    uint16_t    reserved;
+    uint16_t    entry_count;            /* number table entries. <= 2047 */
+    uint32_t    reserved2[5];
+} VHDXMetadataTableHeader;
+
+#define VHDX_META_FLAGS_IS_USER         0x01    /* max 1024 entries */
+#define VHDX_META_FLAGS_IS_VIRTUAL_DISK 0x02    /* virtual disk metadata if set,
+                                                   otherwise file metdata */
+#define VHDX_META_FLAGS_IS_REQUIRED     0x04    /* parse must understand this
+                                                   entry to open the file */
+typedef struct QEMU_PACKED VHDXMetadataTableEntry {
+    MSGUID      item_id;                /* 128-bit identifier for metadata */
+    uint32_t    offset;                 /* byte offset of the metadata.  At
+                                           least 64kB.  Relative to start of
+                                           metadata region */
+                                        /* note: if length = 0, so is offset */
+    uint32_t    length;                 /* length of metadata. <= 1MB. */
+    uint32_t    data_bits;      /* least-significant 3 bits are flags, the
+                                   rest are reserved (see above) */
+    uint32_t    reserved2;
+} VHDXMetadataTableEntry;
+
+#define VHDX_PARAMS_LEAVE_BLOCKS_ALLOCED 0x01   /* Do not change any blocks to
+                                                   be BLOCK_NOT_PRESENT.
+                                                   If set indicates a fixed
+                                                   size VHDX file */
+#define VHDX_PARAMS_HAS_PARENT           0x02    /* has parent / backing file */
+typedef struct QEMU_PACKED VHDXFileParameters {
+    uint32_t    block_size;             /* size of each payload block, always
+                                           power of 2, <= 256MB and >= 1MB. */
+    uint32_t data_bits;     /* least-significant 2 bits are flags, the rest
+                               are reserved (see above) */
+} VHDXFileParameters;
+
+typedef struct QEMU_PACKED VHDXVirtualDiskSize {
+    uint64_t    virtual_disk_size;      /* Size of the virtual disk, in bytes.
+                                           Must be multiple of the sector size,
+                                           max of 64TB */
+} VHDXVirtualDiskSize;
+
+typedef struct QEMU_PACKED VHDXPage83Data {
+    MSGUID      page_83_data[16];       /* unique id for scsi devices that
+                                           support page 0x83 */
+} VHDXPage83Data;
+
+typedef struct QEMU_PACKED VHDXVirtualDiskLogicalSectorSize {
+    uint32_t    logical_sector_size;    /* virtual disk sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} VHDXVirtualDiskLogicalSectorSize;
+
+typedef struct QEMU_PACKED VHDXVirtualDiskPhysicalSectorSize {
+    uint32_t    physical_sector_size;   /* physical sector size (in bytes).
+                                           Can only be 512 or 4096 bytes */
+} VHDXVirtualDiskPhysicalSectorSize;
+
+typedef struct QEMU_PACKED VHDXParentLocatorHeader {
+    MSGUID      locator_type[16];       /* type of the parent virtual disk. */
+    uint16_t    reserved;
+    uint16_t    key_value_count;        /* number of key/value pairs for this
+                                           locator */
+} VHDXParentLocatorHeader;
+
+/* key and value strings are UNICODE strings, UTF-16 LE encoding, no NULs */
+typedef struct QEMU_PACKED VHDXParentLocatorEntry {
+    uint32_t    key_offset;             /* offset in metadata for key, > 0 */
+    uint32_t    value_offset;           /* offset in metadata for value, >0 */
+    uint16_t    key_length;             /* length of entry key, > 0 */
+    uint16_t    value_length;           /* length of entry value, > 0 */
+} VHDXParentLocatorEntry;
+
+
+/* ----- END VHDX SPECIFICATION STRUCTURES ---- */
+
+#endif
Signature · 1 line
1.8.1.4
author
#3 lore raw

PATCH v3 3/4 block: initial VHDX driver support framework - supports open and probe

In reply to Jeff Cody

This is the initial block driver framework for VHDX image support (i.e. Hyper-V image file formats), that supports opening VHDX files, and parsing the headers.

This commit does not yet enable:
- reading
- writing
- updating the header
- differencing files (images with parents)
- log replay / dirty logs (only clean images)

This is based on Microsoft's VHDX specification:

    "VHDX Format Specification v0.95", published 4/12/2012
    https://www.microsoft.com/en-us/download/details.aspx?id=29681
---
 block/Makefile.objs |   1 +
 block/vhdx.c        | 853 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h        |  14 +
 3 files changed, 868 insertions(+)
 create mode 100644 block/vhdx.c
block/Makefile.objs +1 -0
index 6c4b5bc..5f0358a 100644
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
 block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
+block-obj-y += vhdx.o
 block-obj-y += parallels.o blkdebug.o blkverify.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/vhdx.c new file +853 -0

853 lines not on this page · raw message

block/vhdx.h +14 -0
index fcddd37..c3b64c6 100644
@@ -308,4 +308,18 @@ typedef struct QEMU_PACKED VHDXParentLocatorEntry {
 
 /* ----- END VHDX SPECIFICATION STRUCTURES ---- */
 
+
+uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,
+                            int crc_offset);
+
+bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset);
+
+
+static void leguid_to_cpus(MSGUID *guid)
+{
+    le32_to_cpus(&guid->data1);
+    le16_to_cpus(&guid->data2);
+    le16_to_cpus(&guid->data3);
+}
+
 #endif
Signature · 1 line
1.8.1.4
author
#4 lore raw

PATCH v3 4/4 block: add read-only support to VHDX image format.

In reply to Jeff Cody

This adds in read-only support to the VHDX image format. This supports reads for fixed-size, and dynamic sized VHDX images.

Differencing files are still unsupported.

The image must be opened without BDRV_O_RDWR set, because we do not yet update the headers. I.e., pass 'readonly=on' in the drive image options from the QEMU commandline.

---
 block/vhdx.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 121 insertions(+), 2 deletions(-)
block/vhdx.c +121 -2
index 0ee10a7..e9704b1 100644
@@ -114,6 +114,17 @@ typedef struct VHDXMetadataEntries {
 } VHDXMetadataEntries;
 
 
+typedef struct VHDXSectorInfo {
+    uint32_t bat_idx;       /* BAT entry index */
+    uint32_t sectors_avail; /* sectors available in payload block */
+    uint32_t bytes_left;    /* bytes left in the block after data to r/w */
+    uint32_t bytes_avail;   /* bytes available in payload block */
+    uint64_t file_offset;   /* absolute offset in bytes, in file */
+    uint64_t block_offset;  /* block offset, in bytes */
+} VHDXSectorInfo;
+
+
+
 typedef struct BDRVVHDXState {
     CoMutex lock;
 
@@ -792,7 +803,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags)
         goto fail;
     }
 
-    /* TODO: differencing files, read, write */
+    /* TODO: differencing files, write */
 
     return 0;
 fail:
@@ -810,10 +821,118 @@ static int vhdx_reopen_prepare(BDRVReopenState *state,
 }
 
 
+/*
+ * Perform sector to block offset translations, to get various
+ * sector and file offsets into the image.  See VHDXSectorInfo
+ */
+static void vhdx_block_translate(BDRVVHDXState *s, int64_t sector_num,
+                                 int nb_sectors, VHDXSectorInfo *sinfo)
+{
+    uint32_t block_offset;
+
+    sinfo->bat_idx = sector_num >> s->sectors_per_block_bits;
+    /* effectively a modulo - this gives us the offset into the block
+     * (in sector sizes) for our sector number */
+    block_offset = sector_num - (sinfo->bat_idx << s->sectors_per_block_bits);
+    /* the chunk ratio gives us the interleaving of the sector
+     * bitmaps, so we need to advance our page block index by the
+     * sector bitmaps entry number */
+    sinfo->bat_idx += sinfo->bat_idx >> s->chunk_ratio_bits;
+
+    /* the number of sectors we can read/write in this cycle */
+    sinfo->sectors_avail = s->sectors_per_block - block_offset;
+
+    sinfo->bytes_left = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    if (sinfo->sectors_avail > nb_sectors) {
+        sinfo->sectors_avail = nb_sectors;
+    }
+
+    sinfo->bytes_avail = sinfo->sectors_avail << s->logical_sector_size_bits;
+
+    sinfo->file_offset = s->bat[sinfo->bat_idx] >> VHDX_BAT_FILE_OFF_BITS;
+
+    sinfo->block_offset = block_offset << s->logical_sector_size_bits;
+
+    /* The file offset must be past the header section, so must be > 0 */
+    if (sinfo->file_offset == 0) {
+        return;
+    }
+
+    /* block offset is the offset in vhdx logical sectors, in
+     * the payload data block. Convert that to a byte offset
+     * in the block, and add in the payload data block offset
+     * in the file, in bytes, to get the final read address */
+
+    sinfo->file_offset <<= 20;  /* now in bytes, rather than 1MB units */
+    sinfo->file_offset += sinfo->block_offset;
+}
+
+
+
 static coroutine_fn int vhdx_co_readv(BlockDriverState *bs, int64_t sector_num,
                                       int nb_sectors, QEMUIOVector *qiov)
 {
-    return -ENOTSUP;
+    BDRVVHDXState *s = bs->opaque;
+    int ret = 0;
+    VHDXSectorInfo sinfo;
+    uint64_t bytes_done = 0;
+    QEMUIOVector hd_qiov;
+
+    qemu_iovec_init(&hd_qiov, qiov->niov);
+
+    qemu_co_mutex_lock(&s->lock);
+
+    while (nb_sectors > 0) {
+        /* We are a differencing file, so we need to inspect the sector bitmap
+         * to see if we have the data or not */
+        if (s->params.data_bits & VHDX_PARAMS_HAS_PARENT) {
+            /* not supported yet */
+            ret = -ENOTSUP;
+            goto exit;
+        } else {
+            vhdx_block_translate(s, sector_num, nb_sectors, &sinfo);
+
+            qemu_iovec_reset(&hd_qiov);
+            qemu_iovec_concat(&hd_qiov, qiov,  bytes_done, sinfo.bytes_avail);
+
+            /* check the payload block state */
+            switch (s->bat[sinfo.bat_idx] & VHDX_BAT_STATE_BIT_MASK) {
+            case PAYLOAD_BLOCK_NOT_PRESENT: /* fall through */
+            case PAYLOAD_BLOCK_UNDEFINED:   /* fall through */
+            case PAYLOAD_BLOCK_UNMAPPED:    /* fall through */
+            case PAYLOAD_BLOCK_ZERO:
+                /* return zero */
+                qemu_iovec_memset(&hd_qiov, 0, 0, sinfo.bytes_avail);
+                break;
+            case PAYLOAD_BLOCK_FULL_PRESENT:
+                qemu_co_mutex_unlock(&s->lock);
+                ret = bdrv_co_readv(bs->file,
+                                    sinfo.file_offset >> BDRV_SECTOR_BITS,
+                                    sinfo.sectors_avail, &hd_qiov);
+                qemu_co_mutex_lock(&s->lock);
+                if (ret < 0) {
+                    goto exit;
+                }
+                break;
+            case PAYLOAD_BLOCK_PARTIALLY_PRESENT:
+                /* we don't yet support difference files, fall through
+                 * to error */
+            default:
+                ret = -EIO;
+                goto exit;
+                break;
+            }
+            nb_sectors -= sinfo.sectors_avail;
+            sector_num += sinfo.sectors_avail;
+            bytes_done += sinfo.bytes_avail;
+        }
+    }
+    ret = 0;
+exit:
+    qemu_co_mutex_unlock(&s->lock);
+    qemu_iovec_destroy(&hd_qiov);
+    return ret;
 }
 
 
Signature · 1 line
1.8.1.4
#5 lore raw
In reply to Jeff Cody
Quoted 42 lines · Jeff Cody

Differences from v2:

Patch 2/4: changed 2 uint8_t[16] to MSGUID (Kevin)
renamed all strucs to proper style (Kevin) indentations / style (Kevin) Removed/fixed outdated comments (Stefan) dropped vhdx_header_padded (Kevin)

Patch 3/4: removed hunk that was meant for old patch 5/5 (Fam)
check VHDX header version field (Fam)
validate file type id in open and not just probe (Kevin) allow headers with seq # of 0 if other header is invalid (Kevin) fixed masking typo for parent meta present (Kevin) removed error check for 512-byte sectors (Kevin) verify region and metadata table guid are unique (Kevin) removed signed/unsigned comparisons (Stefan)
sanity checking on rt.entry_count (Stefan)
use clz_() instead of inline code (Stefan) bat_rt.length sanity check - compute it rather
than trust the entry length (Stefan)

Patch 5 (v2): Dropped

This adds the initial support for VHDX image files.

It currently only supports read operations of VHDX, for fixed and dynamic files.

Notably, the following is not yet supported:
* Differencing files
* Log replay (so we will refuse to open any images that are not 'clean')
* .bdrv_create()
* write operations other than to the header

Jeff Cody (4):
qemu: add castagnoli crc32c checksum algorithm
block: vhdx header for the QEMU support of VHDX images
block: initial VHDX driver support framework - supports open and probe
block: add read-only support to VHDX image format.

Quoted 11 lines
 block/Makefile.objs   |   1 +
 block/vhdx.c          | 972 ++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h          | 325 +++++++++++++++++
 include/qemu/crc32c.h |  35 ++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++
 6 files changed, 1449 insertions(+)
 create mode 100644 block/vhdx.c
 create mode 100644 block/vhdx.h
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c

Do you have links to vhdx files I can test?

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan

author
#6 lore raw
Quoted 55 lines · Stefan Hajnoczi
Quoted 42 lines · Jeff Cody

Differences from v2:

Patch 2/4: changed 2 uint8_t[16] to MSGUID (Kevin)
renamed all strucs to proper style (Kevin) indentations / style (Kevin) Removed/fixed outdated comments (Stefan) dropped vhdx_header_padded (Kevin)

Patch 3/4: removed hunk that was meant for old patch 5/5 (Fam)
check VHDX header version field (Fam)
validate file type id in open and not just probe (Kevin) allow headers with seq # of 0 if other header is invalid (Kevin) fixed masking typo for parent meta present (Kevin) removed error check for 512-byte sectors (Kevin) verify region and metadata table guid are unique (Kevin) removed signed/unsigned comparisons (Stefan)
sanity checking on rt.entry_count (Stefan)
use clz_() instead of inline code (Stefan) bat_rt.length sanity check - compute it rather
than trust the entry length (Stefan)

Patch 5 (v2): Dropped

This adds the initial support for VHDX image files.

It currently only supports read operations of VHDX, for fixed and dynamic files.

Notably, the following is not yet supported:
* Differencing files
* Log replay (so we will refuse to open any images that are not 'clean')
* .bdrv_create()
* write operations other than to the header

Jeff Cody (4):
qemu: add castagnoli crc32c checksum algorithm
block: vhdx header for the QEMU support of VHDX images
block: initial VHDX driver support framework - supports open and probe
block: add read-only support to VHDX image format.

Quoted 11 lines
 block/Makefile.objs   |   1 +
 block/vhdx.c          | 972 ++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h          | 325 +++++++++++++++++
 include/qemu/crc32c.h |  35 ++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++
 6 files changed, 1449 insertions(+)
 create mode 100644 block/vhdx.c
 create mode 100644 block/vhdx.h
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
Quoted 2 lines

Do you have links to vhdx files I can test?

I created a new VHDX file, because the ones I've been testing with have grown too large to share easily. It is a 32GB dynamic image that contains a single EXT4 partition:

https://www.dropbox.com/s/52q84zy2n59ix6h/TestDisk32GBDynamic-ext4.vhdx.7z

I'll keep this link around for a while, but it will eventually go away.

Inside the ext4 partition there are 10 image files generated from /dev/urandom, and corresponding SHA1 hashes in a single SHA1SUM file, for verification.

Also, if you wish to check the SHA1SUM of the entire image from the view of the guest (e.g. sha1sum /dev/vdb), the resulting hash should be:

f897e5cadd3a0b1e776625caf79ed0eaf4baa3d8 /dev/vdb

(The above has was calculated from a Linux guest under Hyper-V 2012, and verified with these patches and QEMU).

Quoted 3 lines

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Thanks!

Jeff

#7 lore raw
Quoted 72 lines · Jeff Cody
Quoted 55 lines · Stefan Hajnoczi
Quoted 42 lines · Jeff Cody

Differences from v2:

Patch 2/4: changed 2 uint8_t[16] to MSGUID (Kevin)
renamed all strucs to proper style (Kevin) indentations / style (Kevin) Removed/fixed outdated comments (Stefan) dropped vhdx_header_padded (Kevin)

Patch 3/4: removed hunk that was meant for old patch 5/5 (Fam)
check VHDX header version field (Fam)
validate file type id in open and not just probe (Kevin) allow headers with seq # of 0 if other header is invalid (Kevin) fixed masking typo for parent meta present (Kevin) removed error check for 512-byte sectors (Kevin) verify region and metadata table guid are unique (Kevin) removed signed/unsigned comparisons (Stefan)
sanity checking on rt.entry_count (Stefan)
use clz_() instead of inline code (Stefan) bat_rt.length sanity check - compute it rather
than trust the entry length (Stefan)

Patch 5 (v2): Dropped

This adds the initial support for VHDX image files.

It currently only supports read operations of VHDX, for fixed and dynamic files.

Notably, the following is not yet supported:
* Differencing files
* Log replay (so we will refuse to open any images that are not 'clean')
* .bdrv_create()
* write operations other than to the header

Jeff Cody (4):
qemu: add castagnoli crc32c checksum algorithm
block: vhdx header for the QEMU support of VHDX images
block: initial VHDX driver support framework - supports open and probe
block: add read-only support to VHDX image format.

Quoted 11 lines
 block/Makefile.objs   |   1 +
 block/vhdx.c          | 972 ++++++++++++++++++++++++++++++++++++++++++++++++++
 block/vhdx.h          | 325 +++++++++++++++++
 include/qemu/crc32c.h |  35 ++
 util/Makefile.objs    |   1 +
 util/crc32c.c         | 115 ++++++
 6 files changed, 1449 insertions(+)
 create mode 100644 block/vhdx.c
 create mode 100644 block/vhdx.h
 create mode 100644 include/qemu/crc32c.h
 create mode 100644 util/crc32c.c
Quoted 2 lines

Do you have links to vhdx files I can test?

I created a new VHDX file, because the ones I've been testing with have grown too large to share easily. It is a 32GB dynamic image that contains a single EXT4 partition:

https://www.dropbox.com/s/52q84zy2n59ix6h/TestDisk32GBDynamic-ext4.vhdx.7z

I'll keep this link around for a while, but it will eventually go away.

Inside the ext4 partition there are 10 image files generated from /dev/urandom, and corresponding SHA1 hashes in a single SHA1SUM file, for verification.

Quoted 8 lines

Also, if you wish to check the SHA1SUM of the entire image from the view of the guest (e.g. sha1sum /dev/vdb), the resulting hash should be:

f897e5cadd3a0b1e776625caf79ed0eaf4baa3d8 /dev/vdb

(The above has was calculated from a Linux guest under Hyper-V 2012, and verified with these patches and QEMU).

Cool, thanks for making this available!

Stefan