VHDX log replay and write support, .bdrv_create()
v7 cover VHDX log replay and write support, .bdrv_create()
This patch series contains the initial VHDX log parsing, replay, write support, and image creation.
=== v7 changes ===
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v7-upstream
Rebased to latest qemu/master (picked up vhdx r/o tests, migration blocker)
Patch 8/19: * validate log descriptor_count (Stefan)
* fix typos in comments (Stefan)
* Removed unneccessary initialization (Stefan)
* Replay log prior to metadata (Stefan)
* In vhdx_log_flush(), call bdrv_flush() prior to zeroing out the log guid in the header.
* In vhdx_close(), set freed pointers to NULL
Patch 9/19: * correct logic for region overlap (Stefan)
Patch 10/19: * add missing goto exit in error case (Stefan)
* add bdrv_flush() to ensure data is stable on disk (Stefan)
Patch 11/19: * fixed typos in comments (Stefan)
* QEMU coding style changes (Stefan)
* rename bat_entry to bat_entry_le for clarity (Stefan)
* Add PAYLOAD_BLOCK_ZERO explicit zero padding for protocols that do not support zero init (Stefan)
* rename PAYLOAD_BLOCK_FULL_PRESENT to PAYLOAD_BLOCK_FULLY_PRESENT (Stefan)
Patch 13/19: * Fixed typo in commit message (Stefan)
Patch 19/19: * New, adds qemu-io test for log replay of data sector
v6 Patch 17/20: * Dropped (already upstream)
v6 Patch 18/20: * Dropped (already upstream)
=== v6 changes ===
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v6-upstream
Rebased to latest qemu/master:
Patch 16/20: .bdrv_create() propagates Error, and bdrv_unref() used
instead of bdrv_delete().
Patch 17 & 18 are already included in another series:
[PATCH v3 0/3] qemu-iotests with sample images, vhdx test, cleanup
They are included here to provide a base for patches 19 & 20. If the above series is applied before this series, then patches 17 and 18 can be ignored.
Patch 19/20: In qemu-io tests _make_test_img(), filter out vhdx-specific
options for .bdrv_create().
Patch 20/20: Add VHDX write test case to 064.
=== v5 changes ===
v5 is also available for testing from:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v5-upstream
Most of the patches from v4 -> v5 are the same, but there are a few differences and a few new patches. Here is a summary of which patches are different and/or new:
Patch highlights:
Patch 7 just some minor code movement, in prep for changes in patch 8
Patch 8 incorporates review feedback from Stefan, for the previous Patch 7
in v4.
Patch 9 adds region checking for log, region table, and metadata tables, per
suggestion from Stefan.
Patch 10 minor change from changes made in 8/16 (vhdx_guid_is_zero() is gone)
Patch 12 is just some minor housekeeping, to get rid of bit shifting that
doesn't need to happen.
=== v4 changes ===
v4 patches are available from github as well, on branch vhdx-write-v4-upstream:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v4-upstream
https://github.com/codyprime/qemu-kvm-jtc.git
Those in the midst of reviewing v3, don't fear - the only changes with v4 is the addition of patches on the end of the series (patches 10-13). These patches enable creating VHDX images. Image files created have been (briefly & lightly) tested on Hyper-V running on Windows Server 2012.
Some of the new patches could be squashed with earlier patches in the series, but I refrained from doing so, since some of the patches have already been reviewed, and others are in the midst of review. I want to make it as easy as possible on those currently reviewing. There is nothing critical that needs to be pushed into the earlier patches.
New patches:
Patch 10: Breaks out some more endian translation functions
(likely squashable into patch 5)
Patch 11: Break out some operations into seperate helper functions
Patch 12: More comment typos and header fixes in vhdx.h
(likely squashable into patch 1)
Patch 13: Adds .bdrv_create() for vhdx. VHDX images are can be created for
Fixed or Dynamic images.
Patches 1-9 are unchanged.
=== end v4 changelog ===
=== v3 changes ===
Thank you Kevin & Stefan for the feedback; incoporated in v3:
Patch 1: --- nil ---
Patch 2: * use sizeof(crc) instead of 4
* remove outdated comment
* use sizeof(MSGUID) instead of 16
* direct assignment of guid structs rather than memcpy
* rename 'rw' to 'generate_data_write_guid'
* use offsetof() instead of 4
* comment typos
* add missing error checking
* MSGUID is now QEMU_PACKED
* configure enable for vhdx is now correct and not braindead
Patch 3: --- nil ---
Patch 4: * code style fixes
* removed unused struct (VHDXLogEntryInfo)
Patch 5: * more direct assignment of guid rather than memcpy
* order of operation in export/import the same now
* became less generous with newlines (bah-humbug!)
Patch 6: * more direct assignment of guid rather than memcpy
* add error check in vhdx_user_visible_write(), now returns int
Patch 7: * check error return now of vhdx_user_visible_write()
Patch 8: * check error return now of vhdx_user_visible_write()
* vhdx_log_write_sectors() uses bdrv_pwrite() vs bdrv_pwrite_sync()
* more direct assignment of guid rather than memcpy
* use offsetof() instead of 4
Patch 9: --- nil ---
=== end v3 changelog ===
v2 changes: Incorporated Fam's review feedback - Thank you Fam for the feedback
=== end v2 changelog ===
This will allow an existing log in a VHDX image to be replayed (e.g., a VHDX image from a Hyper-V host that crashed). In addition, metadata writes are enabled through the log. This allows write support to be enabled for VHDX, as the BAT can be updated safely via the log journal.
These exact patches are available from github, for testing:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v2-upstream
The latest vhdx work (including anything beyond these patches, such as backing/parent file support) can be found at:
https://github.com/codyprime/qemu-kvm-jtc/tree/jtc-vhdx-latest
Jeff Cody (19):
block: vhdx - minor comments and typo correction.
block: vhdx - add header update capability.
block: vhdx code movement - VHDXMetadataEntries and BDRVVHDXState to
header.
block: vhdx - log support struct and defines
block: vhdx - break endian translation functions out
block: vhdx - update log guid in header, and first write tracker
block: vhdx code movement - move vhdx_close() above vhdx_open()
block: vhdx - log parsing, replay, and flush support
block: vhdx - add region overlap detection for image files
block: vhdx - add log write support
block: vhdx write support
block: vhdx - remove BAT file offset bit shifting
block: vhdx - move more endian translations to vhdx-endian.c
block: vhdx - break out code operations to functions
block: vhdx - fix comment typos in header, fix incorrect struct fields
block: vhdx - add .bdrv_create() support
block: vhdx - update _make_test_img() to filter out vhdx options
block: qemu-iotests for vhdx, add write test support
block: vhdx qemu-iotest - log replay of data sector
block/Makefile.objs | 2 +- block/vhdx-endian.c | 216 ++++ block/vhdx-log.c | 1007 ++++++++++++++++ block/vhdx.c | 1203 +++++++++++++++++--- block/vhdx.h | 178 ++- configure | 24 + tests/qemu-iotests/064 | 11 + tests/qemu-iotests/064.out | 14 + tests/qemu-iotests/065 | 67 ++ tests/qemu-iotests/065.out | 8 + tests/qemu-iotests/common | 1 - tests/qemu-iotests/common.rc | 4 +- tests/qemu-iotests/group | 1 + .../sample_images/iotest-dirtylog-10G-4M.vhdx.bz2 | Bin 0 -> 4490 bytes 14 files changed, 2552 insertions(+), 184 deletions(-) create mode 100644 block/vhdx-endian.c create mode 100644 block/vhdx-log.c create mode 100755 tests/qemu-iotests/065 create mode 100644 tests/qemu-iotests/065.out create mode 100644 tests/qemu-iotests/sample_images/iotest-dirtylog-10G-4M.vhdx.bz2
Signature · 1 line
1.8.3.1
PATCH v7 1/19 block: vhdx - minor comments and typo correction.
Just a couple of minor comments to help note where allocated buffers are freed, and a typo fix.
--- block/vhdx.c | 6 ++++-- block/vhdx.h | 6 +++--- 2 files changed, 7 insertions(+), 5 deletions(-)
block/vhdx.c +4 -2
@@ -6,9 +6,9 @@* Authors: * Jeff Cody <[email protected]> * -* This is based on the "VHDX Format Specification v0.95", published 4/12/2012 +* This is based on the "VHDX Format Specification v1.00", published 8/25/2012 * by Microsoft: -* https://www.microsoft.com/en-us/download/details.aspx?id=29681 +* https://www.microsoft.com/en-us/download/details.aspx?id=34750 * * This work is licensed under the terms of the GNU LGPL, version 2 or later. * See the COPYING.LIB file in the top-level directory.
@@ -264,6 +264,7 @@ static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s)uint64_t h2_seq =0 ;uint8_t *buffer; +/* header1 & header2 are freed in vhdx_close() */ header1 =qemu_blockalign (bs,sizeof (VHDXHeader)); header2 =qemu_blockalign (bs,sizeof (VHDXHeader));
@@ -790,6 +791,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } +/* s->bat is freed in vhdx_close() */ s->bat =qemu_blockalign (bs, s->bat_rt.length); ret =bdrv_pread (bs->file, s->bat_offset, s->bat, s->bat_rt.length);
block/vhdx.h +3 -3
@@ -6,9 +6,9 @@* Authors: * Jeff Cody <[email protected]> * -* This is based on the "VHDX Format Specification v0.95", published 4/12/2012 +* This is based on the "VHDX Format Specification v1.00", published 8/25/2012 * by Microsoft: -* https://www.microsoft.com/en-us/download/details.aspx?id=29681 +* https://www.microsoft.com/en-us/download/details.aspx?id=34750 * * This work is licensed under the terms of the GNU LGPL, version 2 or later. * See the COPYING.LIB file in the top-level directory.
@@ -116,7 +116,7 @@ typedef struct QEMU_PACKED VHDXHeader {valid. */ uint16_t log_version;/* version of the log format. Mustn't be zero, unless log_guid is also zero */ -uint16_t version;/* version of th evhdx file. Currently, +uint16_t version;/* version of the vhdx file. Currently, only supported version is "1" */ uint32_t log_length;/* length of the log. Must be multiple of 1MB */
Signature · 1 line
1.8.3.1
PATCH v7 2/19 block: vhdx - add header update capability.
In reply to Jeff CodyThis adds the ability to update the headers in a VHDX image, including generating a new MS-compatible GUID.
As VHDX depends on uuid.h, VHDX is now a configurable build option. If VHDX support is enabled, that will also enable uuid as well. The default is to have VHDX enabled.
To enable/disable VHDX: --enable-vhdx, --disable-vhdx
--- block/Makefile.objs | 2 +- block/vhdx.c | 161 +++++++++++++++++++++++++++++++++++++++++++++++++++- block/vhdx.h | 14 ++++- configure | 24 ++++++++ 4 files changed, 196 insertions(+), 5 deletions(-)
block/Makefile.objs +1 -1
@@ -2,7 +2,7 @@ block-obj-y += raw_bsd.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o v block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o -block-obj-y += vhdx.o +block-obj-$(CONFIG_VHDX) += vhdx.o block-obj-y += parallels.o blkdebug.o blkverify.o block-obj-y += snapshot.o qapi.o block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
block/vhdx.c +159 -2
203 lines not on this page · raw message
block/vhdx.h +12 -2
@@ -67,7 +67,7 @@ typedef struct VHDXFileIdentifier {* Microsoft is not just 16 bytes though - it is a structure that is defined, * so we need to follow it here so that endianness does not trip us up */ -typedef struct MSGUID { +typedef struct QEMU_PACKED MSGUID {uint32_t data1;uint16_t data2;uint16_t data3;
@@ -309,17 +309,27 @@ typedef struct QEMU_PACKED VHDXParentLocatorEntry {/* ----- END VHDX SPECIFICATION STRUCTURES ---- */ +void vhdx_guid_generate (MSGUID *guid); + +uint32_t vhdx_update_checksum (uint8_t *buf,size_t size,int crc_offset);uint32_t vhdx_checksum_calc (uint32_t crc,uint8_t *buf,size_t size,int crc_offset);bool vhdx_checksum_is_valid (uint8_t *buf,size_t size,int crc_offset); -static void leguid_to_cpus (MSGUID *guid) +static inline void leguid_to_cpus (MSGUID *guid) {le32_to_cpus (&guid->data1);le16_to_cpus (&guid->data2);le16_to_cpus (&guid->data3); } +static inline void cpu_to_leguids (MSGUID *guid) +{ +cpu_to_le32s (&guid->data1); +cpu_to_le16s (&guid->data2); +cpu_to_le16s (&guid->data3); +} +#endif
configure +24 -0
@@ -246,6 +246,7 @@ gtk="" gtkabi="2.0" tpm="no" libssh2="" +vhdx="" # parse CC options first for opt do
@@ -972,6 +973,10 @@ for opt do ;; --enable-libssh2) libssh2="yes" ;; + --enable-vhdx) vhdx="yes" + ;; + --disable-vhdx) vhdx="no" + ;; *) echo "ERROR: unknown option $opt"; show_help="yes" ;; esac
@@ -1203,6 +1208,8 @@ echo " --gcov=GCOV use specified gcov [$gcov_tool]" echo " --enable-tpm enable TPM support" echo " --disable-libssh2 disable ssh block device support" echo " --enable-libssh2 enable ssh block device support" +echo " --disable-vhdx disables support for the Microsoft VHDX image format" +echo " --enable-vhdx enable support for the Microsoft VHDX image format" echo "" echo "NOTE: The object files are built at the place where configure is launched" exit 1
@@ -2015,6 +2022,18 @@ EOF fi fi +if test "$vhdx" = "yes" ; then + if test "$uuid" = "no" ; then + error_exit "uuid required for VHDX support" + fi +elif test "$vhdx" != "no" ; then + if test "$uuid" = "yes" ; then + vhdx=yes + else + vhdx=no + fi +fi + ########################################## # xfsctl() probe, used for raw-posix if test "$xfs" != "no" ; then
@@ -3758,6 +3777,7 @@ echo "TPM support $tpm" echo "libssh2 support $libssh2" echo "TPM passthrough $tpm_passthrough" echo "QOM debugging $qom_cast_debug" +echo "vhdx $vhdx" if test "$sdl_too_old" = "yes"; then echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -4160,6 +4180,10 @@ if test "$virtio_blk_data_plane" = "yes" ; then echo 'CONFIG_VIRTIO_BLK_DATA_PLANE=$(CONFIG_VIRTIO)' >> $config_host_mak fi +if test "$vhdx" = "yes" ; then + echo "CONFIG_VHDX=y" >> $config_host_mak +fi + # USB host support if test "$libusb" = "yes"; then echo "HOST_USB=libusb legacy" >> $config_host_mak
Signature · 1 line
1.8.3.1
PATCH v7 3/19 block: vhdx code movement - VHDXMetadataEntries and BDRVVHDXState to header.
In reply to Jeff CodyIn preparation for VHDX log support, move these structures to the header.
--- block/vhdx.c | 52 ---------------------------------------------------- block/vhdx.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 52 deletions(-)
block/vhdx.c +0 -52
@@ -105,16 +105,6 @@ static const MSGUID parent_vhdx_guid = { .data1 = 0xb04aefb7, META_PAGE_83_PRESENT | META_LOGICAL_SECTOR_SIZE_PRESENT | \ META_PHYS_SECTOR_SIZE_PRESENT) -typedef struct VHDXMetadataEntries { -VHDXMetadataTableEntry file_parameters_entry; -VHDXMetadataTableEntry virtual_disk_size_entry; -VHDXMetadataTableEntry page83_data_entry; -VHDXMetadataTableEntry logical_sector_size_entry; -VHDXMetadataTableEntry phys_sector_size_entry; -VHDXMetadataTableEntry parent_locator_entry; -uint16_t present; -}VHDXMetadataEntries ; - typedef struct VHDXSectorInfo { uint32_t bat_idx; /* BAT entry index */
@@ -125,48 +115,6 @@ typedef struct VHDXSectorInfo {uint64_t block_offset;/* block offset, in bytes */ } VHDXSectorInfo; - - -typedef struct BDRVVHDXState { -CoMutex lock; - -int curr_header; -VHDXHeader *headers[2 ]; - -VHDXRegionTableHeader rt; -VHDXRegionTableEntry bat_rt;/* region table for the BAT */ -VHDXRegionTableEntry metadata_rt;/* region table for the metadata */ - -VHDXMetadataTableHeader metadata_hdr; -VHDXMetadataEntries metadata_entries; - -VHDXFileParameters params; -uint32_t block_size; -uint32_t block_size_bits; -uint32_t sectors_per_block; -uint32_t sectors_per_block_bits; - -uint64_t virtual_disk_size; -uint32_t logical_sector_size; -uint32_t physical_sector_size; - -uint64_t chunk_ratio; -uint32_t chunk_ratio_bits; -uint32_t logical_sector_size_bits; - -uint32_t bat_entries; -VHDXBatEntry *bat; -uint64_t bat_offset; - -MSGUID session_guid; - - -VHDXParentLocatorHeader parent_header; -VHDXParentLocatorEntry *parent_entries; - -Error *migration_blocker; -}BDRVVHDXState ; -/* Calculates new checksum. * * Zero is substituted during crc calculation for the original crc field
block/vhdx.h +48 -0
54 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v7 4/19 block: vhdx - log support struct and defines
In reply to Jeff CodyThis adds some magic number defines, and internal structure definitions for VHDX log replay support. The struct VHDXLogEntries does not reflect an on-disk data structure, and thus does not need to be packed.
Some minor code style fixes are applied as well.
--- block/vhdx.h | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-)
block/vhdx.h +30 -16
@@ -30,12 +30,12 @@* 0.........64KB...........128KB........192KB..........256KB................1MB */ -#define VHDX_HEADER_BLOCK_SIZE (64*1024) +#define VHDX_HEADER_BLOCK_SIZE (64 * 1024)#define VHDX_FILE_ID_OFFSET 0 -#define VHDX_HEADER1_OFFSET (VHDX_HEADER_BLOCK_SIZE*1) -#define VHDX_HEADER2_OFFSET (VHDX_HEADER_BLOCK_SIZE*2) -#define VHDX_REGION_TABLE_OFFSET (VHDX_HEADER_BLOCK_SIZE*3) +#define VHDX_HEADER1_OFFSET (VHDX_HEADER_BLOCK_SIZE * 1) +#define VHDX_HEADER2_OFFSET (VHDX_HEADER_BLOCK_SIZE * 2) +#define VHDX_REGION_TABLE_OFFSET (VHDX_HEADER_BLOCK_SIZE * 3)/*
@@ -77,10 +77,10 @@ typedef struct QEMU_PACKED MSGUID {#define guid_eq (a, b) \ (memcmp(&(a), &(b), sizeof(MSGUID)) == 0) -#define VHDX_HEADER_SIZE (4*1024)/* although the vhdx_header struct in disk -is only 582 bytes, for purposes of crc -the header is the first 4KB of the 64KB -block */ +#define VHDX_HEADER_SIZE (4 * 1024)/* although the vhdx_header struct in disk +is only 582 bytes, for purposes of crc +the header is the first 4KB of the 64KB +block */ /* The full header is 4KB, although the actual header data is much smaller. * But for the checksum calculation, it is over the entire 4KB structure,
@@ -92,7 +92,7 @@ typedef struct QEMU_PACKED VHDXHeader {VHDX file has 2 of these headers, and only the header with the highest sequence number is valid */ -MSGUID file_write_guid;/* 128 bit unique identifier. Must be +MSGUID file_write_guid;/* 128 bit unique identifier. Must be updated to new, unique value before the first modification is made to file */
@@ -151,7 +151,10 @@ typedef struct QEMU_PACKED VHDXRegionTableEntry {/* ---- LOG ENTRY STRUCTURES ---- */ +#define VHDX_LOG_MIN_SIZE (1024 * 1024) +#define VHDX_LOG_SECTOR_SIZE 4096#define VHDX_LOG_HDR_SIZE 64 +#define VHDX_LOG_SIGNATURE 0x65676f6c typedef struct QEMU_PACKED VHDXLogEntryHeader {uint32_t signature;/* "loge" in ASCII */ uint32_t checksum;/* CRC-32C hash of the 64KB table */
@@ -174,7 +177,8 @@ typedef struct QEMU_PACKED VHDXLogEntryHeader { } VHDXLogEntryHeader;#define VHDX_LOG_DESC_SIZE 32 - +#define VHDX_LOG_DESC_SIGNATURE 0x63736564 +#define VHDX_LOG_ZERO_SIGNATURE 0x6f72657a typedef struct QEMU_PACKED VHDXLogDescriptor {uint32_t signature;/* "zero" or "desc" in ASCII */ union {
@@ -194,6 +198,7 @@ typedef struct QEMU_PACKED VHDXLogDescriptor {vhdx_log_entry_header */ } VHDXLogDescriptor; +#define VHDX_LOG_DATA_SIGNATURE 0x61746164 typedef struct QEMU_PACKED VHDXLogDataSector {uint32_t data_signature;/* "data" in ASCII */ uint32_t sequence_high;/* 4 MSB of 8 byte sequence_number */
@@ -219,12 +224,12 @@ typedef struct QEMU_PACKED VHDXLogDataSector {#define SB_BLOCK_PRESENT 6/* per the spec */ -#define VHDX_MAX_SECTORS_PER_BLOCK (1<<23) +#define VHDX_MAX_SECTORS_PER_BLOCK (1 << 23)/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state other bits are reserved */ #define VHDX_BAT_STATE_BIT_MASK 0x07 -#define VHDX_BAT_FILE_OFF_BITS (64-44) +#define VHDX_BAT_FILE_OFF_BITS (64 - 44)typedef uint64_t VHDXBatEntry ;/* ---- METADATA REGION STRUCTURES ---- */
@@ -252,8 +257,8 @@ typedef struct QEMU_PACKED VHDXMetadataTableEntry {metadata region */ /* note: if length = 0, so is offset */ uint32_t length;/* length of metadata. <= 1MB. */ -uint32_t data_bits;/* least-significant 3 bits are flags, the -rest are reserved (see above) */ +uint32_t data_bits;/* least-significant 3 bits are flags, +the rest are reserved (see above) */ uint32_t reserved2; } VHDXMetadataTableEntry;
@@ -265,8 +270,8 @@ typedef struct QEMU_PACKED VHDXMetadataTableEntry { typedef struct QEMU_PACKED VHDXFileParameters {uint32_t block_size;/* size of each payload block, always power of 2, <= 256MB and >= 1MB. */ -uint32_t data_bits;/* least-significant 2 bits are flags, the rest -are reserved (see above) */ +uint32_t data_bits;/* least-significant 2 bits are flags, +the rest are reserved (see above) */ } VHDXFileParameters; typedef struct QEMU_PACKED VHDXVirtualDiskSize {
@@ -318,6 +323,13 @@ typedef struct VHDXMetadataEntries {uint16_t present; } VHDXMetadataEntries; +typedef struct VHDXLogEntries { +uint64_t offset; +uint64_t length; +uint32_t head; +uint32_t tail; +}VHDXLogEntries ; + typedef struct BDRVVHDXState { CoMutex lock;
@@ -351,6 +363,8 @@ typedef struct BDRVVHDXState {MSGUID session_guid; +VHDXLogEntries log; +VHDXParentLocatorHeader parent_header;VHDXParentLocatorEntry *parent_entries;
Signature · 1 line
1.8.3.1
PATCH v7 5/19 block: vhdx - break endian translation functions out
In reply to Jeff CodyThis moves the endian translation functions out from the vhdx.c source, into a separate source file. In addition to the previously defined endian functions, new endian translation functions for log support are added as well.
--- block/Makefile.objs | 2 +- block/vhdx-endian.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.c | 43 ---------------- block/vhdx.h | 8 +++ 4 files changed, 150 insertions(+), 44 deletions(-) create mode 100644 block/vhdx-endian.c
block/Makefile.objs +1 -1
@@ -2,7 +2,7 @@ block-obj-y += raw_bsd.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o v block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o -block-obj-$(CONFIG_VHDX) += vhdx.o +block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o block-obj-y += parallels.o blkdebug.o blkverify.o block-obj-y += snapshot.o qapi.o block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
block/vhdx-endian.c new file +141 -0
141 lines not on this page · raw message
block/vhdx.c +0 -43
49 lines not on this page · raw message
block/vhdx.h +8 -0
@@ -394,4 +394,12 @@ static inline void cpu_to_leguids(MSGUID *guid) cpu_to_le16s(&guid->data3); } +void vhdx_header_le_import (VHDXHeader *h); +void vhdx_header_le_export (VHDXHeader *orig_h,VHDXHeader *new_h); +void vhdx_log_desc_le_import (VHDXLogDescriptor *d); +void vhdx_log_desc_le_export (VHDXLogDescriptor *d); +void vhdx_log_data_le_export (VHDXLogDataSector *d); +void vhdx_log_entry_hdr_le_import (VHDXLogEntryHeader *hdr); +void vhdx_log_entry_hdr_le_export (VHDXLogEntryHeader *hdr); +#endif
Signature · 1 line
1.8.3.1
PATCH v7 6/19 block: vhdx - update log guid in header, and first write tracker
In reply to Jeff CodyAllow tracking of first file write in the VHDX image, as well as the ability to update the GUID in the header. This is in preparation for log support.
--- block/vhdx.c | 30 ++++++++++++++++++++++++------ block/vhdx.h | 6 ++++++ 2 files changed, 30 insertions(+), 6 deletions(-)
block/vhdx.c +24 -6
@@ -230,7 +230,7 @@ static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)* - non-current header is updated with largest sequence number */ static int vhdx_update_header (BlockDriverState *bs,BDRVVHDXState *s, -bool generate_data_write_guid) +bool generate_data_write_guid,MSGUID *log_guid) {int ret =0 ;int hdr_idx =0 ;
@@ -262,6 +262,11 @@ static int vhdx_update_header(BlockDriverState *bs, BDRVVHDXState *s,vhdx_guid_generate (&inactive_header->data_write_guid); } +/* update the log guid if present */ +if (log_guid) { + inactive_header->log_guid = *log_guid; + } +/* the header checksum is not over just the packed size of VHDXHeader, * but rather over the entire 'reserved' range for the header, which is * 4KB (VHDX_HEADER_SIZE). */
@@ -294,16 +299,16 @@ exit:* The VHDX spec calls for header updates to be performed twice, so that both * the current and non-current header have valid info */ -static int vhdx_update_headers (BlockDriverState *bs,BDRVVHDXState *s, -bool generate_data_write_guid) +int vhdx_update_headers (BlockDriverState *bs,BDRVVHDXState *s, +bool generate_data_write_guid,MSGUID *log_guid) {int ret; - ret =vhdx_update_header (bs, s, generate_data_write_guid); + ret =vhdx_update_header (bs, s, generate_data_write_guid, log_guid);if (ret <0 ) {return ret; } - ret =vhdx_update_header (bs, s, generate_data_write_guid); + ret =vhdx_update_header (bs, s, generate_data_write_guid, log_guid);return ret; }
@@ -784,6 +789,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, s->bat =NULL ; + s->first_visible_write = true;qemu_co_mutex_init (&s->lock);
@@ -864,7 +870,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, }if (flags &BDRV_O_RDWR ) { - ret =vhdx_update_headers (bs, s, false); + ret =vhdx_update_headers (bs, s, false,NULL );if (ret <0 ) { goto fail; }
@@ -1010,6 +1016,18 @@ exit: +/* Per the spec, on the first write of guest-visible data to the file the +* data write guid must be updated in the header */ +int vhdx_user_visible_write (BlockDriverState *bs,BDRVVHDXState *s) +{ +int ret =0 ; +if (s->first_visible_write) { + s->first_visible_write = false; + ret =vhdx_update_headers (bs, s, true,NULL ); + } +return ret; +} + static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num,int nb_sectors,QEMUIOVector *qiov) {
block/vhdx.h +6 -0
@@ -361,6 +361,7 @@ typedef struct BDRVVHDXState {VHDXBatEntry *bat;uint64_t bat_offset; +bool first_visible_write;MSGUID session_guid;VHDXLogEntries log;
@@ -373,6 +374,9 @@ typedef struct BDRVVHDXState {void vhdx_guid_generate (MSGUID *guid); +int vhdx_update_headers (BlockDriverState *bs,BDRVVHDXState *s,bool rw, +MSGUID *log_guid); +uint32_t vhdx_update_checksum (uint8_t *buf,size_t size,int crc_offset);uint32_t vhdx_checksum_calc (uint32_t crc,uint8_t *buf,size_t size,int crc_offset);
@@ -402,4 +406,6 @@ void vhdx_log_data_le_export(VHDXLogDataSector *d);void vhdx_log_entry_hdr_le_import (VHDXLogEntryHeader *hdr);void vhdx_log_entry_hdr_le_export (VHDXLogEntryHeader *hdr); +int vhdx_user_visible_write (BlockDriverState *bs,BDRVVHDXState *s); +#endif
Signature · 1 line
1.8.3.1
PATCH v7 7/19 block: vhdx code movement - move vhdx_close() above vhdx_open()
In reply to Jeff Cody--- block/vhdx.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
block/vhdx.c +11 -11
@@ -778,6 +778,17 @@ exit: } +static void vhdx_close (BlockDriverState *bs) +{ +BDRVVHDXState *s = bs->opaque; +qemu_vfree (s->headers[0 ]); +qemu_vfree (s->headers[1 ]); +qemu_vfree (s->bat); +qemu_vfree (s->parent_entries); +migrate_del_blocker (s->migration_blocker); +error_free (s->migration_blocker); +} +static int vhdx_open (BlockDriverState *bs,QDict *options,int flags,Error **errp) {
@@ -1035,17 +1046,6 @@ static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num, } -static void vhdx_close (BlockDriverState *bs) -{ -BDRVVHDXState *s = bs->opaque; -qemu_vfree (s->headers[0 ]); -qemu_vfree (s->headers[1 ]); -qemu_vfree (s->bat); -qemu_vfree (s->parent_entries); -migrate_del_blocker (s->migration_blocker); -error_free (s->migration_blocker); -} -static BlockDriver bdrv_vhdx = { .format_name ="vhdx" , .instance_size =sizeof (BDRVVHDXState),
Signature · 1 line
1.8.3.1
PATCH v7 8/19 block: vhdx - log parsing, replay, and flush support
In reply to Jeff CodyThis adds support for VHDX v0 logs, as specified in Microsoft's VHDX Specification Format v1.00:
https://www.microsoft.com/en-us/download/details.aspx?id=34750
The following support is added:
* Log parsing, and validation - validate that an existing log is correct.
* Log search - search through an existing log, to find any valid sequence of entries.
* Log replay and flush - replay an existing log, and flush/clear the log when complete.
The VHDX log is a circular buffer, with elements (sectors) of 4KB.
A log entry is a variably-length number of sectors, that is comprised of a header and 'descriptors', that describe each sector.
A log may contain multiple entries, know as a log sequence. In a log sequence, each log entry immediately follows the previous entry, with an incrementing sequence number. There can only ever be one active and valid sequence in the log.
Each log entry must match the file log GUID in order to be valid (along with other criteria). Once we have flushed all valid log entries, we marked the file log GUID to be zero, which indicates a buffer with no valid entries.
--- block/Makefile.objs | 2 +- block/vhdx-log.c | 728 ++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.c | 65 +---- block/vhdx.h | 7 +- 4 files changed, 749 insertions(+), 53 deletions(-) create mode 100644 block/vhdx-log.c
block/Makefile.objs +1 -1
@@ -2,7 +2,7 @@ block-obj-y += raw_bsd.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o v block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o -block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o +block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o block-obj-y += parallels.o blkdebug.o blkverify.o block-obj-y += snapshot.o qapi.o block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
block/vhdx-log.c new file +728 -0
728 lines not on this page · raw message
block/vhdx.c +14 -51
112 lines not on this page · raw message
block/vhdx.h +6 -1
@@ -326,7 +326,11 @@ typedef struct VHDXMetadataEntries {typedef struct VHDXLogEntries {uint64_t offset;uint64_t length; -uint32_t head; +uint32_t write; +uint32_t read; +VHDXLogEntryHeader *hdr; +void *desc_buffer; +uint64_t sequence;uint32_t tail; }VHDXLogEntries ;
@@ -383,6 +387,7 @@ uint32_t vhdx_checksum_calc(uint32_t crc, uint8_t *buf, size_t size,bool vhdx_checksum_is_valid (uint8_t *buf,size_t size,int crc_offset); +int vhdx_parse_log (BlockDriverState *bs,BDRVVHDXState *s,bool *flushed);static inline void leguid_to_cpus (MSGUID *guid) {
Signature · 1 line
1.8.3.1
PATCH v7 9/19 block: vhdx - add region overlap detection for image files
In reply to Jeff CodyRegions in the image file cannot overlap - the log, region tables, and metdata must all be unique and non-overlapping.
This adds region checking by means of a QLIST; there can be a variable number of regions and metadata (there may be metadata or region tables that we do not recognize / know about, but are not required).
This adds the capability to register a region for later checking, and to check against registered regions for any overlap.
Also, if neither the BAT or Metadata region tables are found, return error.
--- block/vhdx.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.h | 9 +++++++ 2 files changed, 91 insertions(+)
block/vhdx.c +82 -0
@@ -204,6 +204,50 @@ void vhdx_guid_generate(MSGUID *guid)memcpy (guid, uuid,sizeof (MSGUID )); } +/* Check for region overlaps inside the VHDX image */ +static int vhdx_region_check (BDRVVHDXState *s,uint64_t start,uint64_t length) +{ +int ret =0 ; +uint64_t end; +VHDXRegionEntry *r; + + end = start + length; + QLIST_FOREACH(r, &s->regions, entries) { +if (!((start >= r->end) || (end <= r->start))) { + ret = -EINVAL ; + goto exit; + } + } + +exit: +return ret; +} + +/* Register a region for future checks */ +static void vhdx_region_register (BDRVVHDXState *s, +uint64_t start,uint64_t length) +{ +VHDXRegionEntry *r; + + r =g_malloc0 (sizeof (*r)); + + r->start = start; + r->end = start + length; + +QLIST_INSERT_HEAD (&s->regions, r, entries); +} + +/* Free all registered regions */ +static void vhdx_region_unregister_all (BDRVVHDXState *s) +{ +VHDXRegionEntry *r, *r_next; + + QLIST_FOREACH_SAFE(r, &s->regions, entries, r_next) { +QLIST_REMOVE (r, entries); +g_free (r); + } +} +/* * Per the MS VHDX Specification, for every VHDX file: * - The header section is fixed size - 1 MB
@@ -389,6 +433,9 @@ static int vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s) } } +vhdx_region_register (s, s->headers[s->curr_header]->log_offset, + s->headers[s->curr_header]->log_length); + ret =0 ; goto exit;
@@ -452,6 +499,15 @@ static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)le32_to_cpus (&rt_entry.length);le32_to_cpus (&rt_entry.data_bits); +/* check for region overlap between these entries, and any +* other memory regions in the file */ + ret =vhdx_region_check (s, rt_entry.file_offset, rt_entry.length); +if (ret <0 ) { + goto fail; + } + +vhdx_region_register (s, rt_entry.file_offset, rt_entry.length); +/* see if we recognize the entry */ if (guid_eq (rt_entry.guid, bat_guid)) {/* must be unique; if we have already found it this is invalid */
@@ -482,6 +538,12 @@ static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s) goto fail; } } + +if (!bat_rt_found || !metadata_rt_found) { + ret = -EINVAL ; + goto fail; + } + ret =0 ; fail:
@@ -751,6 +813,7 @@ static void vhdx_close(BlockDriverState *bs)error_free (s->migration_blocker);qemu_vfree (s->log.hdr); s->log.hdr =NULL ; +vhdx_region_unregister_all (s); } static int vhdx_open(BlockDriverState *bs, QDict *options, int flags,
@@ -768,6 +831,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, s->first_visible_write = true;qemu_co_mutex_init (&s->lock); +QLIST_INIT (&s->regions);/* validate the file signature */ ret =bdrv_pread (bs->file,0 , &signature,sizeof (uint64_t ));
@@ -842,8 +906,26 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } +uint64_t payblocks = s->chunk_ratio; +/* endian convert, and verify populated BAT field file offsets against +* region table and log entries */ for (i =0 ; i < s->bat_entries; i++) {le64_to_cpus (&s->bat[i]); +if (payblocks--) { +/* payload bat entries */ +if ((s->bat[i] &VHDX_BAT_STATE_BIT_MASK ) == +PAYLOAD_BLOCK_FULL_PRESENT ) { + ret =vhdx_region_check (s, s->bat[i] &VHDX_BAT_FILE_OFF_MASK , + s->block_size); +if (ret <0 ) { + goto fail; + } + } + }else { + payblocks = s->chunk_ratio; +/* Once differencing files are supported, verify sector bitmap +* blocks here */ + } }if (flags &BDRV_O_RDWR ) {
block/vhdx.h +9 -0
@@ -230,6 +230,7 @@ typedef struct QEMU_PACKED VHDXLogDataSector {other bits are reserved */ #define VHDX_BAT_STATE_BIT_MASK 0x07#define VHDX_BAT_FILE_OFF_BITS (64 - 44) +#define VHDX_BAT_FILE_OFF_MASK 0xFFFFFFFFFFF00000/* upper 44 bits */ typedef uint64_t VHDXBatEntry ;/* ---- METADATA REGION STRUCTURES ---- */
@@ -334,6 +335,12 @@ typedef struct VHDXLogEntries {uint32_t tail; } VHDXLogEntries; +typedef struct VHDXRegionEntry { +uint64_t start; +uint64_t end; +QLIST_ENTRY (VHDXRegionEntry) entries; +}VHDXRegionEntry ; + typedef struct BDRVVHDXState { CoMutex lock;
@@ -374,6 +381,8 @@ typedef struct BDRVVHDXState {VHDXParentLocatorEntry *parent_entries;Error *migration_blocker; + + QLIST_HEAD(VHDXRegionHead, VHDXRegionEntry) regions; } BDRVVHDXState;void vhdx_guid_generate (MSGUID *guid);
Signature · 1 line
1.8.3.1
PATCH v7 10/19 block: vhdx - add log write support
In reply to Jeff CodyThis adds support for writing to the VHDX log.
For spec details, see VHDX Specification Format v1.00:
https://www.microsoft.com/en-us/download/details.aspx?id=34750
There are a few limitations to this log support:
1.) There is no caching yet
2.) The log is flushed after each entry
The primary write interface, vhdx_log_write_and_flush(), performs a log write followed by an immediate flush of the log.
As each log entry sector is a minimum of 4KB, partial sector writes are filled in with data from the disk write destination.
If the current file log GUID is 0, a new GUID is generated and updated in the header.
--- block/vhdx-log.c | 279 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.h | 3 + 2 files changed, 282 insertions(+)
block/vhdx-log.c +279 -0
@@ -156,6 +156,55 @@ exit:return ret; } +/* Writes num_sectors to the log (all log sectors are 4096 bytes), +* from buffer 'buffer'. Upon return, *sectors_written will contain +* the number of sectors successfully written. +* +* It is assumed that 'buffer' is at least 4096*num_sectors large. +* +* 0 is returned on success, -errno otherwise */ +static int vhdx_log_write_sectors (BlockDriverState *bs,VHDXLogEntries *log, +uint32_t *sectors_written,void *buffer, +uint32_t num_sectors) +{ +int ret =0 ; +uint64_t offset; +uint32_t write; +void *buffer_tmp; +BDRVVHDXState *s = bs->opaque; + + ret =vhdx_user_visible_write (bs, s); +if (ret <0 ) { + goto exit; + } + + write = log->write; + + buffer_tmp = buffer; +while (num_sectors) { + + offset = log->offset + write; + write =vhdx_log_inc_idx (write, log->length); +if (write == log->read) { +/* full */ +break ; + } + ret =bdrv_pwrite (bs->file, offset, buffer_tmp,VHDX_LOG_SECTOR_SIZE ); +if (ret <0 ) { + goto exit; + } + buffer_tmp +=VHDX_LOG_SECTOR_SIZE ; + + log->write = write; + *sectors_written = *sectors_written +1 ; + num_sectors--; + } + +exit: +return ret; +} + +/* Validates a log entry header */ static bool vhdx_log_hdr_is_valid (VHDXLogEntries *log,VHDXLogEntryHeader *hdr,BDRVVHDXState *s)
@@ -726,3 +775,233 @@ exit: } + +static void vhdx_log_raw_to_le_sector (VHDXLogDescriptor *desc, +VHDXLogDataSector *sector,void *data, +uint64_t seq) +{ +/* 8 + 4084 + 4 = 4096, 1 log sector */ +memcpy (&desc->leading_bytes, data,8 ); + data +=8 ; +cpu_to_le64s (&desc->leading_bytes); +memcpy (sector->data, data,4084 ); + data +=4084 ; +memcpy (&desc->trailing_bytes, data,4 ); +cpu_to_le32s (&desc->trailing_bytes); + data +=4 ; + + sector->sequence_high = (uint32_t ) (seq >>32 ); + sector->sequence_low = (uint32_t ) (seq &0xffffffff ); + sector->data_signature =VHDX_LOG_DATA_SIGNATURE ; + +vhdx_log_desc_le_export (desc); +vhdx_log_data_le_export (sector); +} + + +static int vhdx_log_write (BlockDriverState *bs,BDRVVHDXState *s, +void *data,uint32_t length,uint64_t offset) +{ +int ret =0 ; +void *buffer =NULL ; +void *merged_sector =NULL ; +void *data_tmp, *sector_write; +unsigned int i; +int sector_offset; +uint32_t desc_sectors, sectors, total_length; +uint32_t sectors_written =0 ; +uint32_t aligned_length; +uint32_t leading_length =0 ; +uint32_t trailing_length =0 ; +uint32_t partial_sectors =0 ; +uint32_t bytes_written =0 ; +uint64_t file_offset; +VHDXHeader *header; +VHDXLogEntryHeader new_hdr; +VHDXLogDescriptor *new_desc =NULL ; +VHDXLogDataSector *data_sector =NULL ; +MSGUID new_guid = {0 }; + + header = s->headers[s->curr_header]; + +/* need to have offset read data, and be on 4096 byte boundary */ + +if (length > header->log_length) { +/* no log present. we could create a log here instead of failing */ + ret = -EINVAL ; + goto exit; + } + +if (guid_eq (header->log_guid, zero_guid)) { +vhdx_guid_generate (&new_guid); +vhdx_update_headers (bs, s, false, &new_guid); + }else { +/* currently, we require that the log be flushed after +* every write. */ + ret = -ENOTSUP ; + goto exit; + } + +/* 0 is an invalid sequence number, but may also represent the first +* log write (or a wrapped seq) */ +if (s->log.sequence ==0 ) { + s->log.sequence =1 ; + } + + sector_offset = offset %VHDX_LOG_SECTOR_SIZE ; + file_offset = (offset /VHDX_LOG_SECTOR_SIZE ) *VHDX_LOG_SECTOR_SIZE ; + + aligned_length = length; + +/* add in the unaligned head and tail bytes */ +if (sector_offset) { + leading_length = (VHDX_LOG_SECTOR_SIZE - sector_offset); + leading_length = leading_length > length ? length : leading_length; + aligned_length -= leading_length; + partial_sectors++; + } + + sectors = aligned_length /VHDX_LOG_SECTOR_SIZE ; + trailing_length = aligned_length - (sectors *VHDX_LOG_SECTOR_SIZE ); +if (trailing_length) { + partial_sectors++; + } + + sectors += partial_sectors; + +/* sectors is now how many sectors the data itself takes, not +* including the header and descriptor metadata */ + + new_hdr = (VHDXLogEntryHeader ) { + .signature =VHDX_LOG_SIGNATURE , + .tail = s->log.tail, + .sequence_number = s->log.sequence, + .descriptor_count = sectors, + .reserved =0 , + .flushed_file_offset =bdrv_getlength (bs->file), + .last_file_offset =bdrv_getlength (bs->file), + }; + + new_hdr.log_guid = header->log_guid; + + desc_sectors =vhdx_compute_desc_sectors (new_hdr.descriptor_count); + + total_length = (desc_sectors + sectors) *VHDX_LOG_SECTOR_SIZE ; + new_hdr.entry_length = total_length; + +vhdx_log_entry_hdr_le_export (&new_hdr); + + buffer =qemu_blockalign (bs, total_length); +memcpy (buffer, &new_hdr,sizeof (new_hdr)); + + new_desc = (VHDXLogDescriptor *) (buffer +sizeof (new_hdr)); + data_sector = buffer + (desc_sectors *VHDX_LOG_SECTOR_SIZE ); + data_tmp = data; + +/* All log sectors are 4KB, so for any partial sectors we must +* merge the data with preexisting data from the final file +* destination */ + merged_sector =qemu_blockalign (bs,VHDX_LOG_SECTOR_SIZE ); + +for (i =0 ; i < sectors; i++) { + new_desc->signature =VHDX_LOG_DESC_SIGNATURE ; + new_desc->sequence_number = s->log.sequence; + new_desc->file_offset = file_offset; + +if (i ==0 && leading_length) { +/* partial sector at the front of the buffer */ + ret =bdrv_pread (bs->file, file_offset, merged_sector, +VHDX_LOG_SECTOR_SIZE ); +if (ret <0 ) { + goto exit; + } +memcpy (merged_sector + sector_offset, data_tmp, leading_length); + bytes_written = leading_length; + sector_write = merged_sector; + }else if (i == sectors -1 && trailing_length) { +/* partial sector at the end of the buffer */ + ret =bdrv_pread (bs->file, + file_offset, + merged_sector + trailing_length, +VHDX_LOG_SECTOR_SIZE - trailing_length); +if (ret <0 ) { + goto exit; + } +memcpy (merged_sector, data_tmp, trailing_length); + bytes_written = trailing_length; + sector_write = merged_sector; + }else { + bytes_written =VHDX_LOG_SECTOR_SIZE ; + sector_write = data_tmp; + } + +/* populate the raw sector data into the proper structures, +* as well as update the descriptor, and convert to proper +* endianness */ +vhdx_log_raw_to_le_sector (new_desc, data_sector, sector_write, + s->log.sequence); + + data_tmp += bytes_written; + data_sector++; + new_desc++; + file_offset +=VHDX_LOG_SECTOR_SIZE ; + } + +/* checksum covers entire entry, from the log header through the +* last data sector */ +vhdx_update_checksum (buffer, total_length, + offsetof(VHDXLogEntryHeader , checksum)); +cpu_to_le32s ((uint32_t *)(buffer +4 )); + +/* now write to the log */ +vhdx_log_write_sectors (bs, &s->log, §ors_written, buffer, + desc_sectors + sectors); +if (ret <0 ) { + goto exit; + } + +if (sectors_written != desc_sectors + sectors) { +/* instead of failing, we could flush the log here */ + ret = -EINVAL ; + goto exit; + } + + s->log.sequence++; +/* write new tail */ + s->log.tail = s->log.write; + +exit: +qemu_vfree (buffer); +qemu_vfree (merged_sector); +return ret; +} + +/* Perform a log write, and then immediately flush the entire log */ +int vhdx_log_write_and_flush (BlockDriverState *bs,BDRVVHDXState *s, +void *data,uint32_t length,uint64_t offset) +{ +int ret =0 ; +VHDXLogSequence logs = { .valid = true, + .count =1 , + .hdr = {0 } }; + + +bdrv_flush (bs); + ret =vhdx_log_write (bs, s, data, length, offset); +if (ret <0 ) { + goto exit; + } + logs.log = s->log; + +bdrv_flush (bs); + ret =vhdx_log_flush (bs, s, &logs); +if (ret <0 ) { + goto exit; + } + + s->log = logs.log; + +exit: +return ret; +} +
block/vhdx.h +3 -0
@@ -398,6 +398,9 @@ bool vhdx_checksum_is_valid(uint8_t *buf, size_t size, int crc_offset);int vhdx_parse_log (BlockDriverState *bs,BDRVVHDXState *s,bool *flushed); +int vhdx_log_write_and_flush (BlockDriverState *bs,BDRVVHDXState *s, +void *data,uint32_t length,uint64_t offset); +static inline void leguid_to_cpus (MSGUID *guid) {le32_to_cpus (&guid->data1);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v7 10/19] block: vhdx - add log write support
block/vhdx-log.c exit: · line 989
Quoted 11 lines · Jeff Cody
block/vhdx-log.c +11
-0+/* Perform a log write, and then immediately flush the entire log */ +int vhdx_log_write_and_flush (BlockDriverState *bs,BDRVVHDXState *s, +void *data,uint32_t length,uint64_t offset) +{ +int ret =0 ; +VHDXLogSequence logs = { .valid = true, + .count =1 , + .hdr = {0 } }; + + +bdrv_flush (bs);
It's useful to comment why a flush is necessary. In this case I think you're flushing the disk write cache because you want newly allocated block contents to be stable before writing the BAT update into the log.
PATCH v7 11/19 block: vhdx write support
In reply to Jeff CodyThis adds support for writing to VHDX image files, using coroutines. Writes into the BAT table goes through the VHDX log. Currently, BAT table writes occur when expanding a dynamic VHDX file, and allocating a new BAT entry.
--- block/vhdx.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- block/vhdx.h | 2 +- 2 files changed, 193 insertions(+), 5 deletions(-)
block/vhdx.c +192 -4
@@ -914,7 +914,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags,if (payblocks--) {/* payload bat entries */ if ((s->bat[i] &VHDX_BAT_STATE_BIT_MASK ) == -PAYLOAD_BLOCK_FULL_PRESENT ) { +PAYLOAD_BLOCK_FULLY_PRESENT ) { ret =vhdx_region_check (s, s->bat[i] &VHDX_BAT_FILE_OFF_MASK , s->block_size);if (ret <0 ) {
@@ -935,7 +935,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags, } } -/* TODO: differencing files, write */ +/* TODO: differencing files */ /* Disable migration when VHDX images are used */ error_set(&s->migration_blocker,
@@ -1040,7 +1040,7 @@ static coroutine_fn int vhdx_co_readv(BlockDriverState *bs, int64_t sector_num,/* return zero */ qemu_iovec_memset (&hd_qiov,0 ,0 , sinfo.bytes_avail);break ; -case PAYLOAD_BLOCK_FULL_PRESENT : +case PAYLOAD_BLOCK_FULLY_PRESENT :qemu_co_mutex_unlock (&s->lock); ret = bdrv_co_readv(bs->file, sinfo.file_offset >> BDRV_SECTOR_BITS,
@@ -1070,7 +1070,43 @@ exit:return ret; } +/* +* Allocate a new payload block at the end of the file. +* +* Allocation will happen at 1MB alignment inside the file +* +* Returns the file offset start of the new payload block +*/ +static int vhdx_allocate_block (BlockDriverState *bs,BDRVVHDXState *s, +uint64_t *new_offset) +{ + *new_offset =bdrv_getlength (bs->file); + +/* per the spec, the address for a block is in units of 1MB */ + *new_offset =ROUND_UP (*new_offset,1024 *1024 ); + +return bdrv_truncate (bs->file, *new_offset + s->block_size); +} + +/* +* Update the BAT table entry with the new file offset, and the new entry +* state */ +static void vhdx_update_bat_table_entry (BlockDriverState *bs,BDRVVHDXState *s, +VHDXSectorInfo *sinfo, +uint64_t *bat_entry_le, +uint64_t *bat_offset,int state) +{ +/* The BAT entry is a uint64, with 44 bits for the file offset in units of +* 1MB, and 3 bits for the block state. */ + s->bat[sinfo->bat_idx] = ((sinfo->file_offset>>20 ) << +VHDX_BAT_FILE_OFF_BITS ); + s->bat[sinfo->bat_idx] |= state &VHDX_BAT_STATE_BIT_MASK ; + + *bat_entry_le =cpu_to_le64 (s->bat[sinfo->bat_idx]); + *bat_offset = s->bat_offset + sinfo->bat_idx *sizeof (VHDXBatEntry); + +}/* Per the spec, on the first write of guest-visible data to the file the * data write guid must be updated in the header */
@@ -1087,7 +1123,159 @@ int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num,int nb_sectors,QEMUIOVector *qiov) { -return -ENOTSUP ; +int ret = -ENOTSUP ; +BDRVVHDXState *s = bs->opaque; +VHDXSectorInfo sinfo; +uint64_t bytes_done =0 ; +uint64_t bat_entry =0 ; +uint64_t bat_entry_offset =0 ; +QEMUIOVector hd_qiov; +struct iovec iov1 = {0 }; +struct iovec iov2 = {0 }; +int sectors_to_write; + +qemu_iovec_init (&hd_qiov, qiov->niov); + +qemu_co_mutex_lock (&s->lock); + + ret =vhdx_user_visible_write (bs, s); +if (ret <0 ) { + goto exit; + } + +while (nb_sectors >0 ) { +bool use_zero_buffers = false; +bool bat_update = false; +if (s->params.data_bits &VHDX_PARAMS_HAS_PARENT ) { +/* not supported yet */ + ret = -ENOTSUP ; + goto exit; + }else { +vhdx_block_translate (s, sector_num, nb_sectors, &sinfo); + sectors_to_write = sinfo.sectors_avail; + +qemu_iovec_reset (&hd_qiov); +/* check the payload block state */ +switch (s->bat[sinfo.bat_idx] &VHDX_BAT_STATE_BIT_MASK ) { +case PAYLOAD_BLOCK_ZERO : +/* in this case, we need to preserve zero writes for +* data that is not part of this write, so we must pad +* the rest of the buffer to zeroes */ + +/* if we are on a posix system with ftruncate() that extends +* a file, then it is zero-filled for us. On Win32, the raw +* layer uses SetFilePointer and SetFileEnd, which does not +* zero fill AFAIK */ + +/* Queue another write of zero buffers if the underlying file +* does not zero-fill on file extension */ + +if (bdrv_has_zero_init (bs->file) ==0 ) { + use_zero_buffers = true; + +/* zero fill the front, if any */ +if (sinfo.block_offset) { + iov1.iov_len = sinfo.block_offset; + iov1.iov_base =qemu_blockalign (bs, iov1.iov_len); +memset (iov1.iov_base,0 , iov1.iov_len); +qemu_iovec_concat_iov (&hd_qiov, &iov1,1 ,0 , + sinfo.block_offset); + sectors_to_write += iov1.iov_len >>BDRV_SECTOR_BITS ; + } + +/* our actual data */ +qemu_iovec_concat (&hd_qiov, qiov, bytes_done, + sinfo.bytes_avail); + +/* zero fill the back, if any */ +if ((sinfo.bytes_avail - sinfo.block_offset) < + s->block_size) { + iov2.iov_len = s->block_size - + (sinfo.bytes_avail + sinfo.block_offset); + iov2.iov_base =qemu_blockalign (bs, iov2.iov_len); +memset (iov2.iov_base,0 , iov2.iov_len); +qemu_iovec_concat_iov (&hd_qiov, &iov2,1 ,0 , + sinfo.block_offset); + sectors_to_write += iov2.iov_len >>BDRV_SECTOR_BITS ; + } + } + +/* fall through */ +case PAYLOAD_BLOCK_NOT_PRESENT :/* fall through */ +case PAYLOAD_BLOCK_UNMAPPED :/* fall through */ +case PAYLOAD_BLOCK_UNDEFINED :/* fall through */ + ret =vhdx_allocate_block (bs, s, &sinfo.file_offset); +if (ret <0 ) { + goto exit; + } +/* once we support differencing files, this may also be +* partially present */ +/* update block state to the newly specified state */ +vhdx_update_bat_table_entry (bs, s, &sinfo, &bat_entry, + &bat_entry_offset, +PAYLOAD_BLOCK_FULLY_PRESENT ); + bat_update = true; +/* since we just allocated a block, file_offset is the +* beginning of the payload block. It needs to be the +* write address, which includes the offset into the block */ +if (!use_zero_buffers) { + sinfo.file_offset += sinfo.block_offset; + } +/* fall through */ +case PAYLOAD_BLOCK_FULLY_PRESENT : +/* if the file offset address is in the header zone, +* there is a problem */ +if (sinfo.file_offset < (1024 *1024 )) { + ret = -EFAULT ; + goto exit; + } + +if (!use_zero_buffers) { +qemu_iovec_concat (&hd_qiov, qiov, bytes_done, + sinfo.bytes_avail); + } +/* block exists, so we can just overwrite it */ +qemu_co_mutex_unlock (&s->lock); + ret =bdrv_co_writev (bs->file, + sinfo.file_offset >>BDRV_SECTOR_BITS , + sectors_to_write, &hd_qiov); +qemu_co_mutex_lock (&s->lock); +if (ret <0 ) { + goto exit; + } +break ; +case PAYLOAD_BLOCK_PARTIALLY_PRESENT : +/* we don't yet support difference files, fall through +* to error */ +default : + ret = -EIO ; + goto exit; +break ; + } + +if (bat_update) { +/* this will update the BAT entry into the log journal, and +* then flush the log journal out to disk */ + ret =vhdx_log_write_and_flush (bs, s, &bat_entry, +sizeof (VHDXBatEntry), + bat_entry_offset); +if (ret <0 ) { + goto exit; + } + } + + nb_sectors -= sinfo.sectors_avail; + sector_num += sinfo.sectors_avail; + bytes_done += sinfo.bytes_avail; + + } + } + +exit: +qemu_vfree (iov1.iov_base); +qemu_vfree (iov2.iov_base); +qemu_co_mutex_unlock (&s->lock); +return ret; }
block/vhdx.h +1 -1
@@ -217,7 +217,7 @@ typedef struct QEMU_PACKED VHDXLogDataSector {#define PAYLOAD_BLOCK_UNDEFINED 1#define PAYLOAD_BLOCK_ZERO 2#define PAYLOAD_BLOCK_UNMAPPED 5 -#define PAYLOAD_BLOCK_FULL_PRESENT 6 +#define PAYLOAD_BLOCK_FULLY_PRESENT 6#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7#define SB_BLOCK_NOT_PRESENT 0
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v7 11/19] block: vhdx write support
Quoted 6 lines · Jeff Cody
block/vhdx.c +6
-0+/* check the payload block state */ +switch (s->bat[sinfo.bat_idx] &VHDX_BAT_STATE_BIT_MASK ) { +case PAYLOAD_BLOCK_ZERO : +/* in this case, we need to preserve zero writes for +* data that is not part of this write, so we must pad +* the rest of the buffer to zeroes */
block/vhdx.c int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) · line 1173
Quoted 10 lines
block/vhdx.c +10
-0+ +/* if we are on a posix system with ftruncate() that extends +* a file, then it is zero-filled for us. On Win32, the raw +* layer uses SetFilePointer and SetFileEnd, which does not +* zero fill AFAIK */ + +/* Queue another write of zero buffers if the underlying file +* does not zero-fill on file extension */ + +if (bdrv_has_zero_init (bs->file) ==0 ) {
Maybe this should be done differently once Peter Lieven's thin provisioning patches are merged. I think bdrv_has_zero_init() only tells you whether the file is zeroed after creation. It doesn't guarantee zeroes for other operations like growing the file.
I'm not 100% sure right now, will follow-up on this point later when I've reviewed Peter's series and thought more about it.
block/vhdx.h typedef struct QEMU_PACKED VHDXLogDataSector { · line 221
Quoted 11 lines
block/vhdx.h +1
-1@@ -217,7 +217,7 @@ typedef struct QEMU_PACKED VHDXLogDataSector {#define PAYLOAD_BLOCK_UNDEFINED 1#define PAYLOAD_BLOCK_ZERO 2#define PAYLOAD_BLOCK_UNMAPPED 5 -#define PAYLOAD_BLOCK_FULL_PRESENT 6 +#define PAYLOAD_BLOCK_FULLY_PRESENT 6#define PAYLOAD_BLOCK_PARTIALLY_PRESENT 7
This change should be in a separate patch.
Re: [Qemu-devel] [PATCH v7 11/19] block: vhdx write support
Quoted 23 lines · Stefan Hajnoczi
Quoted 6 lines · Jeff Cody
block/vhdx.c +6
-0+/* check the payload block state */ +switch (s->bat[sinfo.bat_idx] &VHDX_BAT_STATE_BIT_MASK ) { +case PAYLOAD_BLOCK_ZERO : +/* in this case, we need to preserve zero writes for +* data that is not part of this write, so we must pad +* the rest of the buffer to zeroes */ block/vhdx.c int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) · line 1173
Quoted 10 lines
block/vhdx.c +10
-0+ +/* if we are on a posix system with ftruncate() that extends +* a file, then it is zero-filled for us. On Win32, the raw +* layer uses SetFilePointer and SetFileEnd, which does not +* zero fill AFAIK */ + +/* Queue another write of zero buffers if the underlying file +* does not zero-fill on file extension */ + +if (bdrv_has_zero_init (bs->file) ==0 ) {Maybe this should be done differently once Peter Lieven's thin provisioning patches are merged. I think
bdrv_has_zero_init()only tells you whether the file is zeroed after creation. It doesn't guarantee zeroes for other operations like growing the file.
Quoted 2 lines
I'm not 100% sure right now, will follow-up on this point later when I've reviewed Peter's series and thought more about it.
With Peter's thin provisioning patches this call would be changed to bdrv_has_discard_zeroes(). We're discussing changing the name of that function though.
Stefan
Re: [Qemu-devel] [PATCH v7 11/19] block: vhdx write support
In reply to Jeff Codyblock/vhdx.c int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) · line 1228
Quoted 4 lines · Jeff Cody
block/vhdx.c +4
-0+case PAYLOAD_BLOCK_FULLY_PRESENT : +/* if the file offset address is in the header zone, +* there is a problem */ +if (sinfo.file_offset < (1024 *1024 )) {
When a block is allocated we update s->bat[] and changed the file size. This should be undone so that in-memory and on-disk metadata doesn't get out of sync.
Quoted 6 lines
block/vhdx.c +6
-0+ ret = -EFAULT ; + goto exit; + } + +if (!use_zero_buffers) { + qemu_iovec_concat(&hd_qiov, qiov, bytes_done,
block/vhdx.c int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) · line 1244
Quoted 10 lines
block/vhdx.c +10
-0+ sinfo.bytes_avail); + } +/* block exists, so we can just overwrite it */ +qemu_co_mutex_unlock (&s->lock); + ret =bdrv_co_writev (bs->file, + sinfo.file_offset >>BDRV_SECTOR_BITS , + sectors_to_write, &hd_qiov); +qemu_co_mutex_lock (&s->lock); +if (ret <0 ) { + goto exit;
Same comment about in-memory and on-disk metadata getting out of sync in the error case.
block/vhdx.c int vhdx_user_visible_write(BlockDriverState *bs, BDRVVHDXState *s) · line 1278
Quoted 5 lines
block/vhdx.c +5
-0+exit: +qemu_vfree (iov1.iov_base); +qemu_vfree (iov2.iov_base); +qemu_co_mutex_unlock (&s->lock); +return ret;
Missing qemu_iovec_destroy(&hd_qiov).
PATCH v7 12/19 block: vhdx - remove BAT file offset bit shifting
In reply to Jeff CodyBit shifting can be fun, but in this case it was unnecessary. The upper 44 bits of the 64-bit BAT entry is specifies the File Offset, so we shifted the bits to get access to the value.
However, per the spec the value is in MB. So we dutifully shifted back to the left by 20 bits, to convert to a true uint64_t file offset.
This replaces those steps with just a bit mask, to get rid of the lower 20 bits instead.
--- block/vhdx.c | 6 ++---- block/vhdx.h | 1 - 2 files changed, 2 insertions(+), 5 deletions(-)
block/vhdx.c +2 -4
@@ -985,7 +985,7 @@ static void vhdx_block_translate(BDRVVHDXState *s, int64_t sector_num, sinfo->bytes_avail = sinfo->sectors_avail << s->logical_sector_size_bits; - sinfo->file_offset = s->bat[sinfo->bat_idx] >>VHDX_BAT_FILE_OFF_BITS ; + sinfo->file_offset = s->bat[sinfo->bat_idx] &VHDX_BAT_FILE_OFF_MASK ; sinfo->block_offset = block_offset << s->logical_sector_size_bits;
@@ -999,7 +999,6 @@ static void vhdx_block_translate(BDRVVHDXState *s, int64_t sector_num,* in the block, and add in the payload data block offset * in the file, in bytes, to get the final read address */ - sinfo->file_offset <<=20 ;/* now in bytes, rather than 1MB units */ sinfo->file_offset += sinfo->block_offset; }
@@ -1098,8 +1097,7 @@ static void vhdx_update_bat_table_entry(BlockDriverState *bs, BDRVVHDXState *s, {/* The BAT entry is a uint64, with 44 bits for the file offset in units of * 1MB, and 3 bits for the block state. */ - s->bat[sinfo->bat_idx] = ((sinfo->file_offset>>20 ) << -VHDX_BAT_FILE_OFF_BITS ); + s->bat[sinfo->bat_idx] = sinfo->file_offset; s->bat[sinfo->bat_idx] |= state &VHDX_BAT_STATE_BIT_MASK ;
block/vhdx.h +0 -1
@@ -229,7 +229,6 @@ typedef struct QEMU_PACKED VHDXLogDataSector {/* upper 44 bits are the file offset in 1MB units lower 3 bits are the state other bits are reserved */ #define VHDX_BAT_STATE_BIT_MASK 0x07 -#define VHDX_BAT_FILE_OFF_BITS (64 - 44)#define VHDX_BAT_FILE_OFF_MASK 0xFFFFFFFFFFF00000/* upper 44 bits */ typedef uint64_t VHDXBatEntry ;
Signature · 1 line
1.8.3.1
PATCH v7 13/19 block: vhdx - move more endian translations to vhdx-endian.c
In reply to Jeff CodyIn preparation for vhdx_create(), move more endian translation functions out to vhdx-endian.c.
--- block/vhdx-endian.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.c | 20 +++----------- block/vhdx.h | 9 ++++++- 3 files changed, 87 insertions(+), 17 deletions(-)
block/vhdx-endian.c +75 -0
@@ -139,3 +139,78 @@ void vhdx_log_entry_hdr_le_export(VHDXLogEntryHeader *hdr) } +/* Region table entries */ +void vhdx_region_header_le_import (VHDXRegionTableHeader *hdr) +{ +assert (hdr !=NULL ); + +le32_to_cpus (&hdr->signature); +le32_to_cpus (&hdr->checksum); +le32_to_cpus (&hdr->entry_count); +} + +void vhdx_region_header_le_export (VHDXRegionTableHeader *hdr) +{ +assert (hdr !=NULL ); + +cpu_to_le32s (&hdr->signature); +cpu_to_le32s (&hdr->checksum); +cpu_to_le32s (&hdr->entry_count); +} + +void vhdx_region_entry_le_import (VHDXRegionTableEntry *e) +{ +assert (e !=NULL ); + +leguid_to_cpus (&e->guid); +le64_to_cpus (&e->file_offset); +le32_to_cpus (&e->length); +le32_to_cpus (&e->data_bits); +} + +void vhdx_region_entry_le_export (VHDXRegionTableEntry *e) +{ +assert (e !=NULL ); + +cpu_to_leguids (&e->guid); +cpu_to_le64s (&e->file_offset); +cpu_to_le32s (&e->length); +cpu_to_le32s (&e->data_bits); +} + + +/* Metadata headers & table */ +void vhdx_metadata_header_le_import (VHDXMetadataTableHeader *hdr) +{ +assert (hdr !=NULL ); + +le64_to_cpus (&hdr->signature); +le16_to_cpus (&hdr->entry_count); +} + +void vhdx_metadata_header_le_export (VHDXMetadataTableHeader *hdr) +{ +assert (hdr !=NULL ); + +cpu_to_le64s (&hdr->signature); +cpu_to_le16s (&hdr->entry_count); +} + +void vhdx_metadata_entry_le_import (VHDXMetadataTableEntry *e) +{ +assert (e !=NULL ); + +leguid_to_cpus (&e->item_id); +le32_to_cpus (&e->offset); +le32_to_cpus (&e->length); +le32_to_cpus (&e->data_bits); +} +void vhdx_metadata_entry_le_export (VHDXMetadataTableEntry *e) +{ +assert (e !=NULL ); + +cpu_to_leguids (&e->item_id); +cpu_to_le32s (&e->offset); +cpu_to_le32s (&e->length); +cpu_to_le32s (&e->data_bits); +}
block/vhdx.c +4 -16
@@ -472,10 +472,7 @@ static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s) goto fail; }memcpy (&s->rt, buffer,sizeof (s->rt)); -le32_to_cpus (&s->rt.signature); -le32_to_cpus (&s->rt.checksum); -le32_to_cpus (&s->rt.entry_count); -le32_to_cpus (&s->rt.reserved); +vhdx_region_header_le_import (&s->rt); offset +=sizeof (s->rt); if (!vhdx_checksum_is_valid(buffer, VHDX_HEADER_BLOCK_SIZE, 4) ||
@@ -494,10 +491,7 @@ static int vhdx_open_region_tables(BlockDriverState *bs, BDRVVHDXState *s)memcpy (&rt_entry, buffer + offset,sizeof (rt_entry)); offset +=sizeof (rt_entry); -leguid_to_cpus (&rt_entry.guid); -le64_to_cpus (&rt_entry.file_offset); -le32_to_cpus (&rt_entry.length); -le32_to_cpus (&rt_entry.data_bits); +vhdx_region_entry_le_import (&rt_entry);/* check for region overlap between these entries, and any * other memory regions in the file */
@@ -587,9 +581,7 @@ static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)memcpy (&s->metadata_hdr, buffer,sizeof (s->metadata_hdr)); offset +=sizeof (s->metadata_hdr); -le64_to_cpus (&s->metadata_hdr.signature); -le16_to_cpus (&s->metadata_hdr.reserved); -le16_to_cpus (&s->metadata_hdr.entry_count); +vhdx_metadata_header_le_import (&s->metadata_hdr);if (memcmp (&s->metadata_hdr.signature,"metadata" ,8 )) { ret = -EINVAL ;
@@ -608,11 +600,7 @@ static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s)memcpy (&md_entry, buffer + offset,sizeof (md_entry)); offset +=sizeof (md_entry); -leguid_to_cpus (&md_entry.item_id); -le32_to_cpus (&md_entry.offset); -le32_to_cpus (&md_entry.length); -le32_to_cpus (&md_entry.data_bits); -le32_to_cpus (&md_entry.reserved2); +vhdx_metadata_entry_le_import (&md_entry);if (guid_eq (md_entry.item_id, file_param_guid)) {if (s->metadata_entries.present &META_FILE_PARAMETER_PRESENT ) {
block/vhdx.h +8 -1
@@ -421,7 +421,14 @@ void vhdx_log_desc_le_export(VHDXLogDescriptor *d);void vhdx_log_data_le_export (VHDXLogDataSector *d);void vhdx_log_entry_hdr_le_import (VHDXLogEntryHeader *hdr);void vhdx_log_entry_hdr_le_export (VHDXLogEntryHeader *hdr); - +void vhdx_region_header_le_import (VHDXRegionTableHeader *hdr); +void vhdx_region_header_le_export (VHDXRegionTableHeader *hdr); +void vhdx_region_entry_le_import (VHDXRegionTableEntry *e); +void vhdx_region_entry_le_export (VHDXRegionTableEntry *e); +void vhdx_metadata_header_le_import (VHDXMetadataTableHeader *hdr); +void vhdx_metadata_header_le_export (VHDXMetadataTableHeader *hdr); +void vhdx_metadata_entry_le_import (VHDXMetadataTableEntry *e); +void vhdx_metadata_entry_le_export (VHDXMetadataTableEntry *e);int vhdx_user_visible_write (BlockDriverState *bs,BDRVVHDXState *s);#endif
Signature · 1 line
1.8.3.1
PATCH v7 14/19 block: vhdx - break out code operations to functions
In reply to Jeff CodyThis is preperation for vhdx_create(). The ability to write headers, and calculate the number of BAT entries will be needed within the create() functions, so move this relevant code into helper functions.
--- block/vhdx.c | 121 +++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 80 insertions(+), 41 deletions(-)
block/vhdx.c +80 -41
@@ -248,6 +248,14 @@ static void vhdx_region_unregister_all(BDRVVHDXState *s) } } +static void vhdx_set_shift_bits (BDRVVHDXState *s) +{ + s->logical_sector_size_bits =31 -clz32 (s->logical_sector_size); + s->sectors_per_block_bits =31 -clz32 (s->sectors_per_block); + s->chunk_ratio_bits =63 -clz64 (s->chunk_ratio); + s->block_size_bits =31 -clz32 (s->block_size); +} +/* * Per the MS VHDX Specification, for every VHDX file: * - The header section is fixed size - 1 MB
@@ -267,6 +275,50 @@ static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)return 0 ; } +/* +* Writes the header to the specified offset. +* +* This will optionally read in buffer data from disk (otherwise zero-fill), +* and then update the header checksum. Header is converted to proper +* endianness before being written to the specified file offset +*/ +static int vhdx_write_header (BlockDriverState *bs_file,VHDXHeader *hdr, +uint64_t offset,bool read) +{ +uint8_t *buffer =NULL ; +int ret; +VHDXHeader header_le; + +assert (bs_file !=NULL ); +assert (hdr !=NULL ); + +/* the header checksum is not over just the packed size of VHDXHeader, +* but rather over the entire 'reserved' range for the header, which is +* 4KB (VHDX_HEADER_SIZE). */ + + buffer =qemu_blockalign (bs_file,VHDX_HEADER_SIZE ); +if (read) { +/* if true, we can't assume the extra reserved bytes are 0 */ + ret =bdrv_pread (bs_file, offset, buffer,VHDX_HEADER_SIZE ); +if (ret <0 ) { + goto exit; + } + }else { +memset (buffer,0 ,VHDX_HEADER_SIZE ); + } + +/* overwrite the actual VHDXHeader portion */ +memcpy (buffer, hdr,sizeof (VHDXHeader)); + hdr->checksum =vhdx_update_checksum (buffer,VHDX_HEADER_SIZE , + offsetof(VHDXHeader , checksum)); +vhdx_header_le_export (hdr, &header_le); + ret =bdrv_pwrite_sync (bs_file, offset, &header_le,sizeof (VHDXHeader)); + +exit: +qemu_vfree (buffer); +return ret; +} +/* Update the VHDX headers * * This follows the VHDX spec procedures for header updates.
@@ -282,8 +334,6 @@ static int vhdx_update_header(BlockDriverState *bs, BDRVVHDXState *s,VHDXHeader *active_header;VHDXHeader *inactive_header; -VHDXHeader header_le; -uint8_t *buffer;/* operate on the non-current header */ if (s->curr_header ==0 ) {
@@ -311,31 +361,13 @@ static int vhdx_update_header(BlockDriverState *bs, BDRVVHDXState *s, inactive_header->log_guid = *log_guid; } -/* the header checksum is not over just the packed size of VHDXHeader, -* but rather over the entire 'reserved' range for the header, which is -* 4KB (VHDX_HEADER_SIZE). */ - - buffer =qemu_blockalign (bs,VHDX_HEADER_SIZE ); -/* we can't assume the extra reserved bytes are 0 */ - ret =bdrv_pread (bs->file, header_offset, buffer,VHDX_HEADER_SIZE ); -if (ret <0 ) { - goto exit; - } -/* overwrite the actual VHDXHeader portion */ -memcpy (buffer, inactive_header,sizeof (VHDXHeader)); - inactive_header->checksum = -vhdx_update_checksum (buffer,VHDX_HEADER_SIZE , - offsetof(VHDXHeader , checksum)); -vhdx_header_le_export (inactive_header, &header_le); - ret =bdrv_pwrite_sync (bs->file, header_offset, &header_le, -sizeof (VHDXHeader)); +vhdx_write_header (bs->file, inactive_header, header_offset, true);if (ret <0 ) { goto exit; } s->curr_header = hdr_idx; exit: -qemu_vfree (buffer);return ret; }
@@ -773,10 +805,7 @@ static int vhdx_parse_metadata(BlockDriverState *bs, BDRVVHDXState *s) goto exit; } - s->logical_sector_size_bits =31 -clz32 (s->logical_sector_size); - s->sectors_per_block_bits =31 -clz32 (s->sectors_per_block); - s->chunk_ratio_bits =63 -clz64 (s->chunk_ratio); - s->block_size_bits =31 -clz32 (s->block_size); +vhdx_set_shift_bits (s); ret =0 ;
@@ -785,6 +814,31 @@ exit:return ret; } +/* +* Calculate the number of BAT entries, including sector +* bitmap entries. +*/ +static void vhdx_calc_bat_entries (BDRVVHDXState *s) +{ +uint32_t data_blocks_cnt, bitmap_blocks_cnt; + + data_blocks_cnt = s->virtual_disk_size >> s->block_size_bits; +if (s->virtual_disk_size - (data_blocks_cnt << s->block_size_bits)) { + data_blocks_cnt++; + } + bitmap_blocks_cnt = data_blocks_cnt >> s->chunk_ratio_bits; +if (data_blocks_cnt - (bitmap_blocks_cnt << s->chunk_ratio_bits)) { + bitmap_blocks_cnt++; + } + +if (s->parent_entries) { + s->bat_entries = bitmap_blocks_cnt * (s->chunk_ratio +1 ); + }else { + s->bat_entries = data_blocks_cnt + + ((data_blocks_cnt -1 ) >> s->chunk_ratio_bits); + } + +}static void vhdx_close (BlockDriverState *bs) {
@@ -811,7 +865,6 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags,int ret =0 ;uint32_t i;uint64_t signature; -uint32_t data_blocks_cnt, bitmap_blocks_cnt;bool log_flushed = false;
@@ -862,21 +915,7 @@ static int vhdx_open(BlockDriverState *bs, QDict *options, int flags,* logical_sector_size */ bs->total_sectors = s->virtual_disk_size >> s->logical_sector_size_bits; - data_blocks_cnt = s->virtual_disk_size >> s->block_size_bits; -if (s->virtual_disk_size - (data_blocks_cnt << s->block_size_bits)) { - data_blocks_cnt++; - } - bitmap_blocks_cnt = data_blocks_cnt >> s->chunk_ratio_bits; -if (data_blocks_cnt - (bitmap_blocks_cnt << s->chunk_ratio_bits)) { - bitmap_blocks_cnt++; - } - -if (s->parent_entries) { - s->bat_entries = bitmap_blocks_cnt * (s->chunk_ratio +1 ); - }else { - s->bat_entries = data_blocks_cnt + - ((data_blocks_cnt -1 ) >> s->chunk_ratio_bits); - } +vhdx_calc_bat_entries (s); s->bat_offset = s->bat_rt.file_offset;
Signature · 1 line
1.8.3.1
PATCH v7 15/19 block: vhdx - fix comment typos in header, fix incorrect struct fields
In reply to Jeff CodyVHDXPage83Data and VHDXParentLocatorHeader both incorrectly had their MSGUID fields set as arrays of 16. This is incorrect (it stems from an early version where those fields were uint_8 arrays). Those fields were, up to this patch, unused.
Also, there were a couple of typos and incorrect wording in comments, and those have been fixed up as well.
--- block/vhdx.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
block/vhdx.h +5 -5
@@ -58,7 +58,7 @@typedef struct VHDXFileIdentifier {uint64_t signature;/* "vhdxfile" in ASCII */ uint16_t creator[256 ];/* optional; utf-16 string to identify -the vhdx file creator. Diagnotistic +the vhdx file creator. Diagnostic only */ }VHDXFileIdentifier ;
@@ -114,8 +114,8 @@ typedef struct QEMU_PACKED VHDXHeader {there is no valid log. If non-zero, log entries with this guid are valid. */ -uint16_t log_version;/* version of the log format. Mustn't be -zero, unless log_guid is also zero */ +uint16_t log_version;/* version of the log format. Must be +set to zero */ uint16_t version;/* version of the vhdx file. Currently, only supported version is "1" */ uint32_t log_length;/* length of the log. Must be multiple
@@ -281,7 +281,7 @@ typedef struct QEMU_PACKED VHDXVirtualDiskSize { } VHDXVirtualDiskSize; typedef struct QEMU_PACKED VHDXPage83Data { -MSGUID page_83_data[16 ];/* unique id for scsi devices that +MSGUID page_83_data;/* unique id for scsi devices that support page 0x83 */ } VHDXPage83Data;
@@ -296,7 +296,7 @@ typedef struct QEMU_PACKED VHDXVirtualDiskPhysicalSectorSize { } VHDXVirtualDiskPhysicalSectorSize; typedef struct QEMU_PACKED VHDXParentLocatorHeader { -MSGUID locator_type[16 ];/* type of the parent virtual disk. */ +MSGUID locator_type;/* type of the parent virtual disk. */ uint16_t reserved;uint16_t key_value_count;/* number of key/value pairs for this locator */
Signature · 1 line
1.8.3.1
PATCH v7 16/19 block: vhdx - add .bdrv_create() support
In reply to Jeff CodyThis adds support for VHDX image creation, for images of type "Fixed" and "Dynamic". "Differencing" types (i.e., VHDX images with backing files) are currently not supported.
Options for image creation include:
* log size:
The size of the journaling log for VHDX. Minimum is 1MB, and it must be a multiple of 1MB. Invalid log sizes will be silently fixed by rounding up to the nearest MB.
Default is 1MB.
* block size:
This is the size of a payload block. The range is 1MB to 256MB, inclusive, and must be a multiple of 1MB as well. Invalid sizes and multiples will be silently fixed. If '0' is passed, then a sane size is chosen (depending on virtual image size).
Default is 0 (Auto-select).
* subformat:
- "dynamic"
An image without data pre-allocated.
- "fixed"
An image with data pre-allocated.
Default is "dynamic"
When creating the image file, the lettered sections are created:
-----------------------------------------------------------------.
| (A) | (B) | (C) | (D) | (E) | File ID | Header1 | Header 2 | Region Tbl 1 | Region Tbl 2 | | | | |
.-----------------------------------------------------------------.
0 64KB 128KB 192KB 256KB 320KB
.---- ~ ----------- ~ ------------ ~ ---------------- ~ -----------.
| (F) | (G) | (H) | | Journal Log | BAT / Bitmap | Metadata | .... data ...... | | | |
.---- ~ ----------- ~ ------------ ~ ---------------- ~ -----------.
1MB (var.) (var.) (var.)
--- block/vhdx.c | 557 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/vhdx.h | 15 +- 2 files changed, 571 insertions(+), 1 deletion(-)
block/vhdx.c +557 -0
@@ -23,6 +23,19 @@#include "migration/migration.h" #include <uuid/uuid.h> +#include <glib.h> + +/* Options for VHDX creation */ + +#define VHDX_BLOCK_OPT_LOG_SIZE "log_size" +#define VHDX_BLOCK_OPT_BLOCK_SIZE "block_size" +#define VHDX_BLOCK_OPT_ZERO "block_state_zero" + +typedef enum VHDXImageType { +VHDX_TYPE_DYNAMIC =0 , +VHDX_TYPE_FIXED , +VHDX_TYPE_DIFFERENCING ,/* Currently unsupported */ +}VHDXImageType ;/* Several metadata and region table data entries are identified by * guids in a MS-specific GUID format. */
@@ -1304,6 +1317,547 @@ exit: } + +/* +* Create VHDX Headers +* +* There are 2 headers, and the highest sequence number will represent +* the active header +*/ +static int vhdx_create_new_headers (BlockDriverState *bs,uint64_t image_size, +uint32_t log_size) +{ +int ret =0 ; +VHDXHeader *hdr =NULL ; + + hdr =g_malloc0 (sizeof (VHDXHeader)); + + hdr->signature =VHDX_HEADER_SIGNATURE ; + hdr->sequence_number =g_random_int (); + hdr->log_version =0 ; + hdr->version =1 ; + hdr->log_length = log_size; + hdr->log_offset =VHDX_HEADER_SECTION_END ; +vhdx_guid_generate (&hdr->file_write_guid); +vhdx_guid_generate (&hdr->data_write_guid); + + ret =vhdx_write_header (bs, hdr,VHDX_HEADER1_OFFSET , false); +if (ret <0 ) { + goto exit; + } + hdr->sequence_number++; + ret =vhdx_write_header (bs, hdr,VHDX_HEADER2_OFFSET , false); +if (ret <0 ) { + goto exit; + } + +exit: +g_free (hdr); +return ret; +} + + +/* +* Create the Metadata entries. +* +* For more details on the entries, see section 3.5 (pg 29) in the +* VHDX 1.00 specification. +* +* We support 5 metadata entries (all required by spec): +* File Parameters, +* Virtual Disk Size, +* Page 83 Data, +* Logical Sector Size, +* Physical Sector Size +* +* The first 64KB of the Metadata section is reserved for the metadata +* header and entries; beyond that, the metadata items themselves reside. +*/ +static int vhdx_create_new_metadata (BlockDriverState *bs, +uint64_t image_size, +uint32_t block_size, +uint32_t sector_size, +uint64_t metadata_offset, +VHDXImageType type) +{ +int ret =0 ; +uint32_t offset =0 ; +void *buffer =NULL ; +void *entry_buffer; +VHDXMetadataTableHeader *md_table;; +VHDXMetadataTableEntry *md_table_entry; + +/* Metadata entries */ +VHDXFileParameters *mt_file_params; +VHDXVirtualDiskSize *mt_virtual_size; +VHDXPage83Data *mt_page83; +VHDXVirtualDiskLogicalSectorSize *mt_log_sector_size; +VHDXVirtualDiskPhysicalSectorSize *mt_phys_sector_size; + + entry_buffer =g_malloc0 (sizeof (VHDXFileParameters) + +sizeof (VHDXVirtualDiskSize) + +sizeof (VHDXPage83Data) + +sizeof (VHDXVirtualDiskLogicalSectorSize) + +sizeof (VHDXVirtualDiskPhysicalSectorSize)); + + mt_file_params = entry_buffer; + offset +=sizeof (VHDXFileParameters); + mt_virtual_size = entry_buffer + offset; + offset +=sizeof (VHDXVirtualDiskSize); + mt_page83 = entry_buffer + offset; + offset +=sizeof (VHDXPage83Data); + mt_log_sector_size = entry_buffer + offset; + offset +=sizeof (VHDXVirtualDiskLogicalSectorSize); + mt_phys_sector_size = entry_buffer + offset; + + mt_file_params->block_size =cpu_to_le32 (block_size); +if (type ==VHDX_TYPE_FIXED ) { + mt_file_params->data_bits |=VHDX_PARAMS_LEAVE_BLOCKS_ALLOCED ; +cpu_to_le32s (&mt_file_params->data_bits); + } + +vhdx_guid_generate (&mt_page83->page_83_data); +cpu_to_leguids (&mt_page83->page_83_data); + mt_virtual_size->virtual_disk_size =cpu_to_le64 (image_size); + mt_log_sector_size->logical_sector_size =cpu_to_le32 (sector_size); + mt_phys_sector_size->physical_sector_size =cpu_to_le32 (sector_size); + + buffer =g_malloc0 (VHDX_HEADER_BLOCK_SIZE ); + md_table = buffer; + + md_table->signature =VHDX_METADATA_SIGNATURE ; + md_table->entry_count =5 ; +vhdx_metadata_header_le_export (md_table); + + +/* This will reference beyond the reserved table portion */ + offset =64 * KiB; + + md_table_entry = buffer +sizeof (VHDXMetadataTableHeader); + + md_table_entry[0 ].item_id = file_param_guid; + md_table_entry[0 ].offset = offset; + md_table_entry[0 ].length =sizeof (VHDXFileParameters); + md_table_entry[0 ].data_bits |=VHDX_META_FLAGS_IS_REQUIRED ; + offset += md_table_entry[0 ].length; +vhdx_metadata_entry_le_export (&md_table_entry[0 ]); + + md_table_entry[1 ].item_id = virtual_size_guid; + md_table_entry[1 ].offset = offset; + md_table_entry[1 ].length =sizeof (VHDXVirtualDiskSize); + md_table_entry[1 ].data_bits |=VHDX_META_FLAGS_IS_REQUIRED | +VHDX_META_FLAGS_IS_VIRTUAL_DISK ; + offset += md_table_entry[1 ].length; +vhdx_metadata_entry_le_export (&md_table_entry[1 ]); + + md_table_entry[2 ].item_id = page83_guid; + md_table_entry[2 ].offset = offset; + md_table_entry[2 ].length =sizeof (VHDXPage83Data); + md_table_entry[2 ].data_bits |=VHDX_META_FLAGS_IS_REQUIRED | +VHDX_META_FLAGS_IS_VIRTUAL_DISK ; + offset += md_table_entry[2 ].length; +vhdx_metadata_entry_le_export (&md_table_entry[2 ]); + + md_table_entry[3 ].item_id = logical_sector_guid; + md_table_entry[3 ].offset = offset; + md_table_entry[3 ].length =sizeof (VHDXVirtualDiskLogicalSectorSize); + md_table_entry[3 ].data_bits |=VHDX_META_FLAGS_IS_REQUIRED | +VHDX_META_FLAGS_IS_VIRTUAL_DISK ; + offset += md_table_entry[3 ].length; +vhdx_metadata_entry_le_export (&md_table_entry[3 ]); + + md_table_entry[4 ].item_id = phys_sector_guid; + md_table_entry[4 ].offset = offset; + md_table_entry[4 ].length =sizeof (VHDXVirtualDiskPhysicalSectorSize); + md_table_entry[4 ].data_bits |=VHDX_META_FLAGS_IS_REQUIRED | +VHDX_META_FLAGS_IS_VIRTUAL_DISK ; +vhdx_metadata_entry_le_export (&md_table_entry[4 ]); + + ret =bdrv_pwrite (bs, metadata_offset, buffer,VHDX_HEADER_BLOCK_SIZE ); +if (ret <0 ) { + goto exit; + } + + ret =bdrv_pwrite (bs, metadata_offset + (64 * KiB), entry_buffer, +VHDX_HEADER_BLOCK_SIZE ); +if (ret <0 ) { + goto exit; + } + + +exit: +g_free (buffer); +g_free (entry_buffer); +return ret; +} + +/* This create the actual BAT itself. We currently only support +* 'Dynamic' and 'Fixed' image types. +* +* Dynamic images: default state of the BAT is all zeroes. +* +* Fixed images: default state of the BAT is fully populated, with +* file offsets and state PAYLOAD_BLOCK_FULLY_PRESENT. +*/ +static int vhdx_create_bat (BlockDriverState *bs,BDRVVHDXState *s, +uint64_t image_size,VHDXImageType type, +bool use_zero_blocks,VHDXRegionTableEntry *rt_bat) +{ +int ret =0 ; +uint64_t data_file_offset; +uint64_t total_sectors =0 ; +uint64_t sector_num =0 ; +uint64_t unused; +int block_state; +VHDXSectorInfo sinfo; + +assert (s->bat ==NULL ); + +/* this gives a data start after BAT/bitmap entries, and well +* past any metadata entries (with a 4 MB buffer for future +* expansion */ + data_file_offset = rt_bat->file_offset + rt_bat->length +5 * MiB; + total_sectors = image_size >> s->logical_sector_size_bits; + +if (type ==VHDX_TYPE_DYNAMIC ) { +/* All zeroes, so we can just extend the file - the end of the BAT +* is the furthest thing we have written yet */ + ret =bdrv_truncate (bs, data_file_offset); +if (ret <0 ) { + goto exit; + } + }else if (type ==VHDX_TYPE_FIXED ) { + ret =bdrv_truncate (bs, data_file_offset + image_size); +if (ret <0 ) { + goto exit; + } + }else { + ret = -ENOTSUP ; + goto exit; + } + +if (type ==VHDX_TYPE_FIXED || + use_zero_blocks || +bdrv_has_zero_init (bs) ==0 ) { +/* for a fixed file, the default BAT entry is not zero */ + s->bat =g_malloc0 (rt_bat->length); + block_state = type ==VHDX_TYPE_FIXED ?PAYLOAD_BLOCK_FULLY_PRESENT : +PAYLOAD_BLOCK_NOT_PRESENT ; + block_state = use_zero_blocks ?PAYLOAD_BLOCK_ZERO : block_state; +/* fill the BAT by emulating sector writes of sectors_per_block size */ +while (sector_num < total_sectors) { +vhdx_block_translate (s, sector_num, s->sectors_per_block, &sinfo); + sinfo.file_offset = data_file_offset + + (sector_num << s->logical_sector_size_bits); + sinfo.file_offset =ROUND_UP (sinfo.file_offset, MiB); +vhdx_update_bat_table_entry (bs, s, &sinfo, &unused, &unused, + block_state); +cpu_to_le64s (&s->bat[sinfo.bat_idx]); + sector_num += s->sectors_per_block; + } + ret =bdrv_pwrite (bs, rt_bat->file_offset, s->bat, rt_bat->length); +if (ret <0 ) { + goto exit; + } + } + + + +exit: +g_free (s->bat); +return ret; +} + +/* Creates the region table header, and region table entries. +* There are 2 supported region table entries: BAT, and Metadata/ +* +* As the calculations for the BAT region table are also needed +* to create the BAT itself, we will also cause the BAT to be +* created. +*/ +static int vhdx_create_new_region_table (BlockDriverState *bs, +uint64_t image_size, +uint32_t block_size, +uint32_t sector_size, +uint32_t log_size, +bool use_zero_blocks, +VHDXImageType type, +uint64_t *metadata_offset) +{ +int ret =0 ; +uint32_t offset =0 ; +void *buffer =NULL ; +BDRVVHDXState *s =NULL ; +VHDXRegionTableHeader *region_table; +VHDXRegionTableEntry *rt_bat; +VHDXRegionTableEntry *rt_metadata; + +assert (metadata_offset !=NULL ); + +/* Populate enough of the BDRVVHDXState to be able to use the +* pre-existing BAT calculation, translation, and update functions */ + s =g_malloc0 (sizeof (BDRVVHDXState)); + + s->chunk_ratio = (VHDX_MAX_SECTORS_PER_BLOCK ) * + (uint64_t ) sector_size / (uint64_t ) block_size; + + s->sectors_per_block = block_size / sector_size; + s->virtual_disk_size = image_size; + s->block_size = block_size; + s->logical_sector_size = sector_size; + +vhdx_set_shift_bits (s); + +vhdx_calc_bat_entries (s); + +/* At this point the VHDX state is populated enough for creation */ + +/* a single buffer is used so we can calculate the checksum over the +* entire 64KB block */ + buffer =g_malloc0 (VHDX_HEADER_BLOCK_SIZE ); + region_table = buffer; + offset +=sizeof (VHDXRegionTableHeader); + rt_bat = buffer + offset; + offset +=sizeof (VHDXRegionTableEntry); + rt_metadata = buffer + offset; + + region_table->signature =VHDX_REGION_SIGNATURE ; + region_table->entry_count =2 ;/* BAT and Metadata */ + + rt_bat->guid = bat_guid; + rt_bat->length =ROUND_UP (s->bat_entries *sizeof (VHDXBatEntry), MiB); + rt_bat->file_offset =ROUND_UP (VHDX_HEADER_SECTION_END + log_size, MiB); + s->bat_offset = rt_bat->file_offset; + + rt_metadata->guid = metadata_guid; + rt_metadata->file_offset =ROUND_UP (rt_bat->file_offset + rt_bat->length, + MiB); + rt_metadata->length =1 * MiB;/* min size, and more than enough */ + *metadata_offset = rt_metadata->file_offset; + +vhdx_update_checksum (buffer,VHDX_HEADER_BLOCK_SIZE , + offsetof(VHDXRegionTableHeader , checksum)); + + +/* The region table gives us the data we need to create the BAT, +* so do that now */ + ret =vhdx_create_bat (bs, s, image_size, type, use_zero_blocks, rt_bat); + +/* Now write out the region headers to disk */ +vhdx_region_header_le_export (region_table); +vhdx_region_entry_le_export (rt_bat); +vhdx_region_entry_le_export (rt_metadata); + + ret =bdrv_pwrite (bs,VHDX_REGION_TABLE_OFFSET , buffer, +VHDX_HEADER_BLOCK_SIZE ); +if (ret <0 ) { + goto exit; + } + + ret =bdrv_pwrite (bs,VHDX_REGION_TABLE2_OFFSET , buffer, +VHDX_HEADER_BLOCK_SIZE ); +if (ret <0 ) { + goto exit; + } + + +exit: +g_free (s); +g_free (buffer); +return ret; +} + +/* We need to create the following elements: +* +* .-----------------------------------------------------------------. +* | (A) | (B) | (C) | (D) | (E) | +* | File ID | Header1 | Header 2 | Region Tbl 1 | Region Tbl 2 | +* | | | | | | +* .-----------------------------------------------------------------. +* 0 64KB 128KB 192KB 256KB 320KB +* +* +* .---- ~ ----------- ~ ------------ ~ ---------------- ~ -----------. +* | (F) | (G) | (H) | | +* | Journal Log | BAT / Bitmap | Metadata | .... data ...... | +* | | | | | +* .---- ~ ----------- ~ ------------ ~ ---------------- ~ -----------. +* 1MB +*/ +static int vhdx_create (const char *filename,QEMUOptionParameter *options, +Error **errp) +{ +int ret =0 ; +uint64_t image_size = (uint64_t )2 * GiB; +uint32_t log_size =1 * MiB; +uint32_t block_size =0 ; +uint64_t signature; +uint64_t metadata_offset; +bool use_zero_blocks = false; + +gunichar2 *creator =NULL ; +glong creator_items; +BlockDriverState *bs; +const char *type =NULL ; +VHDXImageType image_type; +Error *local_err =NULL ; + +while (options && options->name) { +if (!strcmp (options->name,BLOCK_OPT_SIZE )) { + image_size = options->value.n; + }else if (!strcmp (options->name,VHDX_BLOCK_OPT_LOG_SIZE )) { + log_size = options->value.n; + }else if (!strcmp (options->name,VHDX_BLOCK_OPT_BLOCK_SIZE )) { + block_size = options->value.n; + }else if (!strcmp (options->name,BLOCK_OPT_SUBFMT )) { + type = options->value.s; + }else if (!strcmp (options->name,VHDX_BLOCK_OPT_ZERO )) { + use_zero_blocks = options->value.n !=0 ; + } + options++; + } + +if (image_size >VHDX_MAX_IMAGE_SIZE ) { +fprintf (stderr,"Image size too large; max of 64TB\n" ); + ret = -EINVAL ; + goto exit; + } + +if (type ==NULL ) { + type ="dynamic" ; + } + +if (!strcmp (type,"dynamic" )) { + image_type =VHDX_TYPE_DYNAMIC ; + }else if (!strcmp (type,"fixed" )) { + image_type =VHDX_TYPE_FIXED ; + }else if (!strcmp (type,"differencing" )) { +fprintf (stderr,"Differencing files not yet supported\n" ); + ret = -ENOTSUP ; + goto exit; + }else { + ret = -EINVAL ; + goto exit; + } + +/* These are pretty arbitrary, and mainly designed to keep the BAT +* size reasonable to load into RAM */ +if (block_size ==0 ) { +if (image_size >32 * TiB) { + block_size =64 * MiB; + }else if (image_size > (uint64_t )100 * GiB) { + block_size =32 * MiB; + }else if (image_size >1 * GiB) { + block_size =16 * MiB; + }else { + block_size =8 * MiB; + } + } + + +/* make the log size close to what was specified, but must be +* min 1MB, and multiple of 1MB */ + log_size =ROUND_UP (log_size, MiB); + + block_size =ROUND_UP (block_size, MiB); + block_size = block_size >VHDX_BLOCK_SIZE_MAX ?VHDX_BLOCK_SIZE_MAX : + block_size; + + ret =bdrv_create_file (filename, options, &local_err); +if (ret <0 ) { +error_propagate (errp, local_err); + goto exit; + } + + ret =bdrv_file_open (&bs, filename,NULL ,BDRV_O_RDWR , &local_err); +if (ret <0 ) { +error_propagate (errp, local_err); + goto exit; + } + +/* Create (A) */ + +/* The creator field is optional, but may be useful for +* debugging / diagnostics */ + creator =g_utf8_to_utf16 ("QEMU v" QEMU_VERSION ,-1 ,NULL , + &creator_items,NULL ); + signature =cpu_to_le64 (VHDX_FILE_SIGNATURE ); +bdrv_pwrite (bs,VHDX_FILE_ID_OFFSET , &signature,sizeof (signature)); +if (ret <0 ) { + goto delete_and_exit; + } +if (creator) { +bdrv_pwrite (bs,VHDX_FILE_ID_OFFSET +sizeof (signature), creator, + creator_items *sizeof (gunichar2)); +if (ret <0 ) { + goto delete_and_exit; + } + } + + +/* Creates (B),(C) */ + ret =vhdx_create_new_headers (bs, image_size, log_size); +if (ret <0 ) { + goto delete_and_exit; + } + +/* Creates (D),(E),(G) explicitly. (F) created as by-product */ + ret =vhdx_create_new_region_table (bs, image_size, block_size,512 , + log_size, use_zero_blocks, image_type, + &metadata_offset); +if (ret <0 ) { + goto delete_and_exit; + } + +/* Creates (H) */ + ret =vhdx_create_new_metadata (bs, image_size, block_size,512 , + metadata_offset, image_type); +if (ret <0 ) { + goto delete_and_exit; + } + + + +delete_and_exit: +bdrv_unref (bs); +exit: +g_free (creator); +return ret; +} + +static QEMUOptionParameter vhdx_create_options[] = { + { + .name =BLOCK_OPT_SIZE , + .type =OPT_SIZE , + .help ="Virtual disk size; max of 64TB." + }, + { + .name =VHDX_BLOCK_OPT_LOG_SIZE , + .type =OPT_SIZE , + .value.n =1 * MiB, + .help ="Log size; min 1MB." + }, + { + .name =VHDX_BLOCK_OPT_BLOCK_SIZE , + .type =OPT_SIZE , + .value.n =0 , + .help ="Block Size; min 1MB, max 256MB. " \ +"0 means auto-calculate based on image size." + }, + { + .name =BLOCK_OPT_SUBFMT , + .type =OPT_STRING , + .help ="VHDX format type, can be either 'dynamic' or 'fixed'. " \ +"Default is 'dynamic'." + }, + { + .name =VHDX_BLOCK_OPT_ZERO , + .type =OPT_FLAG , + .help ="Force use of payload blocks of type 'ZERO'. Non-standard." + }, + {NULL } +}; +static BlockDriver bdrv_vhdx = { .format_name ="vhdx" , .instance_size =sizeof (BDRVVHDXState),
@@ -1313,6 +1867,9 @@ static BlockDriver bdrv_vhdx = { .bdrv_reopen_prepare = vhdx_reopen_prepare, .bdrv_co_readv = vhdx_co_readv, .bdrv_co_writev = vhdx_co_writev, + .bdrv_create = vhdx_create, + + .create_options = vhdx_create_options, };static void bdrv_vhdx_init (void )
block/vhdx.h +14 -1
@@ -18,6 +18,11 @@#ifndef BLOCK_VHDX_H #define BLOCK_VHDX_H +#define KiB (1 * 1024) +#define MiB (KiB * 1024) +#define GiB (MiB * 1024) +#define TiB ((uint64_t) GiB * 1024) +/* Structures and fields present in the VHDX file */ /* The header section has the following blocks,
@@ -36,8 +41,9 @@#define VHDX_HEADER1_OFFSET (VHDX_HEADER_BLOCK_SIZE * 1)#define VHDX_HEADER2_OFFSET (VHDX_HEADER_BLOCK_SIZE * 2)#define VHDX_REGION_TABLE_OFFSET (VHDX_HEADER_BLOCK_SIZE * 3) +#define VHDX_REGION_TABLE2_OFFSET (VHDX_HEADER_BLOCK_SIZE * 4) - +#define VHDX_HEADER_SECTION_END (1 * MiB)/* * A note on the use of MS-GUID fields. For more details on the GUID, * please see: https://en.wikipedia.org/wiki/Globally_unique_identifier.
@@ -55,6 +61,7 @@/* These structures are ones that are defined in the VHDX specification * document */ +#define VHDX_FILE_SIGNATURE 0x656C696678646876/* "vhdxfile" in ASCII */ typedef struct VHDXFileIdentifier { uint64_t signature; /* "vhdxfile" in ASCII */ uint16_t creator[256]; /* optional; utf-16 string to identify
@@ -85,6 +92,7 @@ typedef struct QEMU_PACKED MSGUID {/* The full header is 4KB, although the actual header data is much smaller. * But for the checksum calculation, it is over the entire 4KB structure, * not just the defined portion of it */ +#define VHDX_HEADER_SIGNATURE 0x64616568 typedef struct QEMU_PACKED VHDXHeader {uint32_t signature;/* "head" in ASCII */ uint32_t checksum;/* CRC-32C hash of the whole header */
@@ -125,6 +133,7 @@ typedef struct QEMU_PACKED VHDXHeader { } VHDXHeader;/* Header for the region table block */ +#define VHDX_REGION_SIGNATURE 0x69676572/* "regi" in ASCII */ typedef struct QEMU_PACKED VHDXRegionTableHeader {uint32_t signature;/* "regi" in ASCII */ uint32_t checksum;/* CRC-32C hash of the 64KB table */
@@ -238,6 +247,7 @@ typedef uint64_t VHDXBatEntry;#define VHDX_METADATA_MAX_ENTRIES 2047/* not including the header */ #define VHDX_METADATA_TABLE_MAX_SIZE \ (VHDX_METADATA_ENTRY_SIZE * (VHDX_METADATA_MAX_ENTRIES+1)) +#define VHDX_METADATA_SIGNATURE 0x617461646174656D/* "metadata" in ASCII */ typedef struct QEMU_PACKED VHDXMetadataTableHeader {uint64_t signature;/* "metadata" in ASCII */ uint16_t reserved;
@@ -267,6 +277,8 @@ typedef struct QEMU_PACKED VHDXMetadataTableEntry {If set indicates a fixed size VHDX file */ #define VHDX_PARAMS_HAS_PARENT 0x02/* has parent / backing file */ +#define VHDX_BLOCK_SIZE_MIN (1 * MiB) +#define VHDX_BLOCK_SIZE_MAX (256 * MiB) typedef struct QEMU_PACKED VHDXFileParameters {uint32_t block_size;/* size of each payload block, always power of 2, <= 256MB and >= 1MB. */
@@ -274,6 +286,7 @@ typedef struct QEMU_PACKED VHDXFileParameters {the rest are reserved (see above) */ } VHDXFileParameters; +#define VHDX_MAX_IMAGE_SIZE ((uint64_t) 64 * TiB) typedef struct QEMU_PACKED VHDXVirtualDiskSize {uint64_t virtual_disk_size;/* Size of the virtual disk, in bytes. Must be multiple of the sector size,
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v7 16/19] block: vhdx - add .bdrv_create() support
block/vhdx.c exit: · line 1721
Quoted 2 lines · Jeff Cody
block/vhdx.c +2
-0+if (image_size >VHDX_MAX_IMAGE_SIZE ) { +fprintf (stderr,"Image size too large; max of 64TB\n" );
Please use error_setg_errno(errp, EINVAL, "Image size too large; max of 64TB") instead of fprintf().
There are other instances in this file that should be converted too.
PATCH v7 17/19 block: vhdx - update _make_test_img() to filter out vhdx options
In reply to Jeff CodyThe non-global option output is suppresed in _make_test_img() for output verification in the 0?? tests. This adds suppression for the vhdx-unique options as well. This allows check -vhdx to run successfully.
--- tests/qemu-iotests/common.rc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tests/qemu-iotests/common.rc +3 -1
@@ -148,7 +148,9 @@ _make_test_img() -e "s# zeroed_grain=\\(on\\|off\\)##g" \ -e "s# subformat='[^']*'##g" \ -e "s# adapter_type='[^']*'##g" \ - -e "s# lazy_refcounts=\\(on\\|off\\)##g" + -e "s# lazy_refcounts=\\(on\\|off\\)##g" \ + -e "s# block_size=[0-9]\\+##g" \ + -e "s# log_size=[0-9]\\+##g" # Start an NBD server on the image file, which is what we'll be talking to if [ $IMGPROTO = "nbd" ]; then
Signature · 1 line
1.8.3.1
PATCH v7 18/19 block: qemu-iotests for vhdx, add write test support
In reply to Jeff CodyThis removes the IMGFMT_GENERIC blocker for read-only, so existing iotests run read/write tests for vhdx images created by qemu-img (e.g. tests 001, 002, 003).
In addition, this updates the sample image test for the Hyper-V created image, to verify we can write it as well.
--- tests/qemu-iotests/064 | 11 +++++++++++ tests/qemu-iotests/064.out | 14 ++++++++++++++ tests/qemu-iotests/common | 1 - 3 files changed, 25 insertions(+), 1 deletion(-)
tests/qemu-iotests/064 +11 -0
@@ -56,6 +56,17 @@ echo echo "=== Verify pattern 0x00, 66M - 1024M ===" $QEMU_IO -r -c "read -pP 0x00 66M 958M" "$TEST_IMG" | _filter_qemu_io +echo +echo "=== Verify pattern write, 0xc3 99M-157M ===" +$QEMU_IO -c "write -pP 0xc3 99M 58M" "$TEST_IMG" | _filter_qemu_io +# first verify we didn't write where we should not have +$QEMU_IO -c "read -pP 0xa5 0 33M" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -pP 0x96 33M 33M" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -pP 0x00 66M 33M" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -pP 0x00 157MM 867MM" "$TEST_IMG" | _filter_qemu_io +# now verify what we should have actually written +$QEMU_IO -c "read -pP 0xc3 99M 58M" "$TEST_IMG" | _filter_qemu_io + # success, all done echo "*** done" rm -f $seq.full
tests/qemu-iotests/064.out +14 -0
@@ -11,4 +11,18 @@ read 34603008/34603008 bytes at offset 34603008 === Verify pattern 0x00, 66M - 1024M === read 1004535808/1004535808 bytes at offset 69206016 958 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Verify pattern write, 0xc3 99M-157M === +wrote 60817408/60817408 bytes at offset 103809024 +58 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 34603008/34603008 bytes at offset 0 +33 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 34603008/34603008 bytes at offset 34603008 +33 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 34603008/34603008 bytes at offset 69206016 +33 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 909115392/909115392 bytes at offset 164626432 +867 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 60817408/60817408 bytes at offset 103809024 +58 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) *** done
tests/qemu-iotests/common +0 -1
@@ -200,7 +200,6 @@ testlist options -vhdx) IMGFMT=vhdx xpand=false - IMGFMT_GENERIC=false ;; -rbd)
Signature · 1 line
1.8.3.1
PATCH v7 19/19 block: vhdx qemu-iotest - log replay of data sector
In reply to Jeff CodyThis tests the replay of a data sector in a VHDX image file.
The image file is a 10G dynamic image, with 4MB block size. The image was created with qemu-img, and the log left unplayed by modification of the vhdx image format driver.
It was verified under both QEMU and Hyper-V that the image file, post log replay, matched.
--- tests/qemu-iotests/065 | 67 +++++++++++++++++++++ tests/qemu-iotests/065.out | 8 +++ tests/qemu-iotests/group | 1 + .../sample_images/iotest-dirtylog-10G-4M.vhdx.bz2 | Bin 0 -> 4490 bytes 4 files changed, 76 insertions(+) create mode 100755 tests/qemu-iotests/065 create mode 100644 tests/qemu-iotests/065.out create mode 100644 tests/qemu-iotests/sample_images/iotest-dirtylog-10G-4M.vhdx.bz2
tests/qemu-iotests/065 new file +67 -0
@@ -0,0 +1,67 @@ +#!/bin/bash +# +# Test VHDX log replay from an image with a journal that needs to be +# replayed +# +# Copyright (C) 2013 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt vhdx +_supported_proto generic +_supported_os Linux + +# With the log replayed, the pattern 0xa5 extends to 0xc025000 +# If the log was not replayed, it would only extend to 0xc000000 +# +# This image is a 10G dynamic image, with 4M block size, and 1 unplayed +# data sector in the log +# +# This image was created with qemu-img, however it was verified using +# Hyper-V to properly replay the logs and give the same post-replay +# image as qemu. +_use_sample_img iotest-dirtylog-10G-4M.vhdx.bz2 + +echo +echo "=== Verify open image read-only fails, due to dirty log ===" +$QEMU_IO -r -c "read -pP 0xa5 0 18M" "$TEST_IMG" 2>&1 | grep -o "Permission denied" + +echo "=== Verify open image replays log ===" +$QEMU_IO -c "read -pP 0xa5 0 18M" "$TEST_IMG" | _filter_qemu_io + +# success, all done +echo "*** done" +rm -f $seq.full +status=0
tests/qemu-iotests/065.out new file +8 -0
@@ -0,0 +1,8 @@ +QA output created by 065 + +=== Verify open image read-only fails, due to dirty log === +Permission denied +=== Verify open image replays log === +read 18874368/18874368 bytes at offset 0 +18 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** done
tests/qemu-iotests/group +1 -0
@@ -70,3 +70,4 @@ 062 rw auto 063 rw auto 064 rw auto +065 rw auto
tests/qemu-iotests/sample_images/iotest-dirtylog-10G-4M.vhdx.bz2 new file binary +0 -0
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v7 19/19] block: vhdx qemu-iotest - log replay of data sector
Quoted 10 lines · Jeff Cody
This tests the replay of a data sector in a VHDX image file.
The image file is a 10G dynamic image, with 4MB block size. The image was created with qemu-img, and the log left unplayed by modification of the vhdx image format driver.
It was verified under both QEMU and Hyper-V that the image file, post log replay, matched.
Signed-off-by Jeff Cody <[email protected]>
Quoted 9 lines
--- tests/qemu-iotests/065 | 67 +++++++++++++++++++++ tests/qemu-iotests/065.out | 8 +++ tests/qemu-iotests/group | 1 + .../sample_images/iotest-dirtylog-10G-4M.vhdx.bz2 | Bin 0 -> 4490 bytes 4 files changed, 76 insertions(+) create mode 100755 tests/qemu-iotests/065 create mode 100644 tests/qemu-iotests/065.out create mode 100644 tests/qemu-iotests/sample_images/iotest-dirtylog-10G-4M.vhdx.bz2
It would be nice to add more extensive tests for log replay and corruption in the future.
But this is a good start and doesn't hold up this series.
Quoted 208 lines · Jeff Cody
This patch series contains the initial VHDX log parsing, replay, write support, and image creation.
=== v7 changes ===
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v7-upstreamRebased to latest qemu/master (picked up vhdx r/o tests, migration blocker)
Patch 8/19: * validate log
descriptor_count(Stefan)
* fix typos in comments (Stefan)
* Removed unneccessary initialization (Stefan)
* Replay log prior to metadata (Stefan)
* Invhdx_log_flush(), callbdrv_flush()prior to zeroing out the log guid in the header.
* Invhdx_close(), set freed pointers to NULLPatch 9/19: * correct logic for region overlap (Stefan)
Patch 10/19: * add missing goto exit in error case (Stefan)
* addbdrv_flush()to ensure data is stable on disk (Stefan)Patch 11/19: * fixed typos in comments (Stefan)
* QEMU coding style changes (Stefan)
* renamebat_entrytobat_entry_lefor clarity (Stefan)
* AddPAYLOAD_BLOCK_ZEROexplicit zero padding for protocols that do not support zero init (Stefan)
* renamePAYLOAD_BLOCK_FULL_PRESENTtoPAYLOAD_BLOCK_FULLY_PRESENT(Stefan)Patch 13/19: * Fixed typo in commit message (Stefan)
Patch 19/19: * New, adds qemu-io test for log replay of data sector
v6 Patch 17/20: * Dropped (already upstream)
v6 Patch 18/20: * Dropped (already upstream)=== v6 changes ===
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v6-upstreamRebased to latest qemu/master:
Patch 16/20: .bdrv_create() propagates Error, and
bdrv_unref()used
instead ofbdrv_delete().Patch 17 & 18 are already included in another series:
[PATCH v3 0/3] qemu-iotests with sample images, vhdx test, cleanupThey are included here to provide a base for patches 19 & 20. If the above series is applied before this series, then patches 17 and 18 can be ignored.
Patch 19/20: In qemu-io tests
_make_test_img(), filter out vhdx-specific
options for .bdrv_create().Patch 20/20: Add VHDX write test case to 064.
=== v5 changes ===
v5 is also available for testing from:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v5-upstreamMost of the patches from v4 -> v5 are the same, but there are a few differences and a few new patches. Here is a summary of which patches are different and/or new:
Patch highlights:
Patch 7 just some minor code movement, in prep for changes in patch 8
Patch 8 incorporates review feedback from Stefan, for the previous Patch 7
in v4.Patch 9 adds region checking for log, region table, and metadata tables, per
suggestion from Stefan.Patch 10 minor change from changes made in 8/16 (
vhdx_guid_is_zero()is gone)Patch 12 is just some minor housekeeping, to get rid of bit shifting that
doesn't need to happen.=== v4 changes ===
v4 patches are available from github as well, on branch vhdx-write-v4-upstream:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v4-upstream
https://github.com/codyprime/qemu-kvm-jtc.gitThose in the midst of reviewing v3, don't fear - the only changes with v4 is the addition of patches on the end of the series (patches 10-13). These patches enable creating VHDX images. Image files created have been (briefly & lightly) tested on Hyper-V running on Windows Server 2012.
Some of the new patches could be squashed with earlier patches in the series, but I refrained from doing so, since some of the patches have already been reviewed, and others are in the midst of review. I want to make it as easy as possible on those currently reviewing. There is nothing critical that needs to be pushed into the earlier patches.
New patches:
Patch 10: Breaks out some more endian translation functions
(likely squashable into patch 5)Patch 11: Break out some operations into seperate helper functions
Patch 12: More comment typos and header fixes in
vhdx.h
(likely squashable into patch 1)Patch 13: Adds .bdrv_create() for vhdx. VHDX images are can be created for
Fixed or Dynamic images.Patches 1-9 are unchanged.
=== end v4 changelog ===
=== v3 changes ===
Thank you Kevin & Stefan for the feedback; incoporated in v3:
Patch 1: --- nil ---
Patch 2: * use sizeof(crc) instead of 4
* remove outdated comment
* use sizeof(MSGUID) instead of 16
* direct assignment of guid structs rather than memcpy
* rename 'rw' to 'generate_data_write_guid'
* useoffsetof()instead of 4
* comment typos
* add missing error checking
* MSGUID is nowQEMU_PACKED
* configure enable for vhdx is now correct and not braindeadPatch 3: --- nil ---
Patch 4: * code style fixes
* removed unused struct (VHDXLogEntryInfo)Patch 5: * more direct assignment of guid rather than memcpy
* order of operation in export/import the same now
* became less generous with newlines (bah-humbug!)Patch 6: * more direct assignment of guid rather than memcpy
* add error check invhdx_user_visible_write(), now returns intPatch 7: * check error return now of
vhdx_user_visible_write()Patch 8: * check error return now of
vhdx_user_visible_write()
*vhdx_log_write_sectors()usesbdrv_pwrite()vsbdrv_pwrite_sync()
* more direct assignment of guid rather than memcpy
* useoffsetof()instead of 4Patch 9: --- nil ---
=== end v3 changelog ===
v2 changes: Incorporated Fam's review feedback - Thank you Fam for the feedback
=== end v2 changelog ===
This will allow an existing log in a VHDX image to be replayed (e.g., a VHDX image from a Hyper-V host that crashed). In addition, metadata writes are enabled through the log. This allows write support to be enabled for VHDX, as the BAT can be updated safely via the log journal.
These exact patches are available from github, for testing:
https://github.com/codyprime/qemu-kvm-jtc/tree/vhdx-write-v2-upstreamThe latest vhdx work (including anything beyond these patches, such as backing/parent file support) can be found at:
https://github.com/codyprime/qemu-kvm-jtc/tree/jtc-vhdx-latestJeff Cody (19):
block: vhdx - minor comments and typo correction.
block: vhdx - add header update capability.
block: vhdx code movement - VHDXMetadataEntries and BDRVVHDXState to
header.
block: vhdx - log support struct and defines
block: vhdx - break endian translation functions out
block: vhdx - update log guid in header, and first write tracker
block: vhdx code movement - movevhdx_close()abovevhdx_open()
block: vhdx - log parsing, replay, and flush support
block: vhdx - add region overlap detection for image files
block: vhdx - add log write support
block: vhdx write support
block: vhdx - remove BAT file offset bit shifting
block: vhdx - move more endian translations tovhdx-endian.c
block: vhdx - break out code operations to functions
block: vhdx - fix comment typos in header, fix incorrect struct fields
block: vhdx - add .bdrv_create() support
block: vhdx - update_make_test_img()to filter out vhdx options
block: qemu-iotests for vhdx, add write test support
block: vhdx qemu-iotest - log replay of data sectorblock/Makefile.objs | 2 +- block/vhdx-endian.c | 216 ++++ block/vhdx-log.c | 1007 ++++++++++++++++ block/vhdx.c | 1203 +++++++++++++++++--- block/vhdx.h | 178 ++- configure | 24 + tests/qemu-iotests/064 | 11 + tests/qemu-iotests/064.out | 14 + tests/qemu-iotests/065 | 67 ++ tests/qemu-iotests/065.out | 8 + tests/qemu-iotests/common | 1 - tests/qemu-iotests/common.rc | 4 +-
Quoted 8 lines
tests/qemu-iotests/group | 1 + .../sample_images/iotest-dirtylog-10G-4M.vhdx.bz2 | Bin 0 -> 4490 bytes 14 files changed, 2552 insertions(+), 184 deletions(-) create mode 100644 block/vhdx-endian.c create mode 100644 block/vhdx-log.c create mode 100755 tests/qemu-iotests/065 create mode 100644 tests/qemu-iotests/065.out create mode 100644 tests/qemu-iotests/sample_images/iotest-dirtylog-10G-4M.vhdx.bz2
I picked up on a couple smaller issues but overall this is looking good.
Stefan