qcow2: persistent dirty bitmaps
v4 cover qcow2: persistent dirty bitmaps
This series add persistent dirty bitmaps feature to qcow2. Specification is in docs/spec/qcow2.txt (not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)
This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and only hbitmap_deserialize_finish from serialization)
This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so, bdrv_dirty_bitmap_serialize_part and friends are not needed, only hbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO and EXTRA_DATA_COMPATIBLE flags
- some fixes made after John's comments on v3
Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: add qcow2_bitmap_load()
qcow2-dirty-bitmap: add qcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: add qcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: add bdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: add IN_USE flag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: add VM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: add EXTRA_DATA_COMPATIBLE flag
block.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-) create mode 100644 block/qcow2-dirty-bitmap.c create mode 100755 tests/qemu-iotests/160 create mode 100644 tests/qemu-iotests/160.out
Signature · 1 line
1.8.3.1
PATCH v4 1/17 hbitmap: load/store
Add functions for load/store HBitmap to BDS, using clusters table:
Last level of the bitmap is splitted into chunks of 'cluster_size' size. Each cell of the table contains offset in bds, to load/store corresponding chunk.
Also,
0 in cell means all-zeroes-chunk (should not be saved) 1 in cell means all-ones-chunk (should not be saved) hbitmap_prepare_store() fills table with
0 for all-zeroes chunks
1 for all-ones chunks
2 for others
--- block/dirty-bitmap.c | 23 ++++++ include/block/dirty-bitmap.h | 11 +++ include/qemu/hbitmap.h | 12 +++ util/hbitmap.c | 182 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 228 insertions(+)
block/dirty-bitmap.c +23 -0
@@ -491,3 +491,26 @@ int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap) {return hbitmap_count (bitmap->bitmap); } + +int bdrv_dirty_bitmap_load (BdrvDirtyBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size) +{ +return hbitmap_load (bitmap->bitmap, bs, table, table_size, cluster_size); +} + +int bdrv_dirty_bitmap_prepare_store (const BdrvDirtyBitmap *bitmap, +uint32_t cluster_size, +uint64_t *table, +uint32_t *table_size) +{ +return hbitmap_prepare_store (bitmap->bitmap, cluster_size, + table, table_size); +} + +int bdrv_dirty_bitmap_store (const BdrvDirtyBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size) +{ +return hbitmap_store (bitmap->bitmap, bs, table, table_size, cluster_size); +}
include/block/dirty-bitmap.h +11 -0
@@ -68,4 +68,15 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap, bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap); +int bdrv_dirty_bitmap_load (BdrvDirtyBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size); +int bdrv_dirty_bitmap_prepare_store (const BdrvDirtyBitmap *bitmap, +uint32_t cluster_size, +uint64_t *table, +uint32_t *table_size); +int bdrv_dirty_bitmap_store (const BdrvDirtyBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size); +#endif
include/qemu/hbitmap.h +12 -0
@@ -339,5 +339,17 @@ static inline size_t hbitmap_iter_next_word(HBitmapIter *hbi, unsigned long *p_c return hbi->pos; } +typedef struct BlockDriverState BlockDriverState ; + +int hbitmap_load (HBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size); + +int hbitmap_prepare_store (const HBitmap *bitmap,uint32_t cluster_size, +uint64_t *table,uint32_t *table_size); + +int hbitmap_store (HBitmap *bitmap,BlockDriverState *bs, +const uint64_t *table,uint32_t table_size, +uint32_t cluster_size);#endif
util/hbitmap.c +182 -0
191 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v4 2/17 qcow2: Bitmaps extension: structs and consts
In reply to Vladimir Sementsov-OgievskiyAdd data structures and constraints accordingly to docs/specs/qcow2.txt
--- block/Makefile.objs | 2 +- block/qcow2-dirty-bitmap.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 34 +++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 block/qcow2-dirty-bitmap.c
block/Makefile.objs +1 -1
@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-dirty-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
block/qcow2-dirty-bitmap.c new file +47 -0
47 lines not on this page · raw message
block/qcow2.h +34 -0
@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */
@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* QCow2BitmapHeader is actually a bitmap directory entry */ +typedef struct QEMU_PACKED QCow2BitmapHeader { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} QCow2BitmapHeader; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;
@@ -160,6 +180,11 @@ typedef struct QCowSnapshot {uint64_t vm_clock_nsec; } QCowSnapshot; +typedef struct QCow2Bitmap { +uint64_t offset; +char *name; +}QCow2Bitmap ; +struct Qcow2Cache ;typedef struct Qcow2Cache Qcow2Cache ;
@@ -222,6 +247,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of QCow2BitmapHeader, it +* is Qcow2 header extension */ +typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Signature · 1 line
1.8.3.1
PATCH v4 3/17 qcow2-dirty-bitmap: read dirty bitmap directory
In reply to Vladimir Sementsov-OgievskiyAdds qcow2_read_bitmaps, reading bitmap directory as specified in docs/specs/qcow2.txt
--- block/qcow2-dirty-bitmap.c | 163 +++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 10 +++ 2 files changed, 173 insertions(+)
block/qcow2-dirty-bitmap.c +163 -0
@@ -25,6 +25,9 @@* THE SOFTWARE. */ +#include "block/block_int.h" +#include "block/qcow2.h" +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for * _internal_ constants. Please do not use this _internal_ abbreviation for * other needs and/or outside of this file. */
@@ -45,3 +48,163 @@typedef enum BitmapType {BT_DIRTY_TRACKING_BITMAP =1 }BitmapType ; + +void qcow2_free_bitmaps (BlockDriverState *bs) +{ +BDRVQcow2State *s = bs->opaque; +int i; + +for (i =0 ; i < s->nb_bitmaps; i++) { +g_free (s->bitmaps[i].name); + } +g_free (s->bitmaps); + s->bitmaps =NULL ; + s->nb_bitmaps =0 ; + +g_free (s->bitmap_directory); + s->bitmap_directory =NULL ; +} + +static void bitmap_header_to_cpu (QCow2BitmapHeader *h) +{ +be64_to_cpus (&h->bitmap_table_offset); +be32_to_cpus (&h->bitmap_table_size); +be32_to_cpus (&h->flags); +be16_to_cpus (&h->name_size); +be32_to_cpus (&h->extra_data_size); +} + +static int calc_dir_entry_size (size_t name_size) +{ +return align_offset (sizeof (QCow2BitmapHeader) + name_size,8 ); +} + +static int dir_entry_size (QCow2BitmapHeader *h) +{ +return calc_dir_entry_size (h->name_size); +} + +static int check_constraints (QCow2BitmapHeader *h,int cluster_size, +uint64_t disk_size) +{ +uint64_t phys_bitmap_bytes = + (uint64_t )h->bitmap_table_size * cluster_size; +uint64_t max_virtual_bits = (phys_bitmap_bytes *8 ) << h->granularity_bits; + +int fail = + (h->bitmap_table_offset % cluster_size) || + (h->bitmap_table_size >BME_MAX_TABLE_SIZE ) || + (phys_bitmap_bytes >BME_MAX_PHYS_SIZE ) || + (disk_size > max_virtual_bits) || + (h->granularity_bits >BME_MAX_GRANULARITY_BITS ) || + (h->granularity_bits <BME_MIN_GRANULARITY_BITS ) || + (h->flags &BME_RESERVED_FLAGS ) || + (h->name_size >BME_MAX_NAME_SIZE ) || + (h->type !=BT_DIRTY_TRACKING_BITMAP ); + +return fail ? -EINVAL :0 ; +} + +static int directory_read (BlockDriverState *bs,Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +QCow2Bitmap *bm, *end; +int64_t nb_sectors =bdrv_nb_sectors (bs); +size_t offset; + +if (nb_sectors <0 ) { +error_setg (errp,"Can't calculate number of disk sectors." ); +return nb_sectors; + } + +if (s->bitmap_directory !=NULL ) { +/* already read */ +error_setg (errp,"Try read bitmaps, when they are already read." ); +return -EEXIST ; + } + + s->bitmap_directory =g_try_malloc0 (s->bitmap_directory_size); +if (s->bitmap_directory ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); +return -ENOMEM ; + } + + ret =bdrv_pread (bs->file->bs, + s->bitmap_directory_offset, + s->bitmap_directory, + s->bitmap_directory_size); +if (ret <0 ) { +error_setg (errp,"Can't read bitmap directory." ); + goto fail; + } + + offset =0 ; + end = s->bitmaps + s->nb_bitmaps; +for (bm = s->bitmaps; bm < end; ++bm) { +QCow2BitmapHeader *h = + (QCow2BitmapHeader *)(s->bitmap_directory + offset); + +if (offset >= s->bitmap_directory_size) { +error_setg (errp,"Broken bitmap directory." ); + goto fail; + } + +bitmap_header_to_cpu (h); + + ret =check_constraints (h, s->cluster_size, + nb_sectors <<BDRV_SECTOR_BITS ); +if (ret <0 ) { +error_setg (errp,"Bitmap doesn't satisfy the constraints." ); + goto fail; + } + + bm->offset = offset; + bm->name =g_strndup ((char *)(h +1 ), h->name_size); + + offset +=dir_entry_size (h); + } +return 0 ; + +fail: +g_free (s->bitmap_directory); + s->bitmap_directory =NULL ; + +return ret; +} + +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; + +if (s->bitmap_directory !=NULL || s->bitmaps !=NULL ) { +/* already read */ +error_setg (errp,"Try read bitmaps, when they are already read." ); +return -EEXIST ; + } + +if (s->nb_bitmaps ==0 ) { +/* No bitmaps - nothing to do */ +return 0 ; + } + + s->bitmaps =g_try_new0 (QCow2Bitmap, s->nb_bitmaps); +if (s->bitmaps ==NULL ) { +error_setg (errp,"Can't allocate space for qcow2 bitmaps." ); + ret = -ENOMEM ; + goto fail; + } + + ret =directory_read (bs, errp); +if (ret <0 ) { + goto fail; + } + +return 0 ; + +fail: +qcow2_free_bitmaps (bs); + +return ret; +}
block/qcow2.h +10 -0
@@ -297,6 +297,12 @@ typedef struct BDRVQcow2State {unsigned int nb_snapshots;QCowSnapshot *snapshots; +uint64_t bitmap_directory_offset; +uint64_t bitmap_directory_size; +uint8_t *bitmap_directory; +unsigned int nb_bitmaps; +QCow2Bitmap *bitmaps; +int flags;int qcow_version;bool use_lazy_refcounts;
@@ -610,6 +616,10 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,void qcow2_free_snapshots (BlockDriverState *bs);int qcow2_read_snapshots (BlockDriverState *bs); +/* qcow2-dirty-bitmap.c functions */ +void qcow2_free_bitmaps (BlockDriverState *bs); +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
Signature · 1 line
1.8.3.1
PATCH v4 4/17 qcow2-dirty-bitmap: add qcow2_bitmap_load()
In reply to Vladimir Sementsov-OgievskiyThis function loads block dirty bitmap from qcow2.
--- block/qcow2-dirty-bitmap.c | 110 +++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 3 ++ include/block/block_int.h | 4 ++ 4 files changed, 119 insertions(+)
block/qcow2-dirty-bitmap.c +110 -0
@@ -105,6 +105,13 @@ static int check_constraints(QCow2BitmapHeader *h, int cluster_size,return fail ? -EINVAL :0 ; } +static QCow2BitmapHeader *bitmap_header (BDRVQcow2State *s, +QCow2Bitmap *bitmap) +{ +return (QCow2BitmapHeader *) + (s->bitmap_directory + bitmap->offset); +} +static int directory_read (BlockDriverState *bs,Error **errp) {int ret;
@@ -208,3 +215,106 @@ fail:return ret; } + +static QCow2Bitmap *find_bitmap_by_name (BlockDriverState *bs,const char *name) +{ +BDRVQcow2State *s = bs->opaque; +QCow2Bitmap *bm, *end = s->bitmaps + s->nb_bitmaps; + +for (bm = s->bitmaps; bm < end; ++bm) { +if (strcmp (bm->name, name) ==0 ) { +return bm; + } + } + +return NULL ; +} + +/* load_bitmap_data() +* load dirty bitmap from bitmap table +* Bitmap table entries are assumed to be in big endian format */ +static int load_bitmap_data (BlockDriverState *bs,const uint64_t *bitmap_table, +uint32_t bitmap_table_size,BdrvDirtyBitmap *bitmap) +{ +int ret =0 ; +BDRVQcow2State *s = bs->opaque; +uint32_t i; +uint64_t *tab =g_memdup (bitmap_table, + bitmap_table_size *sizeof (uint64_t )); + +for (i =0 ; i < bitmap_table_size; ++i) { +be64_to_cpus (tab + i); + } + + ret =bdrv_dirty_bitmap_load (bitmap, bs->file->bs, tab, bitmap_table_size, + s->cluster_size); + +g_free (tab); + +return ret; +} + +static BdrvDirtyBitmap *load_bitmap (BlockDriverState *bs,QCow2Bitmap *bm, +Error **errp) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +QCow2BitmapHeader *bmh; +uint64_t *bitmap_table =NULL ; +uint32_t granularity; +BdrvDirtyBitmap *bitmap =NULL ; + + bmh =bitmap_header (s, bm); + + bitmap_table =g_try_malloc (bmh->bitmap_table_size *sizeof (uint64_t )); +if (bitmap_table ==NULL ) { +error_setg_errno (errp, -ENOMEM , +"Could not allocate bitmap table" ); +return NULL ; + } + + ret =bdrv_pread (bs->file->bs, bmh->bitmap_table_offset, + bitmap_table, + bmh->bitmap_table_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Could not read bitmap_table table from image" ); + goto fail; + } + + granularity =1U << bmh->granularity_bits; + bitmap =bdrv_create_dirty_bitmap (bs, granularity, bm->name, errp); +if (bitmap ==NULL ) { + goto fail; + } + + ret =load_bitmap_data (bs, bitmap_table, bmh->bitmap_table_size, bitmap); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Could not read bitmap from image" ); + goto fail; + } + +g_free (bitmap_table); +return bitmap; + +fail: +g_free (bitmap_table); +bdrv_release_dirty_bitmap (bs, bitmap); + +return NULL ; +} + +BdrvDirtyBitmap *qcow2_bitmap_load (BlockDriverState *bs,const char *name, +Error **errp) +{ +QCow2Bitmap *bm; + + bm =find_bitmap_by_name (bs, name); +if (bm ==NULL ) { +error_setg (errp,"Could not find bitmap '%s' in the node '%s'" , name, +bdrv_get_device_or_node_name (bs)); +return NULL ; + } + +return load_bitmap (bs, bm, errp); +}
block/qcow2.c +2 -0
@@ -3346,6 +3346,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_get_info = qcow2_get_info, .bdrv_get_specific_info = qcow2_get_specific_info, + .bdrv_dirty_bitmap_load = qcow2_bitmap_load, + .bdrv_save_vmstate = qcow2_save_vmstate, .bdrv_load_vmstate = qcow2_load_vmstate,
block/qcow2.h +3 -0
@@ -620,6 +620,9 @@ int qcow2_read_snapshots(BlockDriverState *bs);void qcow2_free_bitmaps (BlockDriverState *bs);int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +BdrvDirtyBitmap *qcow2_bitmap_load (BlockDriverState *bs,const char *name, +Error **errp); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
include/block/block_int.h +4 -0
@@ -214,6 +214,10 @@ struct BlockDriver {int (*bdrv_get_info)(BlockDriverState *bs,BlockDriverInfo *bdi); ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs); +BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs, +const char *name, +Error **errp); +int (*bdrv_save_vmstate)(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos); int (*bdrv_load_vmstate)(BlockDriverState *bs, uint8_t *buf,
Signature · 1 line
1.8.3.1
PATCH v4 5/17 qcow2-dirty-bitmap: add qcow2_bitmap_store()
In reply to Vladimir Sementsov-OgievskiyThis function stores block dirty bitmap to qcow2. If the bitmap with the same name, size and granularity already exists, it will be rewritten, if the bitmap with the same name exists but granularity or size does not match, an error will be genrated.
--- block/qcow2-dirty-bitmap.c | 443 +++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/qcow2.h | 2 + include/block/block_int.h | 3 + 4 files changed, 449 insertions(+)
block/qcow2-dirty-bitmap.c +443 -0
@@ -74,6 +74,15 @@ static void bitmap_header_to_cpu(QCow2BitmapHeader *h)be32_to_cpus (&h->extra_data_size); } +static void bitmap_header_to_be (QCow2BitmapHeader *h) +{ +cpu_to_be64s (&h->bitmap_table_offset); +cpu_to_be32s (&h->bitmap_table_size); +cpu_to_be32s (&h->flags); +cpu_to_be16s (&h->name_size); +cpu_to_be32s (&h->extra_data_size); +} +static int calc_dir_entry_size (size_t name_size) {return align_offset (sizeof (QCow2BitmapHeader) + name_size,8 );
@@ -84,6 +93,17 @@ static int dir_entry_size(QCow2BitmapHeader *h)return calc_dir_entry_size (h->name_size); } +static void directory_to_be (uint8_t *dir,size_t size) +{ +uint8_t *end = dir + size; +while (dir < end) { +QCow2BitmapHeader *h = (QCow2BitmapHeader *)dir; + dir +=dir_entry_size (h); + +bitmap_header_to_be (h); + } +} +static int check_constraints (QCow2BitmapHeader *h,int cluster_size,uint64_t disk_size) {
@@ -318,3 +338,426 @@ BdrvDirtyBitmap *qcow2_bitmap_load(BlockDriverState *bs, const char *name,return load_bitmap (bs, bm, errp); } + +static int update_header_sync (BlockDriverState *bs) +{ +int ret; + + ret =qcow2_update_header (bs); +if (ret <0 ) { +return ret; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { +return ret; + } + +return 0 ; +} + +/* write bitmap directory from the state to new allocated clusters */ +static int64_t directory_write (BlockDriverState *bs,const uint8_t *dir, +size_t size) +{ +int ret =0 ; +uint8_t *dir_be =NULL ; +int64_t dir_offset =0 ; + + dir_be =g_try_malloc (size); +if (dir_be ==NULL ) { +return -ENOMEM ; + } +memcpy (dir_be, dir, size); +directory_to_be (dir_be, size); + +/* Allocate space for the new bitmap directory */ + dir_offset =qcow2_alloc_clusters (bs, size); +if (dir_offset <0 ) { + ret = dir_offset; + goto out; + } + +/* The bitmap directory position has not yet been updated, so these +* clusters must indeed be completely free */ + ret =qcow2_pre_write_overlap_check (bs,0 , dir_offset, size); +if (ret <0 ) { + goto out; + } + + ret =bdrv_pwrite (bs->file->bs, dir_offset, dir_be, size); +if (ret <0 ) { + goto out; + } + +out: +g_free (dir_be); + +if (ret <0 ) { +if (dir_offset >0 ) { +qcow2_free_clusters (bs, dir_offset, size,QCOW2_DISCARD_ALWAYS ); + } + +return ret; + } + +return dir_offset; +} + +static int directory_push_entry (BlockDriverState *bs,QCow2BitmapHeader *header) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +int entry_size =dir_entry_size (header); +int64_t new_offset =0 , old_offset =0 ; +uint64_t new_size = s->bitmap_directory_size + entry_size, old_size =0 ; +void *p; +int64_t nb_sectors =bdrv_nb_sectors (bs); + +if (nb_sectors <0 ) { +return nb_sectors; + } + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + } + + ret =check_constraints (header, s->cluster_size, + nb_sectors <<BDRV_SECTOR_BITS ); +if (ret <0 ) { +return -EINVAL ; + } + + old_offset = s->bitmap_directory_offset; + old_size = s->bitmap_directory_size; + +uint8_t *new_dir =g_try_malloc (new_size); +if (new_dir ==NULL ) { +return -ENOMEM ; + } +memcpy (new_dir, s->bitmap_directory, s->bitmap_directory_size); +memcpy (new_dir + s->bitmap_directory_size, header, entry_size); + + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { + ret = new_offset; + goto fail; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + } + +g_free (s->bitmap_directory); + s->bitmap_directory = new_dir; + +return 0 ; + +fail: +g_free (new_dir); +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + } + + p =g_try_realloc (s->bitmap_directory, s->bitmap_directory_size); +if (p !=NULL ) { + s->bitmap_directory = p; + } + +return ret; +} + +/* store_bitmap() +* update bitmap table by storing bitmap to it. +* Bitmap table entries are assumed to be in big endian format +* On the error, the resulting bitmap table is valid for clearing, but +* may contain invalid bitmap */ +static int store_bitmap (BlockDriverState *bs,uint64_t *bitmap_table, +uint32_t bitmap_table_size, +const BdrvDirtyBitmap *bitmap) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +int cl_size = s->cluster_size; +uint32_t i, tab_size =0 ; +uint64_t *tab; +uint32_t j; + +bdrv_dirty_bitmap_prepare_store (bitmap, s->cluster_size,NULL , &tab_size); + tab =g_try_malloc (tab_size *sizeof (tab[0 ])); +if (tab ==NULL ) { +return -ENOMEM ; + } + + ret =bdrv_dirty_bitmap_prepare_store (bitmap, s->cluster_size, + tab, &tab_size); +if (ret !=0 ) { +g_free (tab); +return ret; + } + +for (i =0 , j =0 ; i < bitmap_table_size; ++i) { +if (tab[i] <=1 ) { +continue ; + } + +for ( ; j < bitmap_table_size && bitmap_table[j] <=1 ; ++j) { + ; + } + +if (j < bitmap_table_size) { + tab[i] =be64_to_cpu (bitmap_table[j]); + }else { + tab[i] =qcow2_alloc_clusters (bs, cl_size); + } + } + +for ( ; j < bitmap_table_size; ++j) { +uint64_t addr =be64_to_cpu (bitmap_table[j]); +if (addr <=1 ) { +continue ; + } + +qcow2_free_clusters (bs, addr, cl_size,QCOW2_DISCARD_ALWAYS ); + bitmap_table[j] =0 ; + } + +bdrv_dirty_bitmap_store (bitmap, bs->file->bs, tab, + bitmap_table_size, s->cluster_size); + +for (i =0 ; i < bitmap_table_size; ++i) { + bitmap_table[i] =cpu_to_be64 (tab[i]); + } + +g_free (tab); + +return 0 ; +} + +static int64_t alloc_zeroed_clusters (BlockDriverState *bs,uint64_t size) +{ +int ret =0 ; +void *buf =NULL ; +int64_t offset =qcow2_alloc_clusters (bs, size); +if (offset <0 ) { +return offset; + } + + buf =g_try_malloc0 (size); +if (buf ==NULL ) { + ret = -ENOMEM ; + goto out; + } + + ret =qcow2_pre_write_overlap_check (bs,0 , offset, size); +if (ret <0 ) { + goto out; + } + + ret =bdrv_pwrite (bs->file->bs, offset, buf, size); +if (ret <0 ) { + goto out; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto out; + } + +out: +g_free (buf); + +if (ret <0 ) { +qcow2_free_clusters (bs, offset, size,QCOW2_DISCARD_ALWAYS ); +return ret; + } + +return offset; +} + +static int directory_push (BlockDriverState *bs,const char *name, +uint64_t size,int granularity) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +int sector_granularity = granularity >>BDRV_SECTOR_BITS ; +size_t name_size =strlen (name); +size_t entry_size =calc_dir_entry_size (name_size); +QCow2BitmapHeader *entry =g_malloc0 (entry_size); +int64_t table_offset =0 ; + + entry->granularity_bits =ctz32 (granularity); + entry->type =BT_DIRTY_TRACKING_BITMAP ; + entry->name_size = name_size; +memcpy (entry +1 , name, name_size); + + entry->bitmap_table_size = +size_to_clusters (s, (((size -1 ) / sector_granularity) >>3 ) +1 ); + table_offset =alloc_zeroed_clusters (bs, entry->bitmap_table_size * +sizeof (uint64_t )); +if (table_offset <0 ) { + ret = table_offset; + goto out; + } + entry->bitmap_table_offset = table_offset; + + ret =directory_push_entry (bs, entry); +if (ret <0 ) { + goto out; + } + +out: +g_free (entry); +if (ret <0 && table_offset >0 ) { +qcow2_free_clusters (bs, table_offset, entry->bitmap_table_size * +sizeof (uint64_t ),QCOW2_DISCARD_ALWAYS ); + } + +return ret; +} + +static int bitmaps_push (BDRVQcow2State *s,const char *name,uint64_t offset) +{ +QCow2Bitmap *bm; +QCow2Bitmap *p; + +printf ("dirty bitmaps push\n" ); + p =g_try_renew (QCow2Bitmap, s->bitmaps, s->nb_bitmaps +1 ); +if (p ==NULL ) { +return -ENOMEM ; + } + s->bitmaps = p; + s->nb_bitmaps++; + + bm = s->bitmaps + s->nb_bitmaps -1 ; + bm->name =g_strdup (name); + bm->offset = offset; + + +return 0 ; +} + +static void bitmaps_pop (BDRVQcow2State *s) +{ +QCow2Bitmap *p; + +if (s->nb_bitmaps ==0 ) { +return ; + } + + p =g_try_renew (QCow2Bitmap, s->bitmaps, s->nb_bitmaps -1 ); +if (p !=NULL ) { + s->bitmaps = p; + } + + s->nb_bitmaps--; +} + +/* if no id is provided, a new one is constructed */ +static int qcow2_bitmap_create (BlockDriverState *bs,const char *name, +uint64_t size,int granularity) +{ +int ret; +BDRVQcow2State *s = bs->opaque; + +if (s->nb_bitmaps >=QCOW_MAX_DIRTY_BITMAPS ) { +return -EFBIG ; + } + +/* Check that the name is unique */ +if (find_bitmap_by_name (bs, name) !=NULL ) { +return -EEXIST ; + } + + ret =bitmaps_push (s, name, s->bitmap_directory_size); +if (ret <0 ) { +return ret; + } + + ret =directory_push (bs, name, size, granularity); +if (ret <0 ) { +bitmaps_pop (s); +return ret; + } + +return 0 ; +} + +void qcow2_bitmap_store (BlockDriverState *bs, +const BdrvDirtyBitmap *bitmap,Error **errp) +{ +BDRVQcow2State *s = bs->opaque; +int ret =0 ; +uint64_t *bitmap_table; +QCow2Bitmap *bm; +QCow2BitmapHeader *bmh; +const char *name =bdrv_dirty_bitmap_name (bitmap); +uint64_t size =bdrv_dirty_bitmap_size (bitmap); +int granularity =bdrv_dirty_bitmap_granularity (bitmap); + +/* find/create dirty bitmap */ + bm =find_bitmap_by_name (bs, name); +if (bm ==NULL ) { + ret =qcow2_bitmap_create (bs, name, size, granularity); +if (ret <0 ) { +error_setg_errno (errp, ret,"Can't create dirty bitmap in qcow2." ); + } + bm = s->bitmaps + s->nb_bitmaps -1 ; + bmh =bitmap_header (s, bm); + }else { + bmh =bitmap_header (s, bm); + +if (granularity != (1U << bmh->granularity_bits)) { +error_setg (errp, +"The bitmap with same name (but other granularity) " +"already exists." ); +return ; + } + } + + bitmap_table =g_try_new (uint64_t, bmh->bitmap_table_size); +if (bitmap_table ==NULL ) { +error_setg (errp,"No memory." ); +return ; + } + ret =bdrv_pread (bs->file->bs, bmh->bitmap_table_offset, + bitmap_table, + bmh->bitmap_table_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, ret,"Can't read dirty bitmap table." ); + goto finish; + } + + ret =store_bitmap (bs, bitmap_table, bmh->bitmap_table_size, + bitmap); +if (ret <0 ) { +error_setg_errno (errp, ret,"Can't store bitmap table." ); + goto finish; + } + + ret =bdrv_pwrite (bs->file->bs, bmh->bitmap_table_offset, + bitmap_table, + bmh->bitmap_table_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, ret,"Can't write dirty bitmap table." ); + goto finish; + } + +finish: +g_free (bitmap_table); +}
block/qcow2.c +1 -0
@@ -3347,6 +3347,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_get_specific_info = qcow2_get_specific_info, .bdrv_dirty_bitmap_load = qcow2_bitmap_load, + .bdrv_dirty_bitmap_store = qcow2_bitmap_store, .bdrv_save_vmstate = qcow2_save_vmstate, .bdrv_load_vmstate = qcow2_load_vmstate,
block/qcow2.h +2 -0
@@ -622,6 +622,8 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp);BdrvDirtyBitmap *qcow2_bitmap_load (BlockDriverState *bs,const char *name,Error **errp); +void qcow2_bitmap_store (BlockDriverState *bs,const BdrvDirtyBitmap *bitmap, +Error **errp);/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);
include/block/block_int.h +3 -0
@@ -217,6 +217,9 @@ struct BlockDriver {BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs,const char *name,Error **errp); +void (*bdrv_dirty_bitmap_store)(BlockDriverState *bs, +const BdrvDirtyBitmap *bitmap, +Error **errp);int (*bdrv_save_vmstate)(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos);
Signature · 1 line
1.8.3.1
PATCH v4 6/17 qcow2: add dirty bitmaps extension
In reply to Vladimir Sementsov-OgievskiyAdd dirty bitmap extension as specified in docs/specs/qcow2.txt.
Load bitmap headers on open. Handle close and update_header.
Handle resize: for now, just block resize if there are dirty bitmaps.
--- block/qcow2.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
block/qcow2.c +89 -0
@@ -61,6 +61,7 @@ typedef struct {#define QCOW2_EXT_MAGIC_END 0#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 +#define QCOW2_EXT_MAGIC_DIRTY_BITMAPS 0x23852875static int qcow2_probe (const uint8_t *buf,int buf_size,const char *filename) {
@@ -90,6 +91,7 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,QCowExtension ext;uint64_t offset;int ret; +Qcow2BitmapHeaderExt bitmaps_ext;#ifdef DEBUG_EXT printf ("qcow2_read_extensions: start=%ld end=%ld\n" , start_offset, end_offset);
@@ -160,6 +162,67 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, }break ; +case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : + ret =bdrv_pread (bs->file->bs, offset, &bitmaps_ext, ext.len); +if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." ); +return -EINVAL ; + } + +be32_to_cpus (&bitmaps_ext.nb_bitmaps); +be64_to_cpus (&bitmaps_ext.bitmap_directory_size); +be64_to_cpus (&bitmaps_ext.bitmap_directory_offset); + +if (bitmaps_ext.nb_bitmaps >QCOW_MAX_DIRTY_BITMAPS ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too many dirty bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.nb_bitmaps ==0 ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"found bitmaps extension with zero bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size -1 )) { +error_setg (errp,"ERROR: bitmaps_ext: " +"wrong dirty bitmap directory offset" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_size > +QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too large dirty bitmap directory" ); +return -EINVAL ; + } + + s->nb_bitmaps = bitmaps_ext.nb_bitmaps; + s->bitmap_directory_offset = + bitmaps_ext.bitmap_directory_offset; + s->bitmap_directory_size = + bitmaps_ext.bitmap_directory_size; + + ret =qcow2_read_bitmaps (bs, errp); +if (ret <0 ) { +return ret; + } + +#ifdef DEBUG_EXT +printf ("Qcow2: Got dirty bitmaps extension:" +" offset=%" PRIu64" nb_bitmaps=%" PRIu32"\n" , + s->bitmaps_offset, s->nb_bitmaps); +#endif +break ; +default :/* unknown magic - save it in case we need to rewrite the header */ {
@@ -1177,6 +1240,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,g_free (s->unknown_header_fields);cleanup_unknown_header_ext (bs);qcow2_free_snapshots (bs); +qcow2_free_bitmaps (bs);qcow2_refcount_close (bs);qemu_vfree (s->l1_table);/* else pre-write overlap checks in cache_destroy may crash */
@@ -1740,6 +1804,7 @@ static void qcow2_close(BlockDriverState *bs)qemu_vfree (s->cluster_data);qcow2_refcount_close (bs);qcow2_free_snapshots (bs); +qcow2_free_bitmaps (bs); }static void qcow2_invalidate_cache (BlockDriverState *bs,Error **errp)
@@ -1937,6 +2002,24 @@ int qcow2_update_header(BlockDriverState *bs) buflen -= ret; } +if (s->nb_bitmaps >0 ) { +Qcow2BitmapHeaderExt bitmaps_header = { + .nb_bitmaps =cpu_to_be32 (s->nb_bitmaps), + .bitmap_directory_size = +cpu_to_be64 (s->bitmap_directory_size), + .bitmap_directory_offset = +cpu_to_be64 (s->bitmap_directory_offset) + }; + ret =header_ext_add (buf,QCOW2_EXT_MAGIC_DIRTY_BITMAPS , + &bitmaps_header,sizeof (bitmaps_header), + buflen); +if (ret <0 ) { + goto fail; + } + buf += ret; + buflen -= ret; + } +/* Keep unknown header extensions */ QLIST_FOREACH(uext, &s->unknown_header_ext, next) { ret =header_ext_add (buf, uext->magic, uext->data, uext->len, buflen);
@@ -2460,6 +2543,12 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)return -ENOTSUP ; } +/* cannot proceed if image has bitmaps */ +if (s->nb_bitmaps) { +error_report ("Can't resize an image which has dirty bitmaps" ); +return -ENOTSUP ; + } +/* shrinking is currently not supported */ if (offset < bs->total_sectors *512 ) {error_report ("qcow2 doesn't support shrinking images yet" );
Signature · 1 line
1.8.3.1
PATCH v4 7/17 qcow2-dirty-bitmap: add qcow2_bitmap_load_check()
In reply to Vladimir Sementsov-OgievskiyThe function checks existing of the bitmap without loading it.
--- block/dirty-bitmap.c | 15 +++++++++++++++ block/qcow2-dirty-bitmap.c | 5 +++++ block/qcow2.c | 1 + block/qcow2.h | 1 + include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 1 + 6 files changed, 25 insertions(+)
block/dirty-bitmap.c +15 -0
@@ -514,3 +514,18 @@ int bdrv_dirty_bitmap_store(const BdrvDirtyBitmap *bitmap, BlockDriverState *bs, {return hbitmap_store (bitmap->bitmap, bs, table, table_size, cluster_size); } + +bool bdrv_load_check_dirty_bitmap (BlockDriverState *file,const char *name) +{ +BlockDriver *drv = file->drv; +if (!drv) { +return false; + } +if (drv->bdrv_dirty_bitmap_load_check) { +return drv->bdrv_dirty_bitmap_load_check (file, name); + } +if (file->file) { +return bdrv_load_check_dirty_bitmap (file->file->bs, name); + } +return false; +}
block/qcow2-dirty-bitmap.c +5 -0
@@ -274,6 +274,11 @@ static int load_bitmap_data(BlockDriverState *bs, const uint64_t *bitmap_table,return ret; } +bool qcow2_bitmap_load_check (BlockDriverState *file,const char *name) +{ +return find_bitmap_by_name (file, name) !=NULL ; +} +static BdrvDirtyBitmap *load_bitmap (BlockDriverState *bs,QCow2Bitmap *bm,Error **errp) {
block/qcow2.c +1 -0
@@ -3436,6 +3436,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_get_specific_info = qcow2_get_specific_info, .bdrv_dirty_bitmap_load = qcow2_bitmap_load, + .bdrv_dirty_bitmap_load_check = qcow2_bitmap_load_check, .bdrv_dirty_bitmap_store = qcow2_bitmap_store, .bdrv_save_vmstate = qcow2_save_vmstate,
block/qcow2.h +1 -0
@@ -620,6 +620,7 @@ int qcow2_read_snapshots(BlockDriverState *bs);void qcow2_free_bitmaps (BlockDriverState *bs);int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +bool qcow2_bitmap_load_check (BlockDriverState *file,const char *name);BdrvDirtyBitmap *qcow2_bitmap_load (BlockDriverState *bs,const char *name,Error **errp); void qcow2_bitmap_store(BlockDriverState *bs, const BdrvDirtyBitmap *bitmap,
include/block/block_int.h +2 -0
@@ -214,6 +214,8 @@ struct BlockDriver {int (*bdrv_get_info)(BlockDriverState *bs,BlockDriverInfo *bdi); ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs); +bool (*bdrv_dirty_bitmap_load_check)(BlockDriverState *file, +const char *name);BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs,const char *name,Error **errp);
include/block/dirty-bitmap.h +1 -0
@@ -78,5 +78,6 @@ int bdrv_dirty_bitmap_prepare_store(const BdrvDirtyBitmap *bitmap,int bdrv_dirty_bitmap_store (const BdrvDirtyBitmap *bitmap,BlockDriverState *bs,const uint64_t *table,uint32_t table_size,uint32_t cluster_size); +bool bdrv_load_check_dirty_bitmap (BlockDriverState *file,const char *name);#endif
Signature · 1 line
1.8.3.1
PATCH v4 8/17 block: store persistent dirty bitmaps
In reply to Vladimir Sementsov-OgievskiyPersistent dirty bitmaps are the bitmaps, for which the new field BdrvDirtyBitmap.file is not NULL. We save all persistent dirty bitmaps owned by BlockDriverState in corresponding bdrv_close(). BdrvDirtyBitmap.file is a BlockDriverState, where we want to save the bitmap. It may be set in bdrv_dirty_bitmap_set_file() only once. bdrv_ref/bdrv_unref are used for BdrvDirtyBitmap.file to be sure that files will be closed and resources will be freed.
--- block.c | 3 +++ block/dirty-bitmap.c | 35 +++++++++++++++++++++++++++++++++++ include/block/dirty-bitmap.h | 5 +++++ 3 files changed, 43 insertions(+)
block.c +3 -0
@@ -2143,6 +2143,9 @@ void bdrv_close(BlockDriverState *bs)blk_dev_change_media_cb (bs->blk, false); } +/* save and release persistent dirty bitmaps */ +bdrv_finalize_persistent_dirty_bitmaps (bs); +if (bs->drv) {BdrvChild *child, *next;
block/dirty-bitmap.c +35 -0
@@ -43,6 +43,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool internal_persistent;/* bitmap must be saved to owner disk image */ QLIST_ENTRY (BdrvDirtyBitmap) list; };
@@ -529,3 +530,37 @@ bool bdrv_load_check_dirty_bitmap(BlockDriverState *file, const char *name) }return false; } + +void bdrv_store_dirty_bitmap (BlockDriverState *bs,BdrvDirtyBitmap *bitmap, +Error **errp) +{ +if (bs ==NULL || bs->drv ==NULL || + bs->drv->bdrv_dirty_bitmap_store ==NULL ) { +error_setg (errp,"Storing bitmap is unsupported for the format." ); +return ; + } + + bs->drv->bdrv_dirty_bitmap_store (bs, bitmap, errp); +} + +void bdrv_dirty_bitmap_set_internal_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->internal_persistent = persistent; +} + +void bdrv_finalize_persistent_dirty_bitmaps (BlockDriverState *bs) +{ +BdrvDirtyBitmap *bm, *bm_next; + + QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, bm_next) { +if (bm->internal_persistent) { +Error *local_err =NULL ; +bdrv_store_dirty_bitmap (bs, bm, &local_err); +if (local_err) { +error_report_err (local_err); + } +bdrv_release_dirty_bitmap (bs, bm); + } + } +}
include/block/dirty-bitmap.h +5 -0
@@ -79,5 +79,10 @@ int bdrv_dirty_bitmap_store(const BdrvDirtyBitmap *bitmap, BlockDriverState *bs, const uint64_t *table, uint32_t table_size, uint32_t cluster_size);bool bdrv_load_check_dirty_bitmap (BlockDriverState *file,const char *name); +void bdrv_dirty_bitmap_set_internal_persistance (BdrvDirtyBitmap *bitmap, +bool persistent); +void bdrv_store_dirty_bitmap (BlockDriverState *bs,BdrvDirtyBitmap *bitmap, +Error **errp); +void bdrv_finalize_persistent_dirty_bitmaps (BlockDriverState *bs);#endif
Signature · 1 line
1.8.3.1
PATCH v4 9/17 block: add bdrv_load_dirty_bitmap()
In reply to Vladimir Sementsov-OgievskiyThe funcion loads dirty bitmap from file, using underlying driver function.
Note: the function doesn't change BdrvDirtyBitmap.file field. This field is only used by bdrv_store_dirty_bitmap() function and is ONLY written by bdrv_dirty_bitmap_set_file() function.
--- block/dirty-bitmap.c | 16 ++++++++++++++++ include/block/dirty-bitmap.h | 2 ++ 2 files changed, 18 insertions(+)
block/dirty-bitmap.c +16 -0
@@ -564,3 +564,19 @@ void bdrv_finalize_persistent_dirty_bitmaps(BlockDriverState *bs) } } } + +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp) +{ +BlockDriver *drv = bs->drv; +if (!drv) { +return NULL ; + } +if (drv->bdrv_dirty_bitmap_load) { +return drv->bdrv_dirty_bitmap_load (bs, name, errp); + } +if (bs->file) { +return bdrv_load_dirty_bitmap (bs, name, errp); + } +return NULL ; +}
include/block/dirty-bitmap.h +2 -0
@@ -84,5 +84,7 @@ void bdrv_dirty_bitmap_set_internal_persistance(BdrvDirtyBitmap *bitmap,void bdrv_store_dirty_bitmap (BlockDriverState *bs,BdrvDirtyBitmap *bitmap,Error **errp);void bdrv_finalize_persistent_dirty_bitmaps (BlockDriverState *bs); +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp);#endif
Signature · 1 line
1.8.3.1
PATCH v4 10/17 qcow2-dirty-bitmap: add autoclear bit
In reply to Vladimir Sementsov-OgievskiyAdd autoclear bit for handling rewriting image by old qemu version.
If autoclear bit is not set, but bitmaps extension is found it would not be loaded and warning will be generated.
--- block/qcow2-dirty-bitmap.c | 4 ++++ block/qcow2.c | 12 ++++++++++-- block/qcow2.h | 9 +++++++++ 3 files changed, 23 insertions(+), 2 deletions(-)
block/qcow2-dirty-bitmap.c +4 -0
@@ -417,6 +417,7 @@ static int directory_push_entry(BlockDriverState *bs, QCow2BitmapHeader *header)int64_t new_offset =0 , old_offset =0 ;uint64_t new_size = s->bitmap_directory_size + entry_size, old_size =0 ;void *p; +uint64_t old_autocl;int64_t nb_sectors =bdrv_nb_sectors (bs);if (nb_sectors <0 ) {
@@ -435,6 +436,7 @@ static int directory_push_entry(BlockDriverState *bs, QCow2BitmapHeader *header) old_offset = s->bitmap_directory_offset; old_size = s->bitmap_directory_size; + old_autocl = s->autoclear_features;uint8_t *new_dir =g_try_malloc (new_size);if (new_dir ==NULL ) {
@@ -456,6 +458,7 @@ static int directory_push_entry(BlockDriverState *bs, QCow2BitmapHeader *header) s->bitmap_directory_offset = new_offset; s->bitmap_directory_size = new_size; + s->autoclear_features |=QCOW2_AUTOCLEAR_DIRTY_BITMAPS ; ret =update_header_sync (bs);if (ret <0 ) {
@@ -477,6 +480,7 @@ fail:qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); s->bitmap_directory_offset = old_offset; s->bitmap_directory_size = old_size; + s->autoclear_features = old_autocl; } p =g_try_realloc (s->bitmap_directory, s->bitmap_directory_size);
block/qcow2.c +10 -2
@@ -163,6 +163,13 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,break ;case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : +if (!(s->autoclear_features &QCOW2_AUTOCLEAR_DIRTY_BITMAPS )) { +fprintf (stderr, +"WARNING: bitmaps_ext: autoclear flag is not " +"set, all bitmaps will be considered as inconsistent" ); +break ; + } + ret =bdrv_pread (bs->file->bs, offset, &bitmaps_ext, ext.len);if (ret <0 ) { error_setg_errno(errp, -ret, "ERROR: bitmaps_ext: "
@@ -1204,8 +1211,9 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Clear unknown autoclear feature bits */ -if (!bs->read_only && !(flags &BDRV_O_INACTIVE ) && s->autoclear_features) { - s->autoclear_features =0 ; +if (!bs->read_only && !(flags &BDRV_O_INACTIVE ) && + (s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK )) { + s->autoclear_features &=QCOW2_AUTOCLEAR_MASK ; ret =qcow2_update_header (bs);if (ret <0 ) {error_setg_errno (errp, -ret,"Could not update qcow2 header" );
block/qcow2.h +9 -0
@@ -220,6 +220,15 @@ enum { QCOW2_COMPAT_FEAT_MASK = QCOW2_COMPAT_LAZY_REFCOUNTS, }; +/* Autoclear feature bits */ +enum { +QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR =0 , +QCOW2_AUTOCLEAR_DIRTY_BITMAPS = +1 <<QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR , + +QCOW2_AUTOCLEAR_MASK =QCOW2_AUTOCLEAR_DIRTY_BITMAPS , +}; +enum qcow2_discard_type {QCOW2_DISCARD_NEVER =0 ,QCOW2_DISCARD_ALWAYS ,
Signature · 1 line
1.8.3.1
PATCH v4 11/17 qemu: command line option for dirty bitmaps
In reply to Vladimir Sementsov-OgievskiyThe patch adds the following command line option:
-dirty-bitmap [option1=val1][,option2=val2]...
Avaliable options are:
name
The name of the bitmap.
Should be unique per 'file'/'node' and per 'for_node'.
node
The node to load and bind the bitmap.
It should be specified as 'id' suboption of one of '-node' options.
granularity
Granularity (in bytes) for created dirty bitmap.
If the bitmap is already exists in specified 'file'/'file_id'/device it's granularity will not be changed but only checked (an error will be generated if this check fails).
enabled
on|off
Enabled flag for the bitmap.
By default the bitmap will be enabled.
create
on|off
By default is off.
If on, then new bitmap will be created in the image, if the bitmap with same name is already exists an error will be generated.
If off, then the bitmap will be loaded from the image, if there is no one an error will be generated.
If create=off and granularity is specified then granularity will be checked for loaded bitmap and if not match an error will be generated.
--- blockdev.c | 36 ++++++++++++++++++++++ include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 +++++++++++++++++++++ vl.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 151 insertions(+)
blockdev.c +36 -0
@@ -181,6 +181,12 @@ QemuOpts *drive_def(const char *optstr)return qemu_opts_parse_noisily (qemu_find_opts ("drive" ), optstr, false); } +QemuOpts *dirty_bitmap_def (const char *optstr) +{ +return qemu_opts_parse_noisily (qemu_find_opts ("dirty-bitmap" ), optstr, + false); +} +QemuOpts *drive_add (BlockInterfaceType type,int index,const char *file,const char *optstr) {
@@ -3953,6 +3959,36 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)return head; } +QemuOptsList qemu_dirty_bitmap_opts = { + .name ="dirty-bitmap" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_dirty_bitmap_opts.head), + .desc = { + { + .name ="name" , + .type =QEMU_OPT_STRING , + .help ="Name of the dirty bitmap" , + },{ + .name ="node" , + .type =QEMU_OPT_STRING , + .help ="node name to bind the bitmap to (and load it from it)" , + },{ + .name ="granularity" , + .type =QEMU_OPT_NUMBER , + .help ="granularity" , + },{ + .name ="enabled" , + .type =QEMU_OPT_BOOL , + .help ="enabled flag (default is 'on')" , + },{ + .name ="create" , + .type =QEMU_OPT_BOOL , + .help ="create flag (default is 'off'), " +"if on, new dirty bitmap will be created, " +"else the existing one will be loaded" + } + } +}; +QemuOptsList qemu_common_drive_opts = { .name ="drive" , .head =QTAILQ_HEAD_INITIALIZER (qemu_common_drive_opts.head),
include/sysemu/blockdev.h +1 -0
@@ -57,6 +57,7 @@ int drive_get_max_devs(BlockInterfaceType type);DriveInfo *drive_get_next (BlockInterfaceType type);QemuOpts *drive_def (const char *optstr); +QemuOpts *dirty_bitmap_def (const char *optstr);QemuOpts *drive_add (BlockInterfaceType type,int index,const char *file,const char *optstr);DriveInfo *drive_new (QemuOpts *arg,BlockInterfaceType block_default_type);
include/sysemu/sysemu.h +1 -0
@@ -248,6 +248,7 @@ bool usb_enabled(void);extern QemuOptsList qemu_legacy_drive_opts;extern QemuOptsList qemu_common_drive_opts; +extern QemuOptsList qemu_dirty_bitmap_opts;extern QemuOptsList qemu_drive_opts;extern QemuOptsList qemu_chardev_opts;extern QemuOptsList qemu_device_opts;
qemu-options.hx +35 -0
@@ -679,6 +679,41 @@ qemu-system-i386 -hda a -hdb b @end example ETEXI +DEF("dirty-bitmap", HAS_ARG, QEMU_OPTION_dirty_bitmap, + "-dirty-bitmap name=name,node=@var{id}\n" + " [,granularity=granularity][,enabled=on|off][,create=on|off]\n", + QEMU_ARCH_ALL) +STEXI +@item -dirty-bitmap @var{option}[,@var{option}[,@var{option}[,...]]] +@findex -dirty-bitmap + +Define a dirty-bitmap. Valid options are: + +@table @option +@item name=@var{name} +The name of the bitmap. Should be unique per @var{file}/@var{node} and per +@var{for_node}. +@item node=@var{node} +The node to load and bind the bitmap. It should be specified as @var{id} suboption +of one of @option{-node} options. +@item granularity=@var{granularity} +Granularity (in bytes) for created dirty bitmap. If the bitmap is already +exists in specified @var{file}/@var{file_id}/@var{device} it's granularity will +not be changed but only checked (an error will be generated if this check +fails). +@item enabled=@var{enabled} +Enabled flag for the bitmap. By default the bitmap will be enabled. +@item create=@var{create} +By default is off. +If on, then new bitmap will be created in the image, if the bitmap with same +name is already exists an error will be generated. +If off, then the bitmap will be loaded from the image, if there is no one an +error will be generated. +If create=off and granularity is specified then granularity will be checked for +loaded bitmap and if not match an error will be generated. +@end table +ETEXI + DEF("mtdblock", HAS_ARG, QEMU_OPTION_mtdblock, "-mtdblock file use 'file' as on-board Flash memory image\n", QEMU_ARCH_ALL)
vl.c +78 -0
102 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v4 12/17 qcow2-dirty-bitmap: add IN_USE flag
In reply to Vladimir Sementsov-OgievskiyThis flag is set on bitmap load and unset on store. If it is already set when loading the bitmap, the bitmap should not be load (it is in use by other drive or it is inconsistent (was not successfully saved))
--- block/qcow2-dirty-bitmap.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-)
block/qcow2-dirty-bitmap.c +44 -1
@@ -40,7 +40,8 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xffffffff +#define BME_RESERVED_FLAGS 0xfffffffe +#define BME_FLAG_IN_USE 1/* bits [1, 8] U [56, 63] are reserved */ #define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001fe
@@ -132,6 +133,29 @@ static QCow2BitmapHeader *bitmap_header(BDRVQcow2State *s, (s->bitmap_directory + bitmap->offset); } +static int update_bitmap_header_sync (BlockDriverState *bs,QCow2Bitmap *bitmap) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +QCow2BitmapHeader *h =bitmap_header (s, bitmap); + +bitmap_header_to_be (h); + ret =bdrv_pwrite (bs->file->bs, + s->bitmap_directory_offset + bitmap->offset, + h,dir_entry_size (h)); +bitmap_header_to_cpu (h); +if (ret <0 ) { +return ret; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { +return ret; + } + +return 0 ; +} +static int directory_read (BlockDriverState *bs,Error **errp) {int ret;
@@ -291,6 +315,11 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs, QCow2Bitmap *bm, bmh =bitmap_header (s, bm); +if (bmh->flags &BME_FLAG_IN_USE ) { +error_setg (errp,"Bitmap '%s' is in use" , bm->name); +return NULL ; + } + bitmap_table =g_try_malloc (bmh->bitmap_table_size *sizeof (uint64_t ));if (bitmap_table ==NULL ) { error_setg_errno(errp, -ENOMEM,
@@ -319,6 +348,13 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs, QCow2Bitmap *bm, goto fail; } + bmh->flags |=BME_FLAG_IN_USE ; + ret =update_bitmap_header_sync (bs, bm); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Could not set in_use in bitmap header" ); + goto fail; + } +g_free (bitmap_table);return bitmap;
@@ -767,6 +803,13 @@ void qcow2_bitmap_store(BlockDriverState *bs, goto finish; } + bmh->flags &= ~BME_FLAG_IN_USE ; + ret =update_bitmap_header_sync (bs, bm); +if (ret <0 ) { +error_setg_errno (errp, ret,"Can't update bitmap header." ); + goto finish; + } + finish:g_free (bitmap_table); }
Signature · 1 line
1.8.3.1
PATCH v4 13/17 qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
In reply to Vladimir Sementsov-OgievskiyCheck, that bitmap is stored to the owning bs.
--- block/dirty-bitmap.c | 12 ++++++++++++ block/qcow2-dirty-bitmap.c | 5 +++++ include/block/dirty-bitmap.h | 2 ++ 3 files changed, 19 insertions(+)
block/dirty-bitmap.c +12 -0
@@ -580,3 +580,15 @@ BdrvDirtyBitmap *bdrv_load_dirty_bitmap(BlockDriverState *bs, const char *name, }return NULL ; } + +bool bdrv_has_dirty_bitmap (BlockDriverState *bs,const BdrvDirtyBitmap *bitmap) +{ +BdrvDirtyBitmap *bm, *next; + QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) { +if (bm == bitmap) { +return true; + } + } + +return false; +}
block/qcow2-dirty-bitmap.c +5 -0
@@ -755,6 +755,11 @@ void qcow2_bitmap_store(BlockDriverState *bs,uint64_t size =bdrv_dirty_bitmap_size (bitmap);int granularity =bdrv_dirty_bitmap_granularity (bitmap); +if (!bdrv_has_dirty_bitmap (bs, bitmap)) { +error_setg (errp,"Can't store bitmap to the other node." ); +return ; + } +/* find/create dirty bitmap */ bm =find_bitmap_by_name (bs, name);if (bm ==NULL ) {
include/block/dirty-bitmap.h +2 -0
@@ -87,4 +87,6 @@ void bdrv_finalize_persistent_dirty_bitmaps(BlockDriverState *bs);BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name,Error **errp); +bool bdrv_has_dirty_bitmap (BlockDriverState *bs,const BdrvDirtyBitmap *bitmap); +#endif
Signature · 1 line
1.8.3.1
PATCH v4 14/17 iotests: add VM.test_launcn()
In reply to Vladimir Sementsov-OgievskiyTest vm can launch and print output in case of fail. This function is needed for testing erroneous cases
--- tests/qemu-iotests/iotests.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
tests/qemu-iotests/iotests.py +19 -0
@@ -226,6 +226,25 @@ class VM(object): os.remove(self._monitor_path) raise + def test_launch(self): + '''Launch the VM, an error is expected''' + try: + self.launch() + except: + self._popen.wait() + print "Test launch failed: %d" % self._popen.returncode + print "--- qemu output ---" + for line in open(self._qemu_log_path): + #filter qtest comments + if not "] OPENED" in line: + print line + print "--- end qemu output ---" + return False + + print "Tast launch successed!" + self.shutdown() + return True + def shutdown(self): '''Terminate the VM and clean up''' if not self._popen is None:
Signature · 1 line
1.8.3.1
PATCH v4 15/17 iotests: test internal persistent dirty bitmap
In reply to Vladimir Sementsov-OgievskiyAdd simple test cases for testing persistent dirty bitmaps.
--- tests/qemu-iotests/160 | 112 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/160.out | 21 ++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 6 +++ 4 files changed, 140 insertions(+) create mode 100755 tests/qemu-iotests/160 create mode 100644 tests/qemu-iotests/160.out
tests/qemu-iotests/160 new file +112 -0
@@ -0,0 +1,112 @@ +#!/usr/bin/env python +# +# Tests for persistent dirty bitmaps. +# +# Copyright: Vladimir Sementsov-Ogievskiy 2015 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests +import time +from iotests import qemu_img + +disk = os.path.join(iotests.test_dir, 'disk') + +size = 0x40000000 # 1G +sector_size = 512 +granularity = 0x10000 +regions1 = [ + { 'start': 0, 'count': 0x100000 }, + { 'start': 0x200000, 'count': 0x100000 } + ] +regions2 = [ + { 'start': 0x10000000, 'count': 0x20000 }, + { 'start': 0x39990000, 'count': 0x10000 } + ] + +class TestPersistentDirtyBitmap(iotests.QMPTestCase): + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, disk, str(size)) + + def mkVm(self, create_bitmap): + vm = iotests.VM().add_drive(disk) + vm.add_dirty_bitmap('12300000-0000-0000-1230-000001230000', 'drive0', create_bitmap) + return vm + + def tearDown(self): + os.remove(disk) + + def getMd5(self): + result = self.vm.qmp('query-block'); + return result['return'][0]['dirty-bitmaps'][0]['md5'] + + def checkBitmap(self, md5): + result = self.vm.qmp('query-block'); + self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5); + + def writeRegions(self, regions): + for r in regions: + self.vm.hmp_qemu_io('drive0', + 'write %d %d' % (r['start'], r['count'])) + + def test_persistent(self): + self.vm = self.mkVm(True) + self.vm.launch() + + self.writeRegions(regions1) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm = self.mkVm(False) + self.vm.launch() + + self.checkBitmap(md5) + self.writeRegions(regions2) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm.launch() + + self.checkBitmap(md5) + + self.vm.shutdown() + + def test_not_exist(self): + vm = self.mkVm(False) + vm.test_launch() + + def test_already_exists(self): + vm = self.mkVm(True) + vm.test_launch() + vm.test_launch() + + def test_in_use(self): + vm = self.mkVm(True) + vm.launch() + vm.shutdown() + + vm1 = self.mkVm(False) + vm1.launch() + + vm2 = self.mkVm(False) + vm2.test_launch() + + vm1.shutdown() + + +if __name__ == '__main__': + iotests.main()
tests/qemu-iotests/160.out new file +21 -0
@@ -0,0 +1,21 @@ +.... +---------------------------------------------------------------------- +Ran 4 tests + +OK +Tast launch successed! +Test launch failed: 1 +--- qemu output --- +qemu-system-x86_64: -dirty-bitmap name=12300000-0000-0000-1230-000001230000,node=drive0,create=on: bitmap '12300000-0000-0000-1230-000001230000' already exists + +--- end qemu output --- +Test launch failed: 1 +--- qemu output --- +qemu-system-x86_64: -dirty-bitmap name=12300000-0000-0000-1230-000001230000,node=drive0,create=off: Bitmap '12300000-0000-0000-1230-000001230000' is in use + +--- end qemu output --- +Test launch failed: 1 +--- qemu output --- +qemu-system-x86_64: -dirty-bitmap name=12300000-0000-0000-1230-000001230000,node=drive0,create=off: Could not find bitmap '12300000-0000-0000-1230-000001230000' in the node 'drive0' + +--- end qemu output ---
tests/qemu-iotests/group +1 -0
@@ -142,3 +142,4 @@ 138 rw auto quick 139 rw auto quick 142 auto +160 rw auto quick
tests/qemu-iotests/iotests.py +6 -0
@@ -166,6 +166,12 @@ class VM(object): self._num_drives += 1 return self + def add_dirty_bitmap(self, name, node, create=False): + '''Add dirty bitmap parameter to VM cmd''' + self._args.append('-dirty-bitmap') + self._args.append('name=%s,node=%s,create=%s' % (name, node, 'on' if create else 'off')) + return self + def pause_drive(self, drive, event=None): '''Pause drive r/w operations''' if not event:
Signature · 1 line
1.8.3.1
PATCH v4 16/17 qcow2-dirty-bitmap: add AUTO flag
In reply to Vladimir Sementsov-OgievskiyThe bitmap should be auto-loaded if auto flag is set. For now, actually, there are no methods to set it.
--- block/qcow2-dirty-bitmap.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
block/qcow2-dirty-bitmap.c +12 -1
@@ -40,8 +40,9 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xfffffffe +#define BME_RESERVED_FLAGS 0xfffffffc#define BME_FLAG_IN_USE 1 +#define BME_FLAG_AUTO (1U << 1)/* bits [1, 8] U [56, 63] are reserved */ #define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001fe
@@ -50,6 +51,9 @@ typedef enum BitmapType { BT_DIRTY_TRACKING_BITMAP = 1 } BitmapType; +static BdrvDirtyBitmap *load_bitmap (BlockDriverState *bs,QCow2Bitmap *bm, +Error **errp); +void qcow2_free_bitmaps (BlockDriverState *bs) {BDRVQcow2State *s = bs->opaque;
@@ -213,6 +217,13 @@ static int directory_read(BlockDriverState *bs, Error **errp) bm->offset = offset; bm->name =g_strndup ((char *)(h +1 ), h->name_size); +if (h->flags &BME_FLAG_AUTO ) { +load_bitmap (bs, bm, errp); +if (*errp !=NULL ) { + goto fail; + } + } + offset +=dir_entry_size (h); }return 0 ;
Signature · 1 line
1.8.3.1
PATCH v4 17/17 qcow2-dirty-bitmap: add EXTRA_DATA_COMPATIBLE flag
In reply to Vladimir Sementsov-OgievskiyIf this flag is unset and exta data present the bitmap should be read-only. For now just return error for this case.
--- block/qcow2-dirty-bitmap.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/qcow2-dirty-bitmap.c +8 -0
@@ -43,6 +43,7 @@#define BME_RESERVED_FLAGS 0xfffffffc#define BME_FLAG_IN_USE 1#define BME_FLAG_AUTO (1U << 1) +#define BME_FLAG_EXTRA_DATA_COMPATIBLE (1U << 1)/* bits [1, 8] U [56, 63] are reserved */ #define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001fe
@@ -331,6 +332,13 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs, QCow2Bitmap *bm,return NULL ; } +if (!(bmh->flags &BME_FLAG_EXTRA_DATA_COMPATIBLE ) && + bmh->extra_data_size !=0 ) { +error_setg (errp,"Uncompatible extra data found for bitmap '%s'" , + bm->name); +return NULL ; + } + bitmap_table =g_try_malloc (bmh->bitmap_table_size *sizeof (uint64_t ));if (bitmap_table ==NULL ) { error_setg_errno(errp, -ENOMEM,
Signature · 1 line
1.8.3.1
Do you have this mirrored in a git repo so I can browse it more effectively? I can't figure out what all the prerequisite patches are, so it will be convenient to just have a repo in that case for the RFC.
Quoted 73 lines · Vladimir Sementsov-Ogievskiy
This series add persistent dirty bitmaps feature to qcow2. Specification is in
docs/spec/qcow2.txt(not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and onlyhbitmap_deserialize_finishfrom serialization)This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so,bdrv_dirty_bitmap_serialize_partand friends are not needed, onlyhbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO andEXTRA_DATA_COMPATIBLEflags
- some fixes made after John's comments on v3Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: addqcow2_bitmap_load()
qcow2-dirty-bitmap: addqcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: addqcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: addbdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: addIN_USEflag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: addVM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: addEXTRA_DATA_COMPATIBLEflagblock.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-) create mode 100644 block/qcow2-dirty-bitmap.c create mode 100755 tests/qemu-iotests/160 create mode 100644 tests/qemu-iotests/160.out
Quoted 3 lines · John Snow
Do you have this mirrored in a git repo so I can browse it more effectively? I can't figure out what all the prerequisite patches are, so it will be convenient to just have a repo in that case for the RFC.
done: https://src.openvz.org/users/vsementsov/repos/qemu/browse
Quoted 75 lines · Vladimir Sementsov-Ogievskiy
Quoted 73 lines
This series add persistent dirty bitmaps feature to qcow2. Specification is in
docs/spec/qcow2.txt(not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and onlyhbitmap_deserialize_finishfrom serialization)This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so,bdrv_dirty_bitmap_serialize_partand friends are not needed, onlyhbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO andEXTRA_DATA_COMPATIBLEflags
- some fixes made after John's comments on v3Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: addqcow2_bitmap_load()
qcow2-dirty-bitmap: addqcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: addqcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: addbdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: addIN_USEflag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: addVM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: addEXTRA_DATA_COMPATIBLEflagblock.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-)create mode 100644
block/qcow2-dirty-bitmap.c
create mode 100755 tests/qemu-iotests/160
create mode 100644 tests/qemu-iotests/160.out
Signature · 2 lines
Best regards, Vladimir
Quoted 69 lines · Vladimir Sementsov-Ogievskiy
This series add persistent dirty bitmaps feature to qcow2. Specification is in
docs/spec/qcow2.txt(not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and onlyhbitmap_deserialize_finishfrom serialization)This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so,bdrv_dirty_bitmap_serialize_partand friends are not needed, onlyhbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO andEXTRA_DATA_COMPATIBLEflags
- some fixes made after John's comments on v3Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: addqcow2_bitmap_load()
qcow2-dirty-bitmap: addqcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: addqcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: addbdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: addIN_USEflag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: addVM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: addEXTRA_DATA_COMPATIBLEflagblock.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-)
Quoted 4 lines
create mode 100644
block/qcow2-dirty-bitmap.c
create mode 100755 tests/qemu-iotests/160
create mode 100644 tests/qemu-iotests/160.out
Guys?
Quoted 64 lines · Vladimir Sementsov-Ogievskiy
This series add persistent dirty bitmaps feature to qcow2. Specification is in
docs/spec/qcow2.txt(not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and onlyhbitmap_deserialize_finishfrom serialization)This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so,bdrv_dirty_bitmap_serialize_partand friends are not needed, onlyhbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO andEXTRA_DATA_COMPATIBLEflags
- some fixes made after John's comments on v3Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: addqcow2_bitmap_load()
qcow2-dirty-bitmap: addqcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: addqcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: addbdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: addIN_USEflag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: addVM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: addEXTRA_DATA_COMPATIBLEflagblock.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++
Quoted 9 lines
tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-) create mode 100644 block/qcow2-dirty-bitmap.c create mode 100755 tests/qemu-iotests/160 create mode 100644 tests/qemu-iotests/160.out
In your prerequisite patches,
"iotests-add-default-node-name" breaks qemu iotests 055 and 118. Didn't look hard enough to find out why, yet.
Quoted 75 lines · John Snow
Quoted 69 lines · Vladimir Sementsov-Ogievskiy
This series add persistent dirty bitmaps feature to qcow2. Specification is in
docs/spec/qcow2.txt(not merged yet, see [PATCH v10] spec: add qcow2 bitmaps extension specification)This series are based on Fam's
[PATCH v2 00/13] Dirty bitmap changes for migration/persistence work (meta bitmaps not used, and onlyhbitmap_deserialize_finishfrom serialization)This also needs some preparation patches, most of them are in my bitmap-migration series. I've not listed them here to keep things simpler, this is RFC any way.
v4:
Previous version was posted more than five months ago, so I will not carefully list all changes.
What should be noted:
- some changes are made to sutisfy last version of specification
- removed staff, related to possibility of saving bitmaps for one disk in the other qcow2.
- to make bitmap store/load zero-copy, I've moved load/store code to HBitmap - this is new patch 01.
so,bdrv_dirty_bitmap_serialize_partand friends are not needed, onlyhbitmap_deserialize_finish, to repair bitmap consistency after loading its last level.
- two patches added about AUTO andEXTRA_DATA_COMPATIBLEflags
- some fixes made after John's comments on v3Vladimir Sementsov-Ogievskiy (17):
hbitmap: load/store
qcow2: Bitmaps extension: structs and consts
qcow2-dirty-bitmap: read dirty bitmap directory
qcow2-dirty-bitmap: addqcow2_bitmap_load()
qcow2-dirty-bitmap: addqcow2_bitmap_store()
qcow2: add dirty bitmaps extension
qcow2-dirty-bitmap: addqcow2_bitmap_load_check()
block: store persistent dirty bitmaps
block: addbdrv_load_dirty_bitmap()
qcow2-dirty-bitmap: add autoclear bit
qemu: command line option for dirty bitmaps
qcow2-dirty-bitmap: addIN_USEflag
qcow2-dirty-bitmaps: disallow stroing bitmap to other bs
iotests: addVM.test_launcn()
iotests: test internal persistent dirty bitmap
qcow2-dirty-bitmap: add AUTO flag
qcow2-dirty-bitmap: addEXTRA_DATA_COMPATIBLEflagblock.c | 3 + block/Makefile.objs | 2 +- block/dirty-bitmap.c | 101 +++++ block/qcow2-dirty-bitmap.c | 839 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 105 +++++- block/qcow2.h | 59 +++ blockdev.c | 36 ++ include/block/block_int.h | 9 + include/block/dirty-bitmap.h | 21 ++ include/qemu/hbitmap.h | 12 + include/sysemu/blockdev.h | 1 + include/sysemu/sysemu.h | 1 + qemu-options.hx | 35 ++ tests/qemu-iotests/160 | 112 ++++++ tests/qemu-iotests/160.out | 21 ++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 25 ++ util/hbitmap.c | 182 +++++++++ vl.c | 78 ++++ 19 files changed, 1640 insertions(+), 3 deletions(-)Quoted 4 lines
create mode 100644
block/qcow2-dirty-bitmap.c
create mode 100755 tests/qemu-iotests/160
create mode 100644 tests/qemu-iotests/160.out
Quoted 4 lines
In your prerequisite patches,
"iotests-add-default-node-name" breaks qemu iotests 055 and 118. Didn't look hard enough to find out why, yet.
118 doesn't work for me anyway (./check -qcow2 118 , yes?), for 055 you are right, it's not hard, the following helps:
def add_drive(self, path, opts='', interface='virtio'):
'''Add a virtio-blk drive to the VM''' options = ['if=%s' % interface,
- 'id=drive%d' % self._num_drives, - 'node-name=drivenode%d' % self._num_drives] + 'id=drive%d' % self._num_drives]
if path is not None:
+ options.append('node-name=drivenode%d' % self._num_drives)
options.append('file=%s' % path) options.append('format=%s' % imgfmt) options.append('cache=%s' % cachemode)
Signature · 2 lines
Best regards, Vladimir