Skip to contentSkip to conversation

block: incremental backup series

PATCH In review John Snow 3 participants 28 messages Feb 20, 2015 – Feb 27, 2015 Block layer

lore
Share & tools
lore

John Snow

author
#0 lore raw

v14 cover block: incremental backup series

This series requires: [PATCH v3] blkdebug: fix "once" rule

Salutations, weary traveler. Might I interest you in some incremental backups? Only slightly used, just like new!

This patchset enables the in-memory part of the incremental backup feature. Bitmap persistence and bitmap migration will be addressed in future patchsets by Vladimir Sementsov-Ogievskiy.

This series and most patches were originally by Fam Zheng, though I have corrupted them far beyond their original presentation.

Highlights (since v11):
- Renamed "node-ref" parameter to "node" in all QMP patches. (A rose by any other name? ...)
- Fuller, more robust iotests.
- Fixed progress percentages for incremental backup jobs.
- This patch relies upon a blkdebug fix for the iotests.
- The ability to resize bitmaps (as part of bdrv_truncate) has been included.
- Documentation for the user-perspective has been added. Developers are left to wonder what they did to deserve this.

V14:
- 04: Edited description.
- 10: Fixed the add transaction. I hope.
- 18: Added a patch that allows bitmaps to be resized via bdrv_truncate().
- 19: Added a documentation file.

V13:
- 02: New patch, trying to iron out type issues.
- 03: Patch no longer includes type fixes.
- 04: Patch no longer includes tpye fixes.
- 07: Error message fixes.
- 08: Removed superfluous empty lines
- Progress updates occur before cancellation check.
- 09: Fixed a critical bug where I added an extra '-' to docs.
- Corrected clear semantics (checks for disabled status)
- More full stops and newlines removed from error_setg.
- 10: Error message cleanup.

 -     disabled status check added to clear transaction
 -     Added version documentation for new transactions
 -     Refiddled the add transaction to not delete erroneously
       if we abort after an unsuccessful prepare
 - 13: Joined lines where possible.
 - 15: Removed unnecessary imports
 -     "%s" changed to "%i"
 -     'is 0' replaced by '== 0'
 -     removed .add_args(["-S"]) and hmp("c") and added a docstring.
 -     Check for '%' in dirnames.

- 16: removed hmp("c"), added a doctsring.
- xx Dropped blkdebug patch, was #16.
- 17: Removed hmp("c"), added a docstring.
- Moved all blkdebug config into QMP, because Max asked very nicely.

V12:
- Changed authorship from Fam Zheng to John Snow on most patches
- 02: Fix the error_setg leak in bdrv_dirty_bitmap_lookup
- Fix error phrasing in bdrv_dirty_bitmap_lookup
- Renamed "node-ref" to "node" for QMP commands.
- 03: Granularity helper no longer requires a BDS argument.
- 04: Return early if the second bitmap is empty.
- 05: Renamed the 'enabled' field to 'disabled to emphasize what the default operating state is.
- We now guard against bit sets or resets with the bitmap is
disabled, making it a more pure "read only" mode.
- Some documentation phrasing changes.
- 06: Removed explicit "frozen" state in favor of an implicit one. A successor present implies a frozen state.
- Updated all functions that target a single bitmap to use
assertions that the bitmap they are trying to modify is not frozen/disabled.
- Functions that target multiple bitmaps use only a conditional,
and will silently skip disabled bitmaps.
- thaw() function removed. It is implicitly handled in reclaim
and abdicate.
- Added check for return code of hbitmap_merge.
- Functions now check against enable OR disable when in frozen
state, for consistency and simplicity.
- Add "frozen" state documentation to remove/enable/disable
QMP commands.
- 07: Some documentation for bdrv_set_dirty_iter.

 -     Move function calls outside of assert()
 -     Cleanup the unused successor if we do not start the backup
 -     Version documentation added for dirty-bitmap to block-core.json
 -     Job progress is now reported for incremental backup jobs.

- 08: bdrv_dirty_bitmap_clear is now in its own patch, here.
- bdrv_dirty_bitmap_clear no longer takes a BDS argument.
- 09: Added a transaction for bdrv_dirty_bitmap_clear.
- 10: Change 'enabled' field to 'disabled' field, to match above decision in patch 05.
- 12: Removed extraneous BDS arguments from most bitmap functions.
- 13-15: New set of iotests.
- 16: blkdebug fix, already posted upstream.
- 17: Final iotest, testing failure case.

Fam Zheng (1):
qapi: Add optional field "name" to block dirty bitmap

John Snow (18):
qmp: Ensure consistent granularity type
qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
block: Introduce bdrv_dirty_bitmap_granularity()
hbitmap: add hbitmap_merge
qmp: Add block-dirty-bitmap-enable and block-dirty-bitmap-disable
block: Add bitmap successors
qmp: Add support of "dirty-bitmap" sync mode for drive-backup
qmp: add block-dirty-bitmap-clear
qapi: Add transaction support to block-dirty-bitmap operations
qmp: Add dirty bitmap status fields in query-block
block: add BdrvDirtyBitmap documentation
block: Ensure consistent bitmap function prototypes
iotests: add invalid input incremental backup tests
iotests: add simple incremental backup case
iotests: add transactional incremental backup test
iotests: add incremental backup failure recovery test
block: Resize bitmaps on bdrv_truncate
docs: incremental backup documentation

 block.c                       | 246 +++++++++++++++++++++++++--
 block/backup.c                | 149 +++++++++++++---
 block/mirror.c                |  46 ++---
 blockdev.c                    | 387 +++++++++++++++++++++++++++++++++++++++++-
 docs/bitmaps.md               | 253 +++++++++++++++++++++++++++
 hmp.c                         |   3 +-
 include/block/block.h         |  34 +++-
 include/block/block_int.h     |   4 +-
 include/qemu/hbitmap.h        |  21 +++
 migration/block.c             |   9 +-
 qapi-schema.json              |  10 +-
 qapi/block-core.json          | 122 ++++++++++++-
 qmp-commands.hx               |  92 +++++++++-
 tests/qemu-iotests/112        | 315 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/112.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |   6 +-
 util/hbitmap.c                |  79 +++++++++
 18 files changed, 1691 insertions(+), 91 deletions(-)
 create mode 100644 docs/bitmaps.md
 create mode 100644 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out
Signature · 1 line
1.9.3
author
#1 lore raw

PATCH v14 1/19 qapi: Add optional field "name" to block dirty bitmap

From: Fam Zheng <[email protected]>

This field will be set for user created dirty bitmap. Also pass in an error pointer to bdrv_create_dirty_bitmap, so when a name is already taken on this BDS, it can report an error message. This is not global check, two BDSes can have dirty bitmap with a common name.

Implemented bdrv_find_dirty_bitmap to find a dirty bitmap by name, will be used later when other QMP commands want to reference dirty bitmap by name.

Add bdrv_dirty_bitmap_make_anon. This unsets the name of dirty bitmap.

---
 block.c               | 32 +++++++++++++++++++++++++++++++-
 block/mirror.c        |  2 +-
 include/block/block.h |  7 ++++++-
 migration/block.c     |  2 +-
 qapi/block-core.json  |  4 +++-
 5 files changed, 42 insertions(+), 5 deletions(-)
block.c +31 -1
index 210fd5f..c9c2359 100644
@@ -53,6 +53,7 @@
 
 struct BdrvDirtyBitmap {
     HBitmap *bitmap;
+    char *name;
     QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
 
@@ -5366,7 +5367,28 @@ bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov)
     return true;
 }
 
-BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, int granularity,
+BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
+{
+    BdrvDirtyBitmap *bm;
+
+    assert(name);
+    QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
+        if (bm->name && !strcmp(name, bm->name)) {
+            return bm;
+        }
+    }
+    return NULL;
+}
+
+void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
+{
+    g_free(bitmap->name);
+    bitmap->name = NULL;
+}
+
+BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
+                                          int granularity,
+                                          const char *name,
                                           Error **errp)
 {
     int64_t bitmap_size;
@@ -5374,6 +5396,10 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, int granularity,
 
     assert((granularity & (granularity - 1)) == 0);
 
+    if (name && bdrv_find_dirty_bitmap(bs, name)) {
+        error_setg(errp, "Bitmap already exists: %s", name);
+        return NULL;
+    }
     granularity >>= BDRV_SECTOR_BITS;
     assert(granularity);
     bitmap_size = bdrv_nb_sectors(bs);
@@ -5384,6 +5410,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, int granularity,
     }
     bitmap = g_new0(BdrvDirtyBitmap, 1);
     bitmap->bitmap = hbitmap_alloc(bitmap_size, ffs(granularity) - 1);
+    bitmap->name = g_strdup(name);
     QLIST_INSERT_HEAD(&bs->dirty_bitmaps, bitmap, list);
     return bitmap;
 }
@@ -5395,6 +5422,7 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
         if (bm == bitmap) {
             QLIST_REMOVE(bitmap, list);
             hbitmap_free(bitmap->bitmap);
+            g_free(bitmap->name);
             g_free(bitmap);
             return;
         }
@@ -5413,6 +5441,8 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         info->count = bdrv_get_dirty_count(bs, bm);
         info->granularity =
             ((int64_t) BDRV_SECTOR_SIZE << hbitmap_granularity(bm->bitmap));
+        info->has_name = !!bm->name;
+        info->name = g_strdup(bm->name);
         entry->value = info;
         *plist = entry;
         plist = &entry->next;
block/mirror.c +1 -1
index 4056164..f073ad7 100644
@@ -703,7 +703,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target,
     s->granularity = granularity;
     s->buf_size = MAX(buf_size, granularity);
 
-    s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, errp);
+    s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp);
     if (!s->dirty_bitmap) {
         return;
     }
include/block/block.h +6 -1
index 321295e..0988b77 100644
@@ -438,8 +438,13 @@ bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov);
 
 struct HBitmapIter;
 typedef struct BdrvDirtyBitmap BdrvDirtyBitmap;
-BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, int granularity,
+BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
+                                          int granularity,
+                                          const char *name,
                                           Error **errp);
+BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
+                                        const char *name);
+void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
migration/block.c +1 -1
index 0c76106..5f9b3e5 100644
@@ -319,7 +319,7 @@ static int set_dirty_tracking(void)
 
     QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
         bmds->dirty_bitmap = bdrv_create_dirty_bitmap(bmds->bs, BLOCK_SIZE,
-                                                      NULL);
+                                                      NULL, NULL);
         if (!bmds->dirty_bitmap) {
             ret = -errno;
             goto fail;
qapi/block-core.json +3 -1
index a3fdaf0..829ba1c 100644
@@ -326,6 +326,8 @@
 #
 # Block dirty bitmap information.
 #
+# @name: #optional the name of the dirty bitmap (Since 2.3)
+#
 # @count: number of dirty bytes according to the dirty bitmap
 #
 # @granularity: granularity of the dirty bitmap in bytes (since 1.4)
@@ -333,7 +335,7 @@
 # Since: 1.3
 ##
 { 'type': 'BlockDirtyInfo',
-  'data': {'count': 'int', 'granularity': 'int'} }
+  'data': {'*name': 'str', 'count': 'int', 'granularity': 'int'} }
 
 ##
 # @BlockInfo:
Signature · 1 line
1.9.3
author
#2 lore raw

PATCH v14 2/19 qmp: Ensure consistent granularity type

In reply to John Snow

We treat this field with a variety of different types everywhere in the code. Now it's just uint32_t.

---
 block.c                   | 11 ++++++-----
 block/mirror.c            |  4 ++--
 include/block/block.h     |  2 +-
 include/block/block_int.h |  2 +-
 qapi/block-core.json      |  2 +-
 5 files changed, 11 insertions(+), 10 deletions(-)
block.c +6 -5
index c9c2359..6e08b26 100644
@@ -5387,12 +5387,13 @@ void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 }
 
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
-                                          int granularity,
+                                          uint32_t granularity,
                                           const char *name,
                                           Error **errp)
 {
     int64_t bitmap_size;
     BdrvDirtyBitmap *bitmap;
+    uint32_t sector_granularity;
 
     assert((granularity & (granularity - 1)) == 0);
 
@@ -5400,8 +5401,8 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
         error_setg(errp, "Bitmap already exists: %s", name);
         return NULL;
     }
-    granularity >>= BDRV_SECTOR_BITS;
-    assert(granularity);
+    sector_granularity = granularity >> BDRV_SECTOR_BITS;
+    assert(sector_granularity);
     bitmap_size = bdrv_nb_sectors(bs);
     if (bitmap_size < 0) {
         error_setg_errno(errp, -bitmap_size, "could not get length of device");
@@ -5409,7 +5410,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
         return NULL;
     }
     bitmap = g_new0(BdrvDirtyBitmap, 1);
-    bitmap->bitmap = hbitmap_alloc(bitmap_size, ffs(granularity) - 1);
+    bitmap->bitmap = hbitmap_alloc(bitmap_size, ffs(sector_granularity) - 1);
     bitmap->name = g_strdup(name);
     QLIST_INSERT_HEAD(&bs->dirty_bitmaps, bitmap, list);
     return bitmap;
@@ -5440,7 +5441,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1);
         info->count = bdrv_get_dirty_count(bs, bm);
         info->granularity =
-            ((int64_t) BDRV_SECTOR_SIZE << hbitmap_granularity(bm->bitmap));
+            ((uint32_t) BDRV_SECTOR_SIZE << hbitmap_granularity(bm->bitmap));
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
         entry->value = info;
block/mirror.c +2 -2
index f073ad7..f8aac33 100644
@@ -656,7 +656,7 @@ static const BlockJobDriver commit_active_job_driver = {
 
 static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target,
                              const char *replaces,
-                             int64_t speed, int64_t granularity,
+                             int64_t speed, uint32_t granularity,
                              int64_t buf_size,
                              BlockdevOnError on_source_error,
                              BlockdevOnError on_target_error,
@@ -717,7 +717,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target,
 
 void mirror_start(BlockDriverState *bs, BlockDriverState *target,
                   const char *replaces,
-                  int64_t speed, int64_t granularity, int64_t buf_size,
+                  int64_t speed, uint32_t granularity, int64_t buf_size,
                   MirrorSyncMode mode, BlockdevOnError on_source_error,
                   BlockdevOnError on_target_error,
                   BlockCompletionFunc *cb,
include/block/block.h +1 -1
index 0988b77..abdf8d6 100644
@@ -439,7 +439,7 @@ bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov);
 struct HBitmapIter;
 typedef struct BdrvDirtyBitmap BdrvDirtyBitmap;
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
-                                          int granularity,
+                                          uint32_t granularity,
                                           const char *name,
                                           Error **errp);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
include/block/block_int.h +1 -1
index 7ad1950..be99ec0 100644
@@ -576,7 +576,7 @@ void commit_active_start(BlockDriverState *bs, BlockDriverState *base,
  */
 void mirror_start(BlockDriverState *bs, BlockDriverState *target,
                   const char *replaces,
-                  int64_t speed, int64_t granularity, int64_t buf_size,
+                  int64_t speed, uint32_t granularity, int64_t buf_size,
                   MirrorSyncMode mode, BlockdevOnError on_source_error,
                   BlockdevOnError on_target_error,
                   BlockCompletionFunc *cb,
qapi/block-core.json +1 -1
index 829ba1c..39291e7 100644
@@ -335,7 +335,7 @@
 # Since: 1.3
 ##
 { 'type': 'BlockDirtyInfo',
-  'data': {'*name': 'str', 'count': 'int', 'granularity': 'int'} }
+  'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32'} }
 
 ##
 # @BlockInfo:
Signature · 1 line
1.9.3
author
#3 lore raw

PATCH v14 3/19 qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

In reply to John Snow

The new command pair is added to manage a user created dirty bitmap. The dirty bitmap's name is mandatory and must be unique for the same device, but different devices can have bitmaps with the same names.

The granularity is an optional field. If it is not specified, we will choose a default granularity based on the cluster size if available, clamped to between 4K and 64K to mirror how the 'mirror' code was already choosing granularity. If we do not have cluster size info available, we choose 64K. This code has been factored out into a helper shared with block/mirror.

This patch also introduces the 'block_dirty_bitmap_lookup' helper, which takes a device name and a dirty bitmap name and validates the lookup, returning NULL and setting errp if there is a problem with either field. This helper will be re-used in future patches in this series.

The types added to block-core.json will be re-used in future patches in this series, see:
'qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}'

---
 block.c               |  20 ++++++++++
 block/mirror.c        |  10 +----
 blockdev.c            | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/block/block.h |   1 +
 qapi/block-core.json  |  55 +++++++++++++++++++++++++++
 qmp-commands.hx       |  51 +++++++++++++++++++++++++
 6 files changed, 228 insertions(+), 9 deletions(-)
block.c +20 -0
index 6e08b26..60dbdbd 100644
@@ -5461,6 +5461,26 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector
     }
 }
 
+/**
+ * Chooses a default granularity based on the existing cluster size,
+ * but clamped between [4K, 64K]. Defaults to 64K in the case that there
+ * is no cluster size information available.
+ */
+uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)
+{
+    BlockDriverInfo bdi;
+    uint32_t granularity;
+
+    if (bdrv_get_info(bs, &bdi) >= 0 && bdi.cluster_size > 0) {
+        granularity = MAX(4096, bdi.cluster_size);
+        granularity = MIN(65536, granularity);
+    } else {
+        granularity = 65536;
+    }
+
+    return granularity;
+}
+
 void bdrv_dirty_iter_init(BlockDriverState *bs,
                           BdrvDirtyBitmap *bitmap, HBitmapIter *hbi)
 {
block/mirror.c +1 -9
index f8aac33..1cb700e 100644
@@ -668,15 +668,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target,
     MirrorBlockJob *s;
 
     if (granularity == 0) {
-        /* Choose the default granularity based on the target file's cluster
-         * size, clamped between 4k and 64k.  */
-        BlockDriverInfo bdi;
-        if (bdrv_get_info(target, &bdi) >= 0 && bdi.cluster_size != 0) {
-            granularity = MAX(4096, bdi.cluster_size);
-            granularity = MIN(65536, granularity);
-        } else {
-            granularity = 65536;
-        }
+        granularity = bdrv_get_default_bitmap_granularity(target);
     }
 
     assert ((granularity & (granularity - 1)) == 0);
blockdev.c +100 -0

112 lines not on this page · raw message

include/block/block.h +1 -0
index abdf8d6..3589132 100644
@@ -447,6 +447,7 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
 void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
+uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
 void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
qapi/block-core.json +55 -0

61 lines not on this page · raw message

qmp-commands.hx +51 -0

57 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v14 3/19
#4 lore raw

Re: [Qemu-devel] [PATCH v14 03/19] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

Quoted 23 lines · John Snow

The new command pair is added to manage a user created dirty bitmap. The dirty bitmap's name is mandatory and must be unique for the same device, but different devices can have bitmaps with the same names.

The granularity is an optional field. If it is not specified, we will choose a default granularity based on the cluster size if available, clamped to between 4K and 64K to mirror how the 'mirror' code was already choosing granularity. If we do not have cluster size info available, we choose 64K. This code has been factored out into a helper shared with block/mirror.

This patch also introduces the 'block_dirty_bitmap_lookup' helper, which takes a device name and a dirty bitmap name and validates the lookup, returning NULL and setting errp if there is a problem with either field. This helper will be re-used in future patches in this series.

The types added to block-core.json will be re-used in future patches in this series, see:
'qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}'

Quoted 9 lines
---
 block.c               |  20 ++++++++++
 block/mirror.c        |  10 +----
 blockdev.c            | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/block/block.h |   1 +
 qapi/block-core.json  |  55 +++++++++++++++++++++++++++
 qmp-commands.hx       |  51 +++++++++++++++++++++++++
 6 files changed, 228 insertions(+), 9 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#5 lore raw

PATCH v14 4/19 block: Introduce bdrv_dirty_bitmap_granularity()

In reply to John Snow

This returns the granularity (in bytes) of dirty bitmap, which matches the QMP interface and the existing query interface.

---
 block.c               | 8 ++++++--
 include/block/block.h | 1 +
 2 files changed, 7 insertions(+), 2 deletions(-)
block.c +6 -2
index 60dbdbd..8b40e38 100644
@@ -5440,8 +5440,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         BlockDirtyInfo *info = g_new0(BlockDirtyInfo, 1);
         BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1);
         info->count = bdrv_get_dirty_count(bs, bm);
-        info->granularity =
-            ((uint32_t) BDRV_SECTOR_SIZE << hbitmap_granularity(bm->bitmap));
+        info->granularity = bdrv_dirty_bitmap_granularity(bm);
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
         entry->value = info;
@@ -5481,6 +5480,11 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)
     return granularity;
 }
 
+uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap)
+{
+    return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
+}
+
 void bdrv_dirty_iter_init(BlockDriverState *bs,
                           BdrvDirtyBitmap *bitmap, HBitmapIter *hbi)
 {
include/block/block.h +1 -0
index 3589132..41b8418 100644
@@ -448,6 +448,7 @@ void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
 uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
+uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
 void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
Signature · 1 line
1.9.3
Reviewed-by on v14 4/19
#6 lore raw

Re: [Qemu-devel] [PATCH v14 04/19] block: Introduce bdrv_dirty_bitmap_granularity()

Quoted 10 lines · John Snow

This returns the granularity (in bytes) of dirty bitmap, which matches the QMP interface and the existing query interface.

---
 block.c               | 8 ++++++--
 include/block/block.h | 1 +
 2 files changed, 7 insertions(+), 2 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#7 lore raw

PATCH v14 5/19 hbitmap: add hbitmap_merge

In reply to John Snow

We add a bitmap merge operation to assist in error cases where we wish to combine two bitmaps together.

This is algorithmically O(bits) provided HBITMAP_LEVELS remains constant. For a full bitmap on a 64bit machine:
sum(bits/64^k, k, 0, HBITMAP_LEVELS) ~= 1.01587 * bits

We may be able to improve running speed for particularly sparse bitmaps by using iterators, but the running time for dense maps will be worse.

We present the simpler solution first, and we can refine it later if needed.

---
 include/qemu/hbitmap.h | 11 +++++++++++
 util/hbitmap.c         | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
include/qemu/hbitmap.h +11 -0
index 550d7ce..c19c1cb 100644
@@ -65,6 +65,17 @@ struct HBitmapIter {
 HBitmap *hbitmap_alloc(uint64_t size, int granularity);
 
 /**
+ * hbitmap_merge:
+ * @a: The bitmap to store the result in.
+ * @b: The bitmap to merge into @a.
+ *
+ * Merge two bitmaps together.
+ * A := A (BITOR) B.
+ * B is left unmodified.
+ */
+bool hbitmap_merge(HBitmap *a, const HBitmap *b);
+
+/**
  * hbitmap_empty:
  * @hb: HBitmap to operate on.
  *
util/hbitmap.c +32 -0
index ab13971..962ff29 100644
@@ -395,3 +395,35 @@ HBitmap *hbitmap_alloc(uint64_t size, int granularity)
     hb->levels[0][0] |= 1UL << (BITS_PER_LONG - 1);
     return hb;
 }
+
+/**
+ * Given HBitmaps A and B, let A := A (BITOR) B.
+ * Bitmap B will not be modified.
+ */
+bool hbitmap_merge(HBitmap *a, const HBitmap *b)
+{
+    int i, j;
+    uint64_t size;
+
+    if ((a->size != b->size) || (a->granularity != b->granularity)) {
+        return false;
+    }
+
+    if (hbitmap_count(b) == 0) {
+        return true;
+    }
+
+    /* This merge is O(size), as BITS_PER_LONG and HBITMAP_LEVELS are constant.
+     * It may be possible to improve running times for sparsely populated maps
+     * by using hbitmap_iter_next, but this is suboptimal for dense maps.
+     */
+    size = a->size;
+    for (i = HBITMAP_LEVELS - 1; i >= 0; i--) {
+        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        for (j = 0; j < size; j++) {
+            a->levels[i][j] |= b->levels[i][j];
+        }
+    }
+
+    return true;
+}
Signature · 1 line
1.9.3
author
#8 lore raw

PATCH v14 6/19 qmp: Add block-dirty-bitmap-enable and block-dirty-bitmap-disable

In reply to John Snow

This allows to put the dirty bitmap into a disabled state where it is read only. A disabled bitmap will ignore any attempts to set or reset any of its bits, but can otherwise be renamed, deleted, or re-enabled.

It will be used before backup or writing to persistent file.

---
 block.c               | 25 +++++++++++++++++++++++++
 blockdev.c            | 40 ++++++++++++++++++++++++++++++++++++++++
 include/block/block.h |  3 +++
 qapi/block-core.json  | 28 ++++++++++++++++++++++++++++
 qmp-commands.hx       | 10 ++++++++++
 5 files changed, 106 insertions(+)
block.c +25 -0
index 8b40e38..1fa2ed5 100644
@@ -54,6 +54,7 @@
 struct BdrvDirtyBitmap {
     HBitmap *bitmap;
     char *name;
+    bool disabled;
     QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
 
@@ -5412,10 +5413,16 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
     bitmap = g_new0(BdrvDirtyBitmap, 1);
     bitmap->bitmap = hbitmap_alloc(bitmap_size, ffs(sector_granularity) - 1);
     bitmap->name = g_strdup(name);
+    bitmap->disabled = false;
     QLIST_INSERT_HEAD(&bs->dirty_bitmaps, bitmap, list);
     return bitmap;
 }
 
+bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap)
+{
+    return !bitmap->disabled;
+}
+
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 {
     BdrvDirtyBitmap *bm, *next;
@@ -5430,6 +5437,16 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
     }
 }
 
+void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
+{
+    bitmap->disabled = true;
+}
+
+void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
+{
+    bitmap->disabled = false;
+}
+
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
 {
     BdrvDirtyBitmap *bm;
@@ -5494,12 +5511,14 @@ void bdrv_dirty_iter_init(BlockDriverState *bs,
 void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors)
 {
+    assert(bdrv_dirty_bitmap_enabled(bitmap));
     hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
 }
 
 void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                              int64_t cur_sector, int nr_sectors)
 {
+    assert(bdrv_dirty_bitmap_enabled(bitmap));
     hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
 }
 
@@ -5508,6 +5527,9 @@ static void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
 {
     BdrvDirtyBitmap *bitmap;
     QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
+        if (!bdrv_dirty_bitmap_enabled(bitmap)) {
+            continue;
+        }
         hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
     }
 }
@@ -5517,6 +5539,9 @@ static void bdrv_reset_dirty(BlockDriverState *bs, int64_t cur_sector,
 {
     BdrvDirtyBitmap *bitmap;
     QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
+        if (!bdrv_dirty_bitmap_enabled(bitmap)) {
+            continue;
+        }
         hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
     }
 }
blockdev.c +40 -0
index 8842e39..d3599ac 100644
@@ -2053,6 +2053,46 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
     aio_context_release(aio_context);
 }
 
+void qmp_block_dirty_bitmap_enable(const char *node, const char *name,
+                                   Error **errp)
+{
+    AioContext *aio_context;
+    BdrvDirtyBitmap *bitmap;
+    BlockDriverState *bs;
+
+    bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
+    if (!bitmap) {
+        return;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+
+    bdrv_enable_dirty_bitmap(bitmap);
+
+    aio_context_release(aio_context);
+}
+
+void qmp_block_dirty_bitmap_disable(const char *node, const char *name,
+                                    Error **errp)
+{
+    AioContext *aio_context;
+    BdrvDirtyBitmap *bitmap;
+    BlockDriverState *bs;
+
+    bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
+    if (!bitmap) {
+        return;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+
+    bdrv_disable_dirty_bitmap(bitmap);
+
+    aio_context_release(aio_context);
+}
+
 int do_drive_del(Monitor *mon, const QDict *qdict, QObject **ret_data)
 {
     const char *id = qdict_get_str(qdict, "id");
include/block/block.h +3 -0
index 41b8418..ae7e2ab 100644
@@ -446,9 +446,12 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
                                         const char *name);
 void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
+void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
+void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
 uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
 uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
+bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
 void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
qapi/block-core.json +28 -0
index 764719d..a82f8fa 100644
@@ -1014,6 +1014,34 @@
   'data': 'BlockDirtyBitmap' }
 
 ##
+# @block-dirty-bitmap-enable
+#
+# Enable a dirty bitmap on the device
+#
+# Returns: nothing on success
+#          If @node is not a valid block device, DeviceNotFound
+#          If @name is not found, GenericError with an explanation
+#
+# Since 2.3
+##
+{ 'command': 'block-dirty-bitmap-enable',
+  'data': 'BlockDirtyBitmap' }
+
+##
+# @block-dirty-bitmap-disable
+#
+# Disable a dirty bitmap on the device
+#
+# Returns: nothing on success
+#          If @node is not a valid block device, DeviceNotFound
+#          If @name is not found, GenericError with an explanation
+#
+# Since 2.3
+##
+{ 'command': 'block-dirty-bitmap-disable',
+  'data': 'BlockDirtyBitmap' }
+
+##
 # @block_set_io_throttle:
 #
 # Change I/O throttle limits for a block drive.
qmp-commands.hx +10 -0
index 881a810..ce7782f 100644
@@ -1253,6 +1253,16 @@ EQMP
         .args_type  = "node:B,name:s",
         .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_remove,
     },
+    {
+        .name       = "block-dirty-bitmap-enable",
+        .args_type  = "node:B,name:s",
+        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_enable,
+    },
+    {
+        .name       = "block-dirty-bitmap-disable",
+        .args_type  = "node:B,name:s",
+        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_disable,
+    },
 
 SQMP
 
Signature · 1 line
1.9.3
author
#9 lore raw

PATCH v14 7/19 block: Add bitmap successors

In reply to John Snow

A bitmap successor is an anonymous BdrvDirtyBitmap that is intended to be created just prior to a sensitive operation (e.g. Incremental Backup) that can either succeed or fail, but during the course of which we still want a bitmap tracking writes.

On creating a successor, we "freeze" the parent bitmap which prevents its deletion, enabling, anonymization, or creating a bitmap with the same name.

On success, the parent bitmap can "abdicate" responsibility to the successor, which will inherit its name. The successor will have been tracking writes during the course of the backup operation. The parent will be safely deleted.

On failure, we can "reclaim" the successor from the parent, unifying them such that the resulting bitmap describes all writes occurring since the last successful backup, for instance. Reclamation will thaw the parent, but not explicitly re-enable it.

BdrvDirtyBitmap operations that target a single bitmap are protected by assertions that the bitmap is not frozen and/or disabled.

BdrvDirtyBitmap operations that target a group of bitmaps, such as bdrv_{set,reset}_dirty will ignore frozen/disabled drives with a conditional instead.

QMP transactions that enable/disable bitmaps have extra error checking surrounding them that prevent modifying bitmaps that are frozen.

---
 block.c               | 104 +++++++++++++++++++++++++++++++++++++++++++++++++-
 blockdev.c            |  22 +++++++++++
 include/block/block.h |  10 +++++
 qapi/block-core.json  |   3 ++
 4 files changed, 138 insertions(+), 1 deletion(-)
block.c +103 -1
index 1fa2ed5..665ae69 100644
@@ -51,8 +51,17 @@
 #include <windows.h>
 #endif
 
+/**
+ * A BdrvDirtyBitmap can be in three possible states:
+ * (1) successor is false and disabled is false: full r/w mode
+ * (2) successor is false and disabled is true: read only mode ("disabled")
+ * (3) successor is set: frozen mode.
+ *     A frozen bitmap cannot be renamed, deleted, anonymized, cleared, set,
+ *     or enabled. A frozen bitmap can only abdicate() or reclaim().
+ */
 struct BdrvDirtyBitmap {
     HBitmap *bitmap;
+    BdrvDirtyBitmap *successor;
     char *name;
     bool disabled;
     QLIST_ENTRY(BdrvDirtyBitmap) list;
@@ -5383,6 +5392,7 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
 
 void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 {
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));
     g_free(bitmap->name);
     bitmap->name = NULL;
 }
@@ -5418,9 +5428,98 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
     return bitmap;
 }
 
+bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap)
+{
+    return bitmap->successor;
+}
+
 bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap)
 {
-    return !bitmap->disabled;
+    return !(bitmap->disabled || bitmap->successor);
+}
+
+/**
+ * Create a successor bitmap destined to replace this bitmap after an operation.
+ * Requires that the bitmap is not frozen and has no successor.
+ */
+int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
+                                       BdrvDirtyBitmap *bitmap, Error **errp)
+{
+    uint64_t granularity;
+    BdrvDirtyBitmap *child;
+
+    if (bdrv_dirty_bitmap_frozen(bitmap)) {
+        error_setg(errp, "Cannot create a successor for a bitmap that is "
+                   "currently frozen");
+        return -1;
+    }
+    assert(!bitmap->successor);
+
+    /* Create an anonymous successor */
+    granularity = bdrv_dirty_bitmap_granularity(bitmap);
+    child = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp);
+    if (!child) {
+        return -1;
+    }
+
+    /* Successor will be on or off based on our current state. */
+    child->disabled = bitmap->disabled;
+
+    /* Install the successor and freeze the parent */
+    bitmap->successor = child;
+    return 0;
+}
+
+/**
+ * For a bitmap with a successor, yield our name to the successor,
+ * Delete the old bitmap, and return a handle to the new bitmap.
+ */
+BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
+                                            BdrvDirtyBitmap *bitmap,
+                                            Error **errp)
+{
+    char *name;
+    BdrvDirtyBitmap *successor = bitmap->successor;
+
+    if (successor == NULL) {
+        error_setg(errp, "Cannot relinquish control if "
+                   "there's no successor present");
+        return NULL;
+    }
+
+    name = bitmap->name;
+    bitmap->name = NULL;
+    successor->name = name;
+    bitmap->successor = NULL;
+    bdrv_release_dirty_bitmap(bs, bitmap);
+
+    return successor;
+}
+
+/**
+ * In cases of failure where we can no longer safely delete the parent,
+ * We may wish to re-join the parent and child/successor.
+ * The merged parent will be un-frozen, but not explicitly re-enabled.
+ */
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
+                                           BdrvDirtyBitmap *parent,
+                                           Error **errp)
+{
+    BdrvDirtyBitmap *successor = parent->successor;
+
+    if (!successor) {
+        error_setg(errp, "Cannot reclaim a successor when none is present");
+        return NULL;
+    }
+
+    if (!hbitmap_merge(parent->bitmap, successor->bitmap)) {
+        error_setg(errp, "Merging of parent and successor bitmap failed");
+        return NULL;
+    }
+    bdrv_release_dirty_bitmap(bs, successor);
+    parent->successor = NULL;
+
+    return parent;
 }
 
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
@@ -5428,6 +5527,7 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
     BdrvDirtyBitmap *bm, *next;
     QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) {
         if (bm == bitmap) {
+            assert(!bdrv_dirty_bitmap_frozen(bm));
             QLIST_REMOVE(bitmap, list);
             hbitmap_free(bitmap->bitmap);
             g_free(bitmap->name);
@@ -5439,11 +5539,13 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 
 void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));
     bitmap->disabled = true;
 }
 
 void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));
     bitmap->disabled = false;
 }
 
blockdev.c +22 -0
index d3599ac..9ecba7d 100644
@@ -2047,9 +2047,16 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
+    if (bdrv_dirty_bitmap_frozen(bitmap)) {
+        error_setg(errp,
+                   "Bitmap '%s' is currently frozen and cannot be removed",
+                   name);
+        goto out;
+    }
     bdrv_dirty_bitmap_make_anon(bs, bitmap);
     bdrv_release_dirty_bitmap(bs, bitmap);
 
+ out:
     aio_context_release(aio_context);
 }
 
@@ -2068,8 +2075,16 @@ void qmp_block_dirty_bitmap_enable(const char *node, const char *name,
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
+    if (bdrv_dirty_bitmap_frozen(bitmap)) {
+        error_setg(errp,
+                   "Bitmap '%s' is currently frozen and cannot be modified",
+                   name);
+        goto out;
+    }
+
     bdrv_enable_dirty_bitmap(bitmap);
 
+ out:
     aio_context_release(aio_context);
 }
 
@@ -2088,8 +2103,15 @@ void qmp_block_dirty_bitmap_disable(const char *node, const char *name,
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
+    if (bdrv_dirty_bitmap_frozen(bitmap)) {
+        error_setg(errp,
+                   "Bitmap '%s' is currently frozen and cannot be modified",
+                   name);
+        goto out;
+    }
     bdrv_disable_dirty_bitmap(bitmap);
 
+ out:
     aio_context_release(aio_context);
 }
 
include/block/block.h +10 -0
index ae7e2ab..60e8b6d 100644
@@ -442,6 +442,15 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                           uint32_t granularity,
                                           const char *name,
                                           Error **errp);
+int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
+                                       BdrvDirtyBitmap *bitmap,
+                                       Error **errp);
+BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
+                                            BdrvDirtyBitmap *bitmap,
+                                            Error **errp);
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
+                                           BdrvDirtyBitmap *bitmap,
+                                           Error **errp);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
                                         const char *name);
 void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
@@ -452,6 +461,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
 uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
 uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
+bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
 void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
qapi/block-core.json +3 -0
index a82f8fa..f00e3bb 100644
@@ -1007,6 +1007,7 @@
 # Returns: nothing on success
 #          If @node is not a valid block device or node, DeviceNotFound
 #          If @name is not found, GenericError with an explanation
+#          if @name is frozen by an operation, GenericError
 #
 # Since 2.3
 ##
@@ -1021,6 +1022,7 @@
 # Returns: nothing on success
 #          If @node is not a valid block device, DeviceNotFound
 #          If @name is not found, GenericError with an explanation
+#          if @name is frozen by an operation, GenericError
 #
 # Since 2.3
 ##
@@ -1035,6 +1037,7 @@
 # Returns: nothing on success
 #          If @node is not a valid block device, DeviceNotFound
 #          If @name is not found, GenericError with an explanation
+#          if @name is frozen by an operation, GenericError
 #
 # Since 2.3
 ##
Signature · 1 line
1.9.3
author
#10 lore raw

PATCH v14 8/19 qmp: Add support of "dirty-bitmap" sync mode for drive-backup

In reply to John Snow

For "dirty-bitmap" sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of "top" sync mode.

---
 block.c                   |   9 +++
 block/backup.c            | 149 +++++++++++++++++++++++++++++++++++++++-------
 block/mirror.c            |   4 ++
 blockdev.c                |  18 +++++-
 hmp.c                     |   3 +-
 include/block/block.h     |   1 +
 include/block/block_int.h |   2 +
 qapi/block-core.json      |  13 ++--
 qmp-commands.hx           |   7 ++-
 9 files changed, 172 insertions(+), 34 deletions(-)
block.c +9 -0
index 665ae69..6e3f817 100644
@@ -5648,6 +5648,15 @@ static void bdrv_reset_dirty(BlockDriverState *bs, int64_t cur_sector,
     }
 }
 
+/**
+ * Advance an HBitmapIter to an arbitrary offset.
+ */
+void bdrv_set_dirty_iter(HBitmapIter *hbi, int64_t offset)
+{
+    assert(hbi->hb);
+    hbitmap_iter_init(hbi, hbi->hb, offset);
+}
+
 int64_t bdrv_get_dirty_count(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 {
     return hbitmap_count(bitmap->bitmap);
block/backup.c +126 -23

221 lines not on this page · raw message

block/mirror.c +4 -0
index 1cb700e..f89eccf 100644
@@ -718,6 +718,10 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target,
     bool is_none_mode;
     BlockDriverState *base;
 
+    if (mode == MIRROR_SYNC_MODE_DIRTY_BITMAP) {
+        error_setg(errp, "Sync mode 'dirty-bitmap' not supported");
+        return;
+    }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
     base = mode == MIRROR_SYNC_MODE_TOP ? bs->backing_hd : NULL;
     mirror_start_job(bs, target, replaces,
blockdev.c +15 -3
index 9ecba7d..6990986 100644
@@ -1569,6 +1569,7 @@ static void drive_backup_prepare(BlkTransactionState *common, Error **errp)
                      backup->sync,
                      backup->has_mode, backup->mode,
                      backup->has_speed, backup->speed,
+                     backup->has_bitmap, backup->bitmap,
                      backup->has_on_source_error, backup->on_source_error,
                      backup->has_on_target_error, backup->on_target_error,
                      &local_err);
@@ -2429,6 +2430,7 @@ void qmp_drive_backup(const char *device, const char *target,
                       enum MirrorSyncMode sync,
                       bool has_mode, enum NewImageMode mode,
                       bool has_speed, int64_t speed,
+                      bool has_bitmap, const char *bitmap,
                       bool has_on_source_error, BlockdevOnError on_source_error,
                       bool has_on_target_error, BlockdevOnError on_target_error,
                       Error **errp)
@@ -2436,6 +2438,7 @@ void qmp_drive_backup(const char *device, const char *target,
     BlockDriverState *bs;
     BlockDriverState *target_bs;
     BlockDriverState *source = NULL;
+    BdrvDirtyBitmap *bmap = NULL;
     AioContext *aio_context;
     BlockDriver *drv = NULL;
     Error *local_err = NULL;
@@ -2534,7 +2537,16 @@ void qmp_drive_backup(const char *device, const char *target,
 
     bdrv_set_aio_context(target_bs, aio_context);
 
-    backup_start(bs, target_bs, speed, sync, on_source_error, on_target_error,
+    if (has_bitmap) {
+        bmap = bdrv_find_dirty_bitmap(bs, bitmap);
+        if (!bmap) {
+            error_setg(errp, "Bitmap '%s' could not be found", bitmap);
+            goto out;
+        }
+    }
+
+    backup_start(bs, target_bs, speed, sync, bmap,
+                 on_source_error, on_target_error,
                  block_job_cb, bs, &local_err);
     if (local_err != NULL) {
         bdrv_unref(target_bs);
@@ -2592,8 +2604,8 @@ void qmp_blockdev_backup(const char *device, const char *target,
 
     bdrv_ref(target_bs);
     bdrv_set_aio_context(target_bs, aio_context);
-    backup_start(bs, target_bs, speed, sync, on_source_error, on_target_error,
-                 block_job_cb, bs, &local_err);
+    backup_start(bs, target_bs, speed, sync, NULL, on_source_error,
+                 on_target_error, block_job_cb, bs, &local_err);
     if (local_err != NULL) {
         bdrv_unref(target_bs);
         error_propagate(errp, local_err);
hmp.c +2 -1
index b47f331..015499f 100644
@@ -1027,7 +1027,8 @@ void hmp_drive_backup(Monitor *mon, const QDict *qdict)
 
     qmp_drive_backup(device, filename, !!format, format,
                      full ? MIRROR_SYNC_MODE_FULL : MIRROR_SYNC_MODE_TOP,
-                     true, mode, false, 0, false, 0, false, 0, &err);
+                     true, mode, false, 0, false, NULL,
+                     false, 0, false, 0, &err);
     hmp_handle_error(mon, &err);
 }
 
include/block/block.h +1 -0
index 60e8b6d..be35b57 100644
@@ -469,6 +469,7 @@ void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                              int64_t cur_sector, int nr_sectors);
 void bdrv_dirty_iter_init(BlockDriverState *bs,
                           BdrvDirtyBitmap *bitmap, struct HBitmapIter *hbi);
+void bdrv_set_dirty_iter(struct HBitmapIter *hbi, int64_t offset);
 int64_t bdrv_get_dirty_count(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 
 void bdrv_enable_copy_on_read(BlockDriverState *bs);
include/block/block_int.h +2 -0
index be99ec0..b8409ef 100644
@@ -588,6 +588,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target,
  * @target: Block device to write to.
  * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
  * @sync_mode: What parts of the disk image should be copied to the destination.
+ * @sync_bitmap: The dirty bitmap if sync_mode is MIRROR_SYNC_MODE_DIRTY_BITMAP.
  * @on_source_error: The action to take upon error reading from the source.
  * @on_target_error: The action to take upon error writing to the target.
  * @cb: Completion function for the job.
@@ -598,6 +599,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target,
  */
 void backup_start(BlockDriverState *bs, BlockDriverState *target,
                   int64_t speed, MirrorSyncMode sync_mode,
+                  BdrvDirtyBitmap *sync_bitmap,
                   BlockdevOnError on_source_error,
                   BlockdevOnError on_target_error,
                   BlockCompletionFunc *cb, void *opaque,
qapi/block-core.json +9 -4
index f00e3bb..dca83c5 100644
@@ -508,10 +508,12 @@
 #
 # @none: only copy data written from now on
 #
+# @dirty-bitmap: only copy data described by the dirty bitmap. Since: 2.3
+#
 # Since: 1.3
 ##
 { 'enum': 'MirrorSyncMode',
-  'data': ['top', 'full', 'none'] }
+  'data': ['top', 'full', 'none', 'dirty-bitmap'] }
 
 ##
 # @BlockJobType:
@@ -686,14 +688,17 @@
 #          probe if @mode is 'existing', else the format of the source
 #
 # @sync: what parts of the disk image should be copied to the destination
-#        (all the disk, only the sectors allocated in the topmost image, or
-#        only new I/O).
+#        (all the disk, only the sectors allocated in the topmost image, from a
+#        dirty bitmap, or only new I/O).
 #
 # @mode: #optional whether and how QEMU should create a new image, default is
 #        'absolute-paths'.
 #
 # @speed: #optional the maximum speed, in bytes per second
 #
+# @bitmap: #optional the name of dirty bitmap if sync is "dirty-bitmap"
+#          (Since 2.3)
+#
 # @on-source-error: #optional the action to take on an error on the source,
 #                   default 'report'.  'stop' and 'enospc' can only be used
 #                   if the block device supports io-status (see BlockInfo).
@@ -711,7 +716,7 @@
 { 'type': 'DriveBackup',
   'data': { 'device': 'str', 'target': 'str', '*format': 'str',
             'sync': 'MirrorSyncMode', '*mode': 'NewImageMode',
-            '*speed': 'int',
+            '*speed': 'int', '*bitmap': 'str',
             '*on-source-error': 'BlockdevOnError',
             '*on-target-error': 'BlockdevOnError' } }
 
qmp-commands.hx +4 -3
index ce7782f..5aa3845 100644
@@ -1048,7 +1048,7 @@ EQMP
     {
         .name       = "drive-backup",
         .args_type  = "sync:s,device:B,target:s,speed:i?,mode:s?,format:s?,"
-                      "on-source-error:s?,on-target-error:s?",
+                      "bitmap:s?,on-source-error:s?,on-target-error:s?",
         .mhandler.cmd_new = qmp_marshal_input_drive_backup,
     },
 
@@ -1075,8 +1075,9 @@ Arguments:
             (json-string, optional)
 - "sync": what parts of the disk image should be copied to the destination;
   possibilities include "full" for all the disk, "top" for only the sectors
-  allocated in the topmost image, or "none" to only replicate new I/O
-  (MirrorSyncMode).
+  allocated in the topmost image, "dirty-bitmap" for only the dirty sectors in
+  the bitmap, or "none" to only replicate new I/O (MirrorSyncMode).
+- "bitmap": dirty bitmap name for sync==dirty-bitmap
 - "mode": whether and how QEMU should create a new image
           (NewImageMode, optional, default 'absolute-paths')
 - "speed": the maximum speed, in bytes per second (json-int, optional)
Signature · 1 line
1.9.3
author
#11 lore raw

PATCH v14 9/19 qmp: add block-dirty-bitmap-clear

In reply to John Snow

Add bdrv_clear_dirty_bitmap and a matching QMP command, qmp_block_dirty_bitmap_clear that enables a user to reset the bitmap attached to a drive.

This allows us to reset a bitmap in the event of a full drive backup.

---
 block.c               |  8 ++++++++
 blockdev.c            | 37 +++++++++++++++++++++++++++++++++++++
 include/block/block.h |  1 +
 qapi/block-core.json  | 14 ++++++++++++++
 qmp-commands.hx       | 24 ++++++++++++++++++++++++
 5 files changed, 84 insertions(+)
block.c +8 -0
index 6e3f817..e2da4cb 100644
@@ -62,6 +62,7 @@
 struct BdrvDirtyBitmap {
     HBitmap *bitmap;
     BdrvDirtyBitmap *successor;
+    int64_t size;
     char *name;
     bool disabled;
     QLIST_ENTRY(BdrvDirtyBitmap) list;
@@ -5422,6 +5423,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
     }
     bitmap = g_new0(BdrvDirtyBitmap, 1);
     bitmap->bitmap = hbitmap_alloc(bitmap_size, ffs(sector_granularity) - 1);
+    bitmap->size = bitmap_size;
     bitmap->name = g_strdup(name);
     bitmap->disabled = false;
     QLIST_INSERT_HEAD(&bs->dirty_bitmaps, bitmap, list);
@@ -5624,6 +5626,12 @@ void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
     hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
 }
 
+void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap)
+{
+    assert(bdrv_dirty_bitmap_enabled(bitmap));
+    hbitmap_reset(bitmap->bitmap, 0, bitmap->size);
+}
+
 static void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
                            int nr_sectors)
 {
blockdev.c +37 -0
index 6990986..8422e94 100644
@@ -2116,6 +2116,43 @@ void qmp_block_dirty_bitmap_disable(const char *node, const char *name,
     aio_context_release(aio_context);
 }
 
+/**
+ * Completely clear a bitmap, for the purposes of synchronizing a bitmap
+ * immediately after a full backup operation.
+ */
+void qmp_block_dirty_bitmap_clear(const char *node, const char *name,
+                                  Error **errp)
+{
+    AioContext *aio_context;
+    BdrvDirtyBitmap *bitmap;
+    BlockDriverState *bs;
+
+    bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
+    if (!bitmap) {
+        return;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+
+    if (bdrv_dirty_bitmap_frozen(bitmap)) {
+        error_setg(errp,
+                   "Bitmap '%s' is currently frozen and cannot be modified",
+                   name);
+        goto out;
+    } else if (!bdrv_dirty_bitmap_enabled(bitmap)) {
+        error_setg(errp,
+                   "Bitmap '%s' is currently disabled and cannot be cleared",
+                   name);
+        goto out;
+    }
+
+    bdrv_clear_dirty_bitmap(bitmap);
+
+ out:
+    aio_context_release(aio_context);
+}
+
 int do_drive_del(Monitor *mon, const QDict *qdict, QObject **ret_data)
 {
     const char *id = qdict_get_str(qdict, "id");
include/block/block.h +1 -0
index be35b57..67227a2 100644
@@ -467,6 +467,7 @@ void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
 void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
                              int64_t cur_sector, int nr_sectors);
+void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_iter_init(BlockDriverState *bs,
                           BdrvDirtyBitmap *bitmap, struct HBitmapIter *hbi);
 void bdrv_set_dirty_iter(struct HBitmapIter *hbi, int64_t offset);
qapi/block-core.json +14 -0
index dca83c5..bfdb293 100644
@@ -1050,6 +1050,20 @@
   'data': 'BlockDirtyBitmap' }
 
 ##
+# @block-dirty-bitmap-clear
+#
+# Clear (reset) a dirty bitmap on the device
+#
+# Returns: nothing on success
+#          If @node is not a valid block device, DeviceNotFound
+#          If @name is not found, GenericError with an explanation
+#
+# Since 2.3
+##
+{ 'command': 'block-dirty-bitmap-clear',
+  'data': 'BlockDirtyBitmap' }
+
+##
 # @block_set_io_throttle:
 #
 # Change I/O throttle limits for a block drive.
qmp-commands.hx +24 -0
index 5aa3845..21416fa 100644
@@ -1264,6 +1264,11 @@ EQMP
         .args_type  = "node:B,name:s",
         .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_disable,
     },
+    {
+        .name       = "block-dirty-bitmap-clear",
+        .args_type  = "node:B,name:s",
+        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_clear,
+    },
 
 SQMP
 
@@ -1303,6 +1308,25 @@ Example:
                                                       "name": "bitmap0" } }
 <- { "return": {} }
 
+block-dirty-bitmap-clear
+------------------------
+Since 2.3
+
+Reset the dirty bitmap associated with a node so that an incremental backup
+from this point in time forward will only backup clusters modified after this
+clear operation.
+
+Arguments:
+
+- "node": device/node on which to remove dirty bitmap (json-string)
+- "name": name of the dirty bitmap to remove (json-string)
+
+Example:
+
+-> { "execute": "block-dirty-bitmap-clear", "arguments": { "node": "drive0",
+                                                           "name": "bitmap0" } }
+<- { "return": {} }
+
 EQMP
 
     {
Signature · 1 line
1.9.3
author
#12 lore raw

PATCH v14 10/19 qapi: Add transaction support to block-dirty-bitmap operations

In reply to John Snow

This adds four qmp commands to transactions.

Users can stop a dirty bitmap, start backup of it, and start another dirty bitmap atomically, so that the dirty bitmap is tracked incrementally and we don't miss any write.

For starting a new incremental backup chain, users can also chain together a bitmap clear and a full block backup.

---
 blockdev.c       | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qapi-schema.json |  10 +++-
 2 files changed, 179 insertions(+), 1 deletion(-)
blockdev.c +170 -0
index 8422e94..7fbcb98 100644
@@ -1675,6 +1675,153 @@ static void blockdev_backup_clean(BlkTransactionState *common)
     }
 }
 
+typedef struct BlockDirtyBitmapState {
+    BlkTransactionState common;
+    BdrvDirtyBitmap *bitmap;
+    BlockDriverState *bs;
+    AioContext *aio_context;
+    bool prepared;
+} BlockDirtyBitmapState;
+
+static void block_dirty_bitmap_add_prepare(BlkTransactionState *common,
+                                           Error **errp)
+{
+    BlockDirtyBitmapAdd *action;
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    action = common->action->block_dirty_bitmap_add;
+    /* AIO context taken within qmp_block_dirty_bitmap_add */
+    qmp_block_dirty_bitmap_add(action->node, action->name,
+                               action->has_granularity, action->granularity,
+                               errp);
+
+    if (*errp == NULL) {
+        state->prepared = true;
+    }
+}
+
+static void block_dirty_bitmap_add_abort(BlkTransactionState *common)
+{
+    BlockDirtyBitmapAdd *action;
+   BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    action = common->action->block_dirty_bitmap_add;
+    /* Should not fail meaningfully: IF the bitmap was added via .prepare(),
+     * then the node reference and bitmap name must have been valid.
+     * THUS: any failure here could only indicate the lack of a bitmap at all.
+     */
+    if (state->prepared) {
+        qmp_block_dirty_bitmap_remove(action->node, action->name, NULL);
+    }
+}
+
+/**
+ * Enable and Disable re-use the same preparation.
+ */
+static void block_dirty_bitmap_toggle_prepare(BlkTransactionState *common,
+                                              Error **errp)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    BlockDirtyBitmap *action;
+    BlockDriverState *bs;
+
+    /* We may be used by either enable or disable;
+     * We use the "enable" member of the union here,
+     * but "disable" should be functionally equivalent: */
+    action = common->action->block_dirty_bitmap_enable;
+    assert(action == common->action->block_dirty_bitmap_disable);
+
+    state->bitmap = block_dirty_bitmap_lookup(action->node,
+                                              action->name,
+                                              &bs,
+                                              errp);
+    if (!state->bitmap) {
+        return;
+    }
+
+    if (bdrv_dirty_bitmap_frozen(state->bitmap)) {
+        error_setg(errp, "Cannot modify a frozen bitmap");
+        return;
+    }
+
+    /* AioContext is released in .clean() */
+    state->aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(state->aio_context);
+}
+
+static void block_dirty_bitmap_enable_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_enable_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_disable_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_disable_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_toggle_clean(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->aio_context) {
+        aio_context_release(state->aio_context);
+    }
+}
+
+static void block_dirty_bitmap_clear_prepare(BlkTransactionState *common,
+                                             Error **errp)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    BlockDirtyBitmap *action;
+
+    action = common->action->block_dirty_bitmap_clear;
+    state->bitmap = block_dirty_bitmap_lookup(action->node,
+                                              action->name,
+                                              &state->bs,
+                                              errp);
+    if (!state->bitmap) {
+        return;
+    }
+
+    if (bdrv_dirty_bitmap_frozen(state->bitmap)) {
+        error_setg(errp, "Cannot modify a frozen bitmap");
+        return;
+    } else if (!bdrv_dirty_bitmap_enabled(state->bitmap)) {
+        error_setg(errp, "Cannot clear a disabled bitmap");
+        return;
+    }
+
+    /* AioContext is released in .clean() */
+    state->aio_context = bdrv_get_aio_context(state->bs);
+    aio_context_acquire(state->aio_context);
+}
+
+static void block_dirty_bitmap_clear_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_clear_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_clear_clean(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->aio_context) {
+        aio_context_release(state->aio_context);
+    }
+}
+
 static void abort_prepare(BlkTransactionState *common, Error **errp)
 {
     error_setg(errp, "Transaction aborted using Abort action");
@@ -1715,6 +1862,29 @@ static const BdrvActionOps actions[] = {
         .abort = internal_snapshot_abort,
         .clean = internal_snapshot_clean,
     },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ADD] = {
+        .instance_size = sizeof(BlkTransactionState),
+        .prepare = block_dirty_bitmap_add_prepare,
+        .abort = block_dirty_bitmap_add_abort,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ENABLE] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_toggle_prepare,
+        .commit = block_dirty_bitmap_enable_commit,
+        .clean = block_dirty_bitmap_toggle_clean,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_DISABLE] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_toggle_prepare,
+        .commit = block_dirty_bitmap_disable_commit,
+        .clean = block_dirty_bitmap_toggle_clean,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_CLEAR] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_clear_prepare,
+        .commit = block_dirty_bitmap_clear_commit,
+        .clean = block_dirty_bitmap_clear_clean,
+    }
 };
 
 /*
qapi-schema.json +9 -1
index e16f8eb..0c31203 100644
@@ -1332,6 +1332,10 @@
 # abort since 1.6
 # blockdev-snapshot-internal-sync since 1.7
 # blockdev-backup since 2.3
+# block-dirty-bitmap-add since 2.3
+# block-dirty-bitmap-enable since 2.3
+# block-dirty-bitmap-disable since 2.3
+# block-dirty-bitmap-clear since 2.3
 ##
 { 'union': 'TransactionAction',
   'data': {
@@ -1339,7 +1343,11 @@
        'drive-backup': 'DriveBackup',
        'blockdev-backup': 'BlockdevBackup',
        'abort': 'Abort',
-       'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal'
+       'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
+       'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd',
+       'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
+       'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
+       'block-dirty-bitmap-clear': 'BlockDirtyBitmap'
    } }
 
 ##
Signature · 1 line
1.9.3
#13 lore raw

Re: [Qemu-devel] [PATCH v14 10/19] qapi: Add transaction support to block-dirty-bitmap operations

Quoted 36 lines · John Snow

This adds four qmp commands to transactions.

Users can stop a dirty bitmap, start backup of it, and start another dirty bitmap atomically, so that the dirty bitmap is tracked incrementally and we don't miss any write.

For starting a new incremental backup chain, users can also chain together a bitmap clear and a full block backup.

---
  blockdev.c       | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
  qapi-schema.json |  10 +++-
  2 files changed, 179 insertions(+), 1 deletion(-)
blockdev.c +12 -0
index 8422e94..7fbcb98 100644
@@ -1675,6 +1675,153 @@ static void blockdev_backup_clean(BlkTransactionState *common)
      }
  }
 
+typedef struct BlockDirtyBitmapState {
+    BlkTransactionState common;
+    BdrvDirtyBitmap *bitmap;
+    BlockDriverState *bs;
+    AioContext *aio_context;
+    bool prepared;
+} BlockDirtyBitmapState;
+
+static void block_dirty_bitmap_add_prepare(BlkTransactionState *common,
+                                           Error **errp)
+{
+    BlockDirtyBitmapAdd *action;

blockdev.c static void blockdev_backup_clean(BlkTransactionState *common) · line 1699

Quoted 10 lines
blockdev.c +10 -0
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    action = common->action->block_dirty_bitmap_add;
+    /* AIO context taken within qmp_block_dirty_bitmap_add */
+    qmp_block_dirty_bitmap_add(action->node, action->name,
+                               action->has_granularity, action->granularity,
+                               errp);
+
+    if (*errp == NULL) {

You don't know whether errp is NULL or not; please use a local Error * object (and error_propagate()).

Quoted 7 lines
blockdev.c +7 -0
+        state->prepared = true;
+    }
+}
+
+static void block_dirty_bitmap_add_abort(BlkTransactionState *common)
+{
+    BlockDirtyBitmapAdd *action;

blockdev.c static void blockdev_backup_clean(BlkTransactionState *common) · line 1716

Quoted 10 lines
blockdev.c +10 -0
+   BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    action = common->action->block_dirty_bitmap_add;
+    /* Should not fail meaningfully: IF the bitmap was added via .prepare(),
+     * then the node reference and bitmap name must have been valid.
+     * THUS: any failure here could only indicate the lack of a bitmap at all.
+     */
+    if (state->prepared) {
+        qmp_block_dirty_bitmap_remove(action->node, action->name, NULL);

Considering that I agree that this cannot fail, I propose you use &error_abort as the errp parameter. I'm leaving the decision up to you.

Quoted 112 lines
blockdev.c +108 -0
+    }
+}
+
+/**
+ * Enable and Disable re-use the same preparation.
+ */
+static void block_dirty_bitmap_toggle_prepare(BlkTransactionState *common,
+                                              Error **errp)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    BlockDirtyBitmap *action;
+    BlockDriverState *bs;
+
+    /* We may be used by either enable or disable;
+     * We use the "enable" member of the union here,
+     * but "disable" should be functionally equivalent: */
+    action = common->action->block_dirty_bitmap_enable;
+    assert(action == common->action->block_dirty_bitmap_disable);
+
+    state->bitmap = block_dirty_bitmap_lookup(action->node,
+                                              action->name,
+                                              &bs,
+                                              errp);
+    if (!state->bitmap) {
+        return;
+    }
+
+    if (bdrv_dirty_bitmap_frozen(state->bitmap)) {
+        error_setg(errp, "Cannot modify a frozen bitmap");
+        return;
+    }
+
+    /* AioContext is released in .clean() */
+    state->aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(state->aio_context);
+}
+
+static void block_dirty_bitmap_enable_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_enable_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_disable_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_disable_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_toggle_clean(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->aio_context) {
+        aio_context_release(state->aio_context);
+    }
+}
+
+static void block_dirty_bitmap_clear_prepare(BlkTransactionState *common,
+                                             Error **errp)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    BlockDirtyBitmap *action;
+
+    action = common->action->block_dirty_bitmap_clear;
+    state->bitmap = block_dirty_bitmap_lookup(action->node,
+                                              action->name,
+                                              &state->bs,
+                                              errp);
+    if (!state->bitmap) {
+        return;
+    }
+
+    if (bdrv_dirty_bitmap_frozen(state->bitmap)) {
+        error_setg(errp, "Cannot modify a frozen bitmap");
+        return;
+    } else if (!bdrv_dirty_bitmap_enabled(state->bitmap)) {
+        error_setg(errp, "Cannot clear a disabled bitmap");
+        return;
+    }
+
+    /* AioContext is released in .clean() */
+    state->aio_context = bdrv_get_aio_context(state->bs);
+    aio_context_acquire(state->aio_context);
+}
+
+static void block_dirty_bitmap_clear_commit(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+    bdrv_clear_dirty_bitmap(state->bitmap);
+}
+
+static void block_dirty_bitmap_clear_clean(BlkTransactionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->aio_context) {
+        aio_context_release(state->aio_context);
+    }
+}
+
  static void abort_prepare(BlkTransactionState *common, Error **errp)
  {
      error_setg(errp, "Transaction aborted using Abort action");

blockdev.c static const BdrvActionOps actions[] = { · line 1866

Quoted 5 lines
blockdev.c +2 -0
@@ -1715,6 +1862,29 @@ static const BdrvActionOps actions[] = {
          .abort = internal_snapshot_abort,
          .clean = internal_snapshot_clean,
      },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ADD] = {
+        .instance_size = sizeof(BlkTransactionState),

Probably because you're writing to a different offset in the structure now than before, this doesn't segfault any more for me (but valgrind does reveal an "invalid write"; and inserting a "state->bitmap = (void *)42;" does make it segfault again); but this should still be sizeof(BlockDirtyBitmapState).

Max

Quoted 52 lines
blockdev.c +21 -0
+        .prepare = block_dirty_bitmap_add_prepare,
+        .abort = block_dirty_bitmap_add_abort,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ENABLE] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_toggle_prepare,
+        .commit = block_dirty_bitmap_enable_commit,
+        .clean = block_dirty_bitmap_toggle_clean,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_DISABLE] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_toggle_prepare,
+        .commit = block_dirty_bitmap_disable_commit,
+        .clean = block_dirty_bitmap_toggle_clean,
+    },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_CLEAR] = {
+        .instance_size = sizeof(BlockDirtyBitmapState),
+        .prepare = block_dirty_bitmap_clear_prepare,
+        .commit = block_dirty_bitmap_clear_commit,
+        .clean = block_dirty_bitmap_clear_clean,
+    }
  };
 
  /*
qapi-schema.json +9 -1
index e16f8eb..0c31203 100644
@@ -1332,6 +1332,10 @@
  # abort since 1.6
  # blockdev-snapshot-internal-sync since 1.7
  # blockdev-backup since 2.3
+# block-dirty-bitmap-add since 2.3
+# block-dirty-bitmap-enable since 2.3
+# block-dirty-bitmap-disable since 2.3
+# block-dirty-bitmap-clear since 2.3
  ##
  { 'union': 'TransactionAction',
    'data': {
@@ -1339,7 +1343,11 @@
         'drive-backup': 'DriveBackup',
         'blockdev-backup': 'BlockdevBackup',
         'abort': 'Abort',
-       'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal'
+       'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
+       'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd',
+       'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
+       'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
+       'block-dirty-bitmap-clear': 'BlockDirtyBitmap'
     } }
 
  ##
author
#14 lore raw

PATCH v14 11/19 qmp: Add dirty bitmap status fields in query-block

In reply to John Snow

Adds the "disabled" and "frozen" status booleans.

---
 block.c              | 2 ++
 qapi/block-core.json | 7 ++++++-
 2 files changed, 8 insertions(+), 1 deletion(-)
block.c +2 -0
index e2da4cb..a7f56f4 100644
@@ -5564,6 +5564,8 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         info->granularity = bdrv_dirty_bitmap_granularity(bm);
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
+        info->disabled = bm->disabled;
+        info->frozen = bdrv_dirty_bitmap_frozen(bm);
         entry->value = info;
         *plist = entry;
         plist = &entry->next;
qapi/block-core.json +6 -1
index bfdb293..02bf2b6 100644
@@ -332,10 +332,15 @@
 #
 # @granularity: granularity of the dirty bitmap in bytes (since 1.4)
 #
+# @disabled: whether the dirty bitmap is disabled (Since 2.3)
+#
+# @frozen: whether the dirty bitmap is frozen (Since 2.3)
+#
 # Since: 1.3
 ##
 { 'type': 'BlockDirtyInfo',
-  'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32'} }
+  'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
+           'disabled': 'bool', 'frozen': 'bool'} }
 
 ##
 # @BlockInfo:
Signature · 1 line
1.9.3
author
#15 lore raw

PATCH v14 12/19 block: add BdrvDirtyBitmap documentation

In reply to John Snow
---
 block.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
block.c +5 -5
index a7f56f4..c1c11aa 100644
@@ -60,11 +60,11 @@
  *     or enabled. A frozen bitmap can only abdicate() or reclaim().
  */
 struct BdrvDirtyBitmap {
-    HBitmap *bitmap;
-    BdrvDirtyBitmap *successor;
-    int64_t size;
-    char *name;
-    bool disabled;
+    HBitmap *bitmap;            /* Dirty sector bitmap implementation */
+    BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
+    char *name;                 /* Optional non-empty unique ID */
+    int64_t size;               /* Size of the bitmap (Number of sectors) */
+    bool disabled;              /* Bitmap is read-only */
     QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
 
Signature · 1 line
1.9.3
author
#16 lore raw

PATCH v14 13/19 block: Ensure consistent bitmap function prototypes

In reply to John Snow

We often don't need the BlockDriverState for functions that operate on bitmaps. Remove it.

---
 block.c               | 13 ++++++-------
 block/backup.c        |  2 +-
 block/mirror.c        | 26 ++++++++++----------------
 blockdev.c            |  2 +-
 include/block/block.h | 11 +++++------
 migration/block.c     |  7 +++----
 6 files changed, 26 insertions(+), 35 deletions(-)
block.c +6 -7
index c1c11aa..f3a6dd4 100644
@@ -5391,7 +5391,7 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
     return NULL;
 }
 
-void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
+void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
 {
     assert(!bdrv_dirty_bitmap_frozen(bitmap));
     g_free(bitmap->name);
@@ -5560,7 +5560,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
     QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
         BlockDirtyInfo *info = g_new0(BlockDirtyInfo, 1);
         BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1);
-        info->count = bdrv_get_dirty_count(bs, bm);
+        info->count = bdrv_get_dirty_count(bm);
         info->granularity = bdrv_dirty_bitmap_granularity(bm);
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
@@ -5608,20 +5608,19 @@ uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap)
     return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
 }
 
-void bdrv_dirty_iter_init(BlockDriverState *bs,
-                          BdrvDirtyBitmap *bitmap, HBitmapIter *hbi)
+void bdrv_dirty_iter_init(BdrvDirtyBitmap *bitmap, HBitmapIter *hbi)
 {
     hbitmap_iter_init(hbi, bitmap->bitmap, 0);
 }
 
-void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors)
 {
     assert(bdrv_dirty_bitmap_enabled(bitmap));
     hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
 }
 
-void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
                              int64_t cur_sector, int nr_sectors)
 {
     assert(bdrv_dirty_bitmap_enabled(bitmap));
@@ -5667,7 +5666,7 @@ void bdrv_set_dirty_iter(HBitmapIter *hbi, int64_t offset)
     hbitmap_iter_init(hbi, hbi->hb, offset);
 }
 
-int64_t bdrv_get_dirty_count(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
+int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap)
 {
     return hbitmap_count(bitmap->bitmap);
 }
block/backup.c +1 -1
index d46c0a3..41bd9af 100644
@@ -313,7 +313,7 @@ static void coroutine_fn backup_run(void *opaque)
         int64_t last_cluster = -1;
         bool polyrhythmic;
 
-        bdrv_dirty_iter_init(bs, job->sync_bitmap, &hbi);
+        bdrv_dirty_iter_init(job->sync_bitmap, &hbi);
         /* Does the granularity happen to match our backup cluster size? */
         polyrhythmic = (bdrv_dirty_bitmap_granularity(job->sync_bitmap) !=
                         BACKUP_CLUSTER_SIZE);
block/mirror.c +10 -16
index f89eccf..dcd6f65 100644
@@ -125,11 +125,9 @@ static void mirror_write_complete(void *opaque, int ret)
     MirrorOp *op = opaque;
     MirrorBlockJob *s = op->s;
     if (ret < 0) {
-        BlockDriverState *source = s->common.bs;
         BlockErrorAction action;
 
-        bdrv_set_dirty_bitmap(source, s->dirty_bitmap, op->sector_num,
-                              op->nb_sectors);
+        bdrv_set_dirty_bitmap(s->dirty_bitmap, op->sector_num, op->nb_sectors);
         action = mirror_error_action(s, false, -ret);
         if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) {
             s->ret = ret;
@@ -143,11 +141,9 @@ static void mirror_read_complete(void *opaque, int ret)
     MirrorOp *op = opaque;
     MirrorBlockJob *s = op->s;
     if (ret < 0) {
-        BlockDriverState *source = s->common.bs;
         BlockErrorAction action;
 
-        bdrv_set_dirty_bitmap(source, s->dirty_bitmap, op->sector_num,
-                              op->nb_sectors);
+        bdrv_set_dirty_bitmap(s->dirty_bitmap, op->sector_num, op->nb_sectors);
         action = mirror_error_action(s, true, -ret);
         if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) {
             s->ret = ret;
@@ -170,10 +166,9 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
 
     s->sector_num = hbitmap_iter_next(&s->hbi);
     if (s->sector_num < 0) {
-        bdrv_dirty_iter_init(source, s->dirty_bitmap, &s->hbi);
+        bdrv_dirty_iter_init(s->dirty_bitmap, &s->hbi);
         s->sector_num = hbitmap_iter_next(&s->hbi);
-        trace_mirror_restart_iter(s,
-                                  bdrv_get_dirty_count(source, s->dirty_bitmap));
+        trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap));
         assert(s->sector_num >= 0);
     }
 
@@ -288,8 +283,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
         next_sector += sectors_per_chunk;
     }
 
-    bdrv_reset_dirty_bitmap(source, s->dirty_bitmap, sector_num,
-                            nb_sectors);
+    bdrv_reset_dirty_bitmap(s->dirty_bitmap, sector_num, nb_sectors);
 
     /* Copy the dirty cluster.  */
     s->in_flight++;
@@ -446,7 +440,7 @@ static void coroutine_fn mirror_run(void *opaque)
 
             assert(n > 0);
             if (ret == 1) {
-                bdrv_set_dirty_bitmap(bs, s->dirty_bitmap, sector_num, n);
+                bdrv_set_dirty_bitmap(s->dirty_bitmap, sector_num, n);
                 sector_num = next;
             } else {
                 sector_num += n;
@@ -454,7 +448,7 @@ static void coroutine_fn mirror_run(void *opaque)
         }
     }
 
-    bdrv_dirty_iter_init(bs, s->dirty_bitmap, &s->hbi);
+    bdrv_dirty_iter_init(s->dirty_bitmap, &s->hbi);
     last_pause_ns = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
     for (;;) {
         uint64_t delay_ns = 0;
@@ -466,7 +460,7 @@ static void coroutine_fn mirror_run(void *opaque)
             goto immediate_exit;
         }
 
-        cnt = bdrv_get_dirty_count(bs, s->dirty_bitmap);
+        cnt = bdrv_get_dirty_count(s->dirty_bitmap);
         /* s->common.offset contains the number of bytes already processed so
          * far, cnt is the number of dirty sectors remaining and
          * s->sectors_in_flight is the number of sectors currently being
@@ -516,7 +510,7 @@ static void coroutine_fn mirror_run(void *opaque)
 
                 should_complete = s->should_complete ||
                     block_job_is_cancelled(&s->common);
-                cnt = bdrv_get_dirty_count(bs, s->dirty_bitmap);
+                cnt = bdrv_get_dirty_count(s->dirty_bitmap);
             }
         }
 
@@ -531,7 +525,7 @@ static void coroutine_fn mirror_run(void *opaque)
              */
             trace_mirror_before_drain(s, cnt);
             bdrv_drain(bs);
-            cnt = bdrv_get_dirty_count(bs, s->dirty_bitmap);
+            cnt = bdrv_get_dirty_count(s->dirty_bitmap);
         }
 
         ret = 0;
blockdev.c +1 -1
index 7fbcb98..3c8ca13 100644
@@ -2224,7 +2224,7 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
                    name);
         goto out;
     }
-    bdrv_dirty_bitmap_make_anon(bs, bitmap);
+    bdrv_dirty_bitmap_make_anon(bitmap);
     bdrv_release_dirty_bitmap(bs, bitmap);
 
  out:
include/block/block.h +5 -6
index 67227a2..f6a50ae 100644
@@ -453,7 +453,7 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
                                            Error **errp);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
                                         const char *name);
-void bdrv_dirty_bitmap_make_anon(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
@@ -463,15 +463,14 @@ uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap);
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector);
-void bdrv_set_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
                            int64_t cur_sector, int nr_sectors);
-void bdrv_reset_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
                              int64_t cur_sector, int nr_sectors);
 void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap);
-void bdrv_dirty_iter_init(BlockDriverState *bs,
-                          BdrvDirtyBitmap *bitmap, struct HBitmapIter *hbi);
+void bdrv_dirty_iter_init(BdrvDirtyBitmap *bitmap, struct HBitmapIter *hbi);
 void bdrv_set_dirty_iter(struct HBitmapIter *hbi, int64_t offset);
-int64_t bdrv_get_dirty_count(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
+int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
 
 void bdrv_enable_copy_on_read(BlockDriverState *bs);
 void bdrv_disable_copy_on_read(BlockDriverState *bs);
migration/block.c +3 -4
index 5f9b3e5..1a9e7cd 100644
@@ -303,7 +303,7 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)
     blk->aiocb = bdrv_aio_readv(bs, cur_sector, &blk->qiov,
                                 nr_sectors, blk_mig_read_cb, blk);
 
-    bdrv_reset_dirty_bitmap(bs, bmds->dirty_bitmap, cur_sector, nr_sectors);
+    bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector, nr_sectors);
     qemu_mutex_unlock_iothread();
 
     bmds->cur_sector = cur_sector + nr_sectors;
@@ -496,8 +496,7 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds,
                 g_free(blk);
             }
 
-            bdrv_reset_dirty_bitmap(bmds->bs, bmds->dirty_bitmap, sector,
-                                    nr_sectors);
+            bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, sector, nr_sectors);
             break;
         }
         sector += BDRV_SECTORS_PER_DIRTY_CHUNK;
@@ -583,7 +582,7 @@ static int64_t get_remaining_dirty(void)
     int64_t dirty = 0;
 
     QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
-        dirty += bdrv_get_dirty_count(bmds->bs, bmds->dirty_bitmap);
+        dirty += bdrv_get_dirty_count(bmds->dirty_bitmap);
     }
 
     return dirty << BDRV_SECTOR_BITS;
Signature · 1 line
1.9.3
author
#17 lore raw

PATCH v14 14/19 iotests: add invalid input incremental backup tests

In reply to John Snow
---
 tests/qemu-iotests/112     | 89 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/112.out |  5 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 95 insertions(+)
 create mode 100644 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out
tests/qemu-iotests/112 new file +89 -0
new file mode 100644
index 0000000..7985cd1
@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+#
+# Tests for incremental drive-backup
+#
+# Copyright (C) 2015 John Snow for Red Hat, Inc.
+#
+# Based on 056.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+
+
+def io_write_patterns(img, patterns):
+    for pattern in patterns:
+        iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
+
+
+class TestIncrementalBackup(iotests.QMPTestCase):
+    def setUp(self):
+        self.bitmaps = list()
+        self.files = list()
+        self.vm = iotests.VM()
+        self.test_img = os.path.join(iotests.test_dir, 'base.img')
+        self.full_bak = os.path.join(iotests.test_dir, 'backup.img')
+        self.foo_img = os.path.join(iotests.test_dir, 'foo.bar')
+        self.img_create(self.test_img, iotests.imgfmt)
+        self.vm.add_drive(self.test_img)
+        # Create a base image with a distinctive patterning
+        io_write_patterns(self.test_img, (('0x41', 0, 512),
+                                          ('0xd5', '1M', '32k'),
+                                          ('0xdc', '32M', '124k')))
+        self.vm.launch()
+
+
+    def img_create(self, img, fmt=iotests.imgfmt, size='64M',
+                   parent=None, parentFormat=None):
+        plist = list()
+        if parent:
+            if parentFormat is None:
+                parentFormat = fmt
+            iotests.qemu_img('create', '-f', fmt, img, size,
+                             '-b', parent, '-F', parentFormat)
+        else:
+            iotests.qemu_img('create', '-f', fmt, img, size)
+        self.files.append(img)
+
+    def test_sync_dirty_bitmap_missing(self):
+        self.assert_no_active_block_jobs()
+        self.files.append(self.foo_img)
+        result = self.vm.qmp('drive-backup', device='drive0',
+                             sync='dirty-bitmap', format=iotests.imgfmt,
+                             target=self.foo_img)
+        self.assert_qmp(result, 'error/class', 'GenericError')
+
+
+    def test_sync_dirty_bitmap_not_found(self):
+        self.assert_no_active_block_jobs()
+        self.files.append(self.foo_img)
+        result = self.vm.qmp('drive-backup', device='drive0',
+                             sync='dirty-bitmap', bitmap='unknown',
+                             format=iotests.imgfmt, target=self.foo_img)
+        self.assert_qmp(result, 'error/class', 'GenericError')
+
+
+    def tearDown(self):
+        self.vm.shutdown()
+        for filename in self.files:
+            try:
+                os.remove(filename)
+            except OSError:
+                pass
+
+
+if __name__ == '__main__':
+    iotests.main(supported_fmts=['qcow2'])
tests/qemu-iotests/112.out new file +5 -0
new file mode 100644
index 0000000..fbc63e6
@@ -0,0 +1,5 @@
+..
+----------------------------------------------------------------------
+Ran 2 tests
+
+OK
tests/qemu-iotests/group +1 -0
index 4b2b93b..b4ddf1b 100644
@@ -114,6 +114,7 @@
 109 rw auto
 110 rw auto backing quick
 111 rw auto quick
+112 rw auto backing
 113 rw auto quick
 114 rw auto quick
 116 rw auto quick
Signature · 1 line
1.9.3
author
#18 lore raw

PATCH v14 15/19 iotests: add simple incremental backup case

In reply to John Snow
---
 tests/qemu-iotests/112     | 122 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/112.out |   4 +-
 2 files changed, 124 insertions(+), 2 deletions(-)
tests/qemu-iotests/112 +122 -0
index 7985cd1..da8f0e0 100644
@@ -28,6 +28,36 @@ def io_write_patterns(img, patterns):
     for pattern in patterns:
         iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
+class Bitmap:
+    def __init__(self, name, node):
+        self.name = name
+        self.node = node
+        self.pattern = os.path.join(iotests.test_dir.replace('%', '%%'),
+                                    '%s.backup.%%i.img' % name)
+        self.num = 0
+        self.backups = list()
+
+    def new_target(self, num=None):
+        if num is None:
+            num = self.num
+        self.num = num + 1
+        target = self.pattern % num
+        self.backups.append(target)
+        return target
+
+    def last_target(self):
+        return self.backups[-1]
+
+    def del_target(self):
+        os.remove(self.backups.pop())
+        self.num -= 1
+
+    def __del__(self):
+        for backup in self.backups:
+            try:
+                os.remove(backup)
+            except OSError:
+                pass
 
 class TestIncrementalBackup(iotests.QMPTestCase):
     def setUp(self):
@@ -58,6 +88,98 @@ class TestIncrementalBackup(iotests.QMPTestCase):
             iotests.qemu_img('create', '-f', fmt, img, size)
         self.files.append(img)
 
+
+    def create_full_backup(self, drive='drive0'):
+        res = self.vm.qmp('drive-backup', device=drive,
+                          sync='full', format=iotests.imgfmt,
+                          target=self.full_bak)
+        self.assert_qmp(res, 'return', {})
+        self.wait_until_completed(drive)
+        self.check_full_backup()
+        self.files.append(self.full_bak)
+
+
+    def check_full_backup(self):
+        self.assertTrue(iotests.compare_images(self.test_img, self.full_bak))
+
+
+    def add_bitmap(self, name, node='drive0'):
+        bitmap = Bitmap(name, node)
+        self.bitmaps.append(bitmap)
+        result = self.vm.qmp('block-dirty-bitmap-add', node=bitmap.node,
+                             name=bitmap.name)
+        self.assert_qmp(result, 'return', {})
+        return bitmap
+
+
+    def create_incremental(self, bitmap=None, num=None,
+                           parent=None, parentFormat=None, validate=True):
+        if bitmap is None:
+            bitmap = self.bitmaps[-1]
+
+        # If this is the first incremental backup for a bitmap,
+        # use the full backup as a backing image. Otherwise, use
+        # the last incremental backup.
+        if parent is None:
+            if bitmap.num == 0:
+                parent = self.full_bak
+            else:
+                parent = bitmap.last_target()
+
+        target = bitmap.new_target(num)
+        self.img_create(target, iotests.imgfmt, parent=parent)
+
+        result = self.vm.qmp('drive-backup', device=bitmap.node,
+                             sync='dirty-bitmap', bitmap=bitmap.name,
+                             format=iotests.imgfmt, target=target,
+                             mode='existing')
+        self.assert_qmp(result, 'return', {})
+
+        event = self.wait_until_completed(bitmap.node, check_offset=validate)
+        if validate:
+            return self.check_incremental(target)
+
+
+    def check_incremental(self, target=None):
+        if target is None:
+            target = self.bitmaps[-1].last_target()
+        self.assertTrue(iotests.compare_images(self.test_img, target))
+        return True
+
+
+    def hmp_io_writes(self, drive, patterns):
+        for pattern in patterns:
+            self.vm.hmp_qemu_io(drive, 'write -P%s %s %s' % pattern)
+        self.vm.hmp_qemu_io(drive, 'flush')
+
+
+    def test_incremental_simple(self):
+        '''
+        Test: Create and verify three incremental backups.
+
+        Create a bitmap and a full backup before VM execution begins,
+        then create a series of three incremental backups "during execution,"
+        i.e.; after IO requests begin modifying the drive.
+        '''
+        self.create_full_backup()
+        self.add_bitmap('bitmap0', 'drive0')
+
+        # Sanity: Create a "hollow" incremental backup
+        self.create_incremental()
+        # Three writes: One complete overwrite, one new segment,
+        # and one partial overlap.
+        self.hmp_io_writes('drive0', (('0xab', 0, 512),
+                                      ('0xfe', '16M', '256k'),
+                                      ('0x64', '32736k', '64k')))
+        self.create_incremental()
+        # Three more writes, one of each kind, like above
+        self.hmp_io_writes('drive0', (('0x9a', 0, 512),
+                                      ('0x55', '8M', '352k'),
+                                      ('0x78', '15872k', '1M')))
+        self.create_incremental()
+        return True
+
+
     def test_sync_dirty_bitmap_missing(self):
         self.assert_no_active_block_jobs()
         self.files.append(self.foo_img)
tests/qemu-iotests/112.out +2 -2
index fbc63e6..8d7e996 100644
@@ -1,5 +1,5 @@
-..
+...
 ----------------------------------------------------------------------
-Ran 2 tests
+Ran 3 tests
 
 OK
Signature · 1 line
1.9.3
author
#19 lore raw

PATCH v14 16/19 iotests: add transactional incremental backup test

In reply to John Snow
---
 tests/qemu-iotests/112     | 49 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/112.out |  4 ++--
 2 files changed, 51 insertions(+), 2 deletions(-)
tests/qemu-iotests/112 +49 -0
index da8f0e0..59eb1b1 100644
@@ -180,6 +180,55 @@ class TestIncrementalBackup(iotests.QMPTestCase):
         return True
 
 
+    def test_incremental_transaction(self):
+        '''Test: Verify backups made from transactionally created bitmaps.
+
+        Create a bitmap "before" VM execution begins, then create a second
+        bitmap AFTER writes have already occurred. Use transactions to create
+        a full backup and synchronize both bitmaps to this backup.
+        Create an incremental backup through both bitmaps and verify that
+        both backups match the full backup.
+        '''
+        bitmap0 = self.add_bitmap('bitmap0', 'drive0')
+        self.hmp_io_writes('drive0', (('0xab', 0, 512),
+                                      ('0xfe', '16M', '256k'),
+                                      ('0x64', '32736k', '64k')))
+        bitmap1 = self.add_bitmap('bitmap1', 'drive0')
+
+        result = self.vm.qmp('transaction', actions=[
+            {
+                'type': 'block-dirty-bitmap-clear',
+                'data': { 'node': 'drive0',
+                          'name': 'bitmap0' },
+            },
+            {
+                'type': 'block-dirty-bitmap-clear',
+                'data': { 'node': 'drive0',
+                          'name': 'bitmap1' },
+            },
+            {
+                'type': 'drive-backup',
+                'data': { 'device': 'drive0',
+                          'sync': 'full',
+                          'format': iotests.imgfmt,
+                          'target': self.full_bak },
+            }
+        ])
+        self.assert_qmp(result, 'return', {})
+        self.wait_until_completed()
+        self.files.append(self.full_bak)
+        self.check_full_backup()
+
+        self.hmp_io_writes('drive0', (('0x9a', 0, 512),
+                                      ('0x55', '8M', '352k'),
+                                      ('0x78', '15872k', '1M')))
+        # Both bitmaps should be in sync and create fully valid
+        # incremental backups
+        res1 = self.create_incremental(bitmap0)
+        res2 = self.create_incremental(bitmap1)
+        self.assertTrue(res1 and res2)
+
+
     def test_sync_dirty_bitmap_missing(self):
         self.assert_no_active_block_jobs()
         self.files.append(self.foo_img)
tests/qemu-iotests/112.out +2 -2
index 8d7e996..89968f3 100644
@@ -1,5 +1,5 @@
-...
+....
 ----------------------------------------------------------------------
-Ran 3 tests
+Ran 4 tests
 
 OK
Signature · 1 line
1.9.3
author
#20 lore raw

PATCH v14 17/19 iotests: add incremental backup failure recovery test

In reply to John Snow

To test the failure case, we modify iotests.py to allow us to specify that we'd like to allow failures when we wait for block job events.

---
 tests/qemu-iotests/112        | 57 ++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/112.out    |  4 +--
 tests/qemu-iotests/iotests.py |  6 +++--
 3 files changed, 62 insertions(+), 5 deletions(-)
tests/qemu-iotests/112 +56 -1
index 59eb1b1..da0bf6d 100644
@@ -135,7 +135,11 @@ class TestIncrementalBackup(iotests.QMPTestCase):
                              mode='existing')
         self.assert_qmp(result, 'return', {})
 
-        event = self.wait_until_completed(bitmap.node, check_offset=validate)
+        event = self.wait_until_completed(bitmap.node, check_offset=validate,
+                                          allow_failures=(not validate))
+        if 'error' in event['data']:
+            bitmap.del_target()
+            return False
         if validate:
             return self.check_incremental(target)
 
@@ -229,6 +233,57 @@ class TestIncrementalBackup(iotests.QMPTestCase):
         self.assertTrue(res1 and res2)
 
 
+    def test_incremental_failure(self):
+        '''Test: Verify backups made after a failure are correct.
+
+        Simulate a failure during an incremental backup block job,
+        emulate additional writes, then create another incremental backup
+        afterwards and verify that the backup created is correct.
+        '''
+
+        # Create a blkdebug interface to this img as 'drive1'
+        result = self.vm.qmp('blockdev-add', options={
+            'id': 'drive1',
+            'driver': iotests.imgfmt,
+            'file': {
+                'driver': 'blkdebug',
+                'image': {
+                    'driver': 'file',
+                    'filename': self.test_img
+                },
+                'set-state': [{
+                    'event': 'flush_to_disk',
+                    'state': 1,
+                    'new_state': 2
+                }],
+                'inject-error': [{
+                    'event': 'read_aio',
+                    'errno': 5,
+                    'state': 2,
+                    'immediately': False,
+                    'once': True
+                }],
+            }
+        })
+        self.assert_qmp(result, 'return', {})
+
+        self.create_full_backup()
+        self.add_bitmap('bitmap0', 'drive1')
+        # Note: at this point, during a normal execution,
+        # Assume that the VM resumes and begins issuing IO requests here.
+
+        self.hmp_io_writes('drive1', (('0xab', 0, 512),
+                                      ('0xfe', '16M', '256k'),
+                                      ('0x64', '32736k', '64k')))
+
+        result = self.create_incremental(validate=False)
+        self.assertFalse(result)
+        self.hmp_io_writes('drive1', (('0x9a', 0, 512),
+                                      ('0x55', '8M', '352k'),
+                                      ('0x78', '15872k', '1M')))
+        self.create_incremental()
+
+
     def test_sync_dirty_bitmap_missing(self):
         self.assert_no_active_block_jobs()
         self.files.append(self.foo_img)
tests/qemu-iotests/112.out +2 -2
index 89968f3..914e373 100644
@@ -1,5 +1,5 @@
-....
+.....
 ----------------------------------------------------------------------
-Ran 4 tests
+Ran 5 tests
 
 OK
tests/qemu-iotests/iotests.py +4 -2
index 241b5ee..cf5faac 100644
@@ -258,14 +258,16 @@ class QMPTestCase(unittest.TestCase):
         self.assert_no_active_block_jobs()
         return result
 
-    def wait_until_completed(self, drive='drive0', check_offset=True):
+    def wait_until_completed(self, drive='drive0', check_offset=True,
+                             allow_failures=False):
         '''Wait for a block job to finish, returning the event'''
         completed = False
         while not completed:
             for event in self.vm.get_qmp_events(wait=True):
                 if event['event'] == 'BLOCK_JOB_COMPLETED':
                     self.assert_qmp(event, 'data/device', drive)
-                    self.assert_qmp_absent(event, 'data/error')
+                    if not allow_failures:
+                        self.assert_qmp_absent(event, 'data/error')
                     if check_offset:
                         self.assert_qmp(event, 'data/offset', event['data']['len'])
                     completed = True
Signature · 1 line
1.9.3
author
#21 lore raw

PATCH v14 18/19 block: Resize bitmaps on bdrv_truncate

In reply to John Snow
---
 block.c                | 20 ++++++++++++++++++++
 include/block/block.h  |  1 +
 include/qemu/hbitmap.h | 10 ++++++++++
 util/hbitmap.c         | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 78 insertions(+)
block.c +20 -0
index f3a6dd4..59a8ec9 100644
@@ -3514,6 +3514,7 @@ int bdrv_truncate(BlockDriverState *bs, int64_t offset)
     ret = drv->bdrv_truncate(bs, offset);
     if (ret == 0) {
         ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
+        bdrv_dirty_bitmap_truncate(bs);
         if (bs->blk) {
             blk_dev_resize_cb(bs->blk);
         }
@@ -5524,6 +5525,25 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
     return parent;
 }
 
+static void dirty_bitmap_truncate(BdrvDirtyBitmap *bitmap, uint64_t size)
+{
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));
+    hbitmap_truncate(bitmap->bitmap, size);
+}
+
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
+{
+    BdrvDirtyBitmap *bitmap;
+    uint64_t size = bdrv_nb_sectors(bs);
+
+    QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
+        if (bdrv_dirty_bitmap_frozen(bitmap)) {
+            continue;
+        }
+        dirty_bitmap_truncate(bitmap, size);
+    }
+}
+
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
 {
     BdrvDirtyBitmap *bm, *next;
include/block/block.h +1 -0
index f6a50ae..aa6912d 100644
@@ -442,6 +442,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                           uint32_t granularity,
                                           const char *name,
                                           Error **errp);
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
 int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
                                        BdrvDirtyBitmap *bitmap,
                                        Error **errp);
include/qemu/hbitmap.h +10 -0
index c19c1cb..a75157e 100644
@@ -65,6 +65,16 @@ struct HBitmapIter {
 HBitmap *hbitmap_alloc(uint64_t size, int granularity);
 
 /**
+ * hbitmap_truncate:
+ * @hb: The bitmap to change the size of.
+ * @size: The number of elements to change the bitmap to accommodate.
+ *
+ * truncate or grow an existing bitmap to accommodate a new number of elements.
+ * This may invalidate existing HBitmapIterators.
+ */
+void hbitmap_truncate(HBitmap *hb, uint64_t size);
+
+/**
  * hbitmap_merge:
  * @a: The bitmap to store the result in.
  * @b: The bitmap to merge into @a.
util/hbitmap.c +47 -0

65 lines not on this page · raw message

Signature · 1 line
1.9.3
#22 lore raw

Re: [Qemu-devel] [PATCH v14 18/19] block: Resize bitmaps on bdrv_truncate

Quoted 21 lines · John Snow
---
  block.c                | 20 ++++++++++++++++++++
  include/block/block.h  |  1 +
  include/qemu/hbitmap.h | 10 ++++++++++
  util/hbitmap.c         | 47 +++++++++++++++++++++++++++++++++++++++++++++++
  4 files changed, 78 insertions(+)
block.c +1 -0
index f3a6dd4..59a8ec9 100644
@@ -3514,6 +3514,7 @@ int bdrv_truncate(BlockDriverState *bs, int64_t offset)
      ret = drv->bdrv_truncate(bs, offset);
      if (ret == 0) {
          ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
+        bdrv_dirty_bitmap_truncate(bs);
          if (bs->blk) {
              blk_dev_resize_cb(bs->blk);
          }

block.c BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs, · line 5530

Quoted 6 lines
block.c +3 -0
@@ -5524,6 +5525,25 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
      return parent;
  }
 
+static void dirty_bitmap_truncate(BdrvDirtyBitmap *bitmap, uint64_t size)
+{
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));

This assertion is valid because the only way to make a bitmap frozen is by starting a drive-backup block job on the BDS, and doing so blocks the HMP block_resize command (and I don't know other ways to invoke bdrv_truncate() online).

(That means: It's fine. I just had to think about whether it is.)

Quoted 86 lines
block.c +16 -0
+    hbitmap_truncate(bitmap->bitmap, size);
+}
+
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
+{
+    BdrvDirtyBitmap *bitmap;
+    uint64_t size = bdrv_nb_sectors(bs);
+
+    QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
+        if (bdrv_dirty_bitmap_frozen(bitmap)) {
+            continue;
+        }
+        dirty_bitmap_truncate(bitmap, size);
+    }
+}
+
  void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
  {
      BdrvDirtyBitmap *bm, *next;
include/block/block.h +1 -0
index f6a50ae..aa6912d 100644
@@ -442,6 +442,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                            uint32_t granularity,
                                            const char *name,
                                            Error **errp);
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
  int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
                                         BdrvDirtyBitmap *bitmap,
                                         Error **errp);
include/qemu/hbitmap.h +10 -0
index c19c1cb..a75157e 100644
@@ -65,6 +65,16 @@ struct HBitmapIter {
  HBitmap *hbitmap_alloc(uint64_t size, int granularity);
 
  /**
+ * hbitmap_truncate:
+ * @hb: The bitmap to change the size of.
+ * @size: The number of elements to change the bitmap to accommodate.
+ *
+ * truncate or grow an existing bitmap to accommodate a new number of elements.
+ * This may invalidate existing HBitmapIterators.
+ */
+void hbitmap_truncate(HBitmap *hb, uint64_t size);
+
+/**
   * hbitmap_merge:
   * @a: The bitmap to store the result in.
   * @b: The bitmap to merge into @a.
util/hbitmap.c +12 -0
index 962ff29..d17b850 100644
@@ -90,6 +90,9 @@ struct HBitmap {
       * bitmap will still allocate HBITMAP_LEVELS arrays.
       */
      unsigned long *levels[HBITMAP_LEVELS];
+
+    /* The length of each levels[] array. */
+    uint64_t sizes[HBITMAP_LEVELS];
  };
 
  /* Advance hbi to the next nonzero word and return it.  hbi->pos
@@ -384,6 +387,7 @@ HBitmap *hbitmap_alloc(uint64_t size, int granularity)
      hb->granularity = granularity;
      for (i = HBITMAP_LEVELS; i-- > 0; ) {
          size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        hb->sizes[i] = size;
          hb->levels[i] = g_new0(unsigned long, size);
      }
 
@@ -396,6 +400,49 @@ HBitmap *hbitmap_alloc(uint64_t size, int granularity)
      return hb;
  }
 
+void hbitmap_truncate(HBitmap *hb, uint64_t size)
+{
+    bool truncate;
+    unsigned i;
+    uint64_t num_elements = size;
+
+    /* Size comes in as logical elements, adjust for granularity. */
+    size = (size + (1ULL << hb->granularity) - 1) >> hb->granularity;

util/hbitmap.c HBitmap *hbitmap_alloc(uint64_t size, int granularity) · line 420

Quoted 10 lines
util/hbitmap.c +10 -0
+    assert(size <= ((uint64_t)1 << HBITMAP_LOG_MAX_SIZE));
+    truncate = size < hb->size;
+
+    if (size == hb->size) {
+        /* A hard day's work */
+        return;
+    }
+
+    hb->size = size;
+    for (i = HBITMAP_LEVELS; i-- > 0; ) {

By now I've heard that this is a relatively wide-spread pattern; but I still don't like it because I find it pretty hard to figure out that it does what it's supposed to if you've never seen it before (and with "by now I've heard" I mean that I read some blog post where someone mentioned that he didn't like it).

At least it's not "i --> 0", so I guess I should be fine with it.

(and with "by now" I mean that I'm the one responsible for it not being "-->" in hbitmap_merge())

util/hbitmap.c HBitmap *hbitmap_alloc(uint64_t size, int granularity) · line 426

Quoted 6 lines
util/hbitmap.c +6 -0
+        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        if (hb->sizes[i] == size) {
+            continue;
+        }
+        hb->sizes[i] = size;
+        hb->levels[i] = g_realloc_n(hb->levels[i], size, sizeof(unsigned long));

How about initializing the new memory in case of truncate == false? I'm not sure whether to initialize it to 0 or 1, though. Probably 0.

util/hbitmap.c HBitmap *hbitmap_alloc(uint64_t size, int granularity) · line 438

Quoted 12 lines
util/hbitmap.c +12 -0
+    }
+    assert(size == 1);
+
+    /* Clear out any "extra space" we may have that the user didn't request:
+     * It may have garbage data in it, now. */
+    if (truncate) {
+        /* Due to granularity fuzziness, we may accidentally reset some of
+         * the last bits that are actually valid. So, record the current value,
+         * reset the "dead range," then re-set the one sector we care about. */
+        bool status = hbitmap_get(hb, num_elements - 1);
+        uint64_t fix_count = (hb->size << hb->granularity) - num_elements;
+        hbitmap_reset(hb, num_elements, fix_count);

This can fail an assertion (or at least I believe it's this function, it may be the hbitmap_set() below, too (but status should be false, so I don't think that)):

./qemu-img create -f raw test.img 64M; echo "{'execute':'qmp_capabilities'}{'execute':'block-dirty-bitmap-add','arguments':{'node':'drv','name':'foo'}}{'execute':'human-monitor-command','arguments':{'command-line':'block_resize drv 32M'}}" | x86_64-softmmu/qemu-system-x86_64 -drive if=none,id=drv,file=test.img,format=raw -qmp stdio -machine accel=qtest -nodefaults -display none Formatting 'test.img', fmt=raw size=67108864
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 2}, "package": ""}, "capabilities": []}}
{"return": {}}
{"return": {}}
qemu-system-x86_64: util/hbitmap.c:149: hbitmap_iter_init: Assertion `pos < hb->size' failed.

It's probably because fix_count == 0 and therefore num_elements is not an index into the bitmap (which hbitmap_reset() and subsequently hb_count_between() don't take into account).

Max

Quoted 10 lines
util/hbitmap.c +7 -0
+        if (status) {
+            hbitmap_set(hb, num_elements - 1, 1);
+        }
+    }
+}
+
+
  /**
   * Given HBitmaps A and B, let A := A (BITOR) B.
   * Bitmap B will not be modified.
author
#23 lore raw

Re: [Qemu-devel] [PATCH v14 18/19] block: Resize bitmaps on bdrv_truncate

Quoted 143 lines · Max Reitz
Quoted 23 lines · John Snow
---
  block.c                | 20 ++++++++++++++++++++
  include/block/block.h  |  1 +
  include/qemu/hbitmap.h | 10 ++++++++++
  util/hbitmap.c         | 47
+++++++++++++++++++++++++++++++++++++++++++++++
  4 files changed, 78 insertions(+)
block.c +1 -0
index f3a6dd4..59a8ec9 100644
@@ -3514,6 +3514,7 @@ int bdrv_truncate(BlockDriverState *bs, int64_t
 offset)
      ret = drv->bdrv_truncate(bs, offset);
      if (ret == 0) {
          ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
+        bdrv_dirty_bitmap_truncate(bs);
          if (bs->blk) {
              blk_dev_resize_cb(bs->blk);
          }

block.c BdrvDirtyBitmap · line 5531

Quoted 7 lines
block.c +3 -0
@@ -5524,6 +5525,25 @@ BdrvDirtyBitmap
 *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
      return parent;
  }
+static void dirty_bitmap_truncate(BdrvDirtyBitmap *bitmap, uint64_t
 size)
+{
+    assert(!bdrv_dirty_bitmap_frozen(bitmap));

This assertion is valid because the only way to make a bitmap frozen is by starting a drive-backup block job on the BDS, and doing so blocks the HMP block_resize command (and I don't know other ways to invoke bdrv_truncate() online).

(That means: It's fine. I just had to think about whether it is.)

Quoted 87 lines
block.c +16 -0
+    hbitmap_truncate(bitmap->bitmap, size);
+}
+
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
+{
+    BdrvDirtyBitmap *bitmap;
+    uint64_t size = bdrv_nb_sectors(bs);
+
+    QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
+        if (bdrv_dirty_bitmap_frozen(bitmap)) {
+            continue;
+        }
+        dirty_bitmap_truncate(bitmap, size);
+    }
+}
+
  void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap
 *bitmap)
  {
      BdrvDirtyBitmap *bm, *next;
include/block/block.h +1 -0
index f6a50ae..aa6912d 100644
@@ -442,6 +442,7 @@ BdrvDirtyBitmap
 *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                            uint32_t granularity,
                                            const char *name,
                                            Error **errp);
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
  int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
                                         BdrvDirtyBitmap *bitmap,
                                         Error **errp);
include/qemu/hbitmap.h +5 -0
index c19c1cb..a75157e 100644
@@ -65,6 +65,16 @@ struct HBitmapIter {
  HBitmap *hbitmap_alloc(uint64_t size, int granularity);
  /**
+ * hbitmap_truncate:
+ * @hb: The bitmap to change the size of.
+ * @size: The number of elements to change the bitmap to accommodate.
+ *
+ * truncate or grow an existing bitmap to accommodate a new number of

elements.
+ * This may invalidate existing HBitmapIterators.
+ */

include/qemu/hbitmap.h +3 -0
+void hbitmap_truncate(HBitmap *hb, uint64_t size);
+
+/**
   * hbitmap_merge:
   * @a: The bitmap to store the result in.
   * @b: The bitmap to merge into @a.
util/hbitmap.c +12 -0
index 962ff29..d17b850 100644
@@ -90,6 +90,9 @@ struct HBitmap {
       * bitmap will still allocate HBITMAP_LEVELS arrays.
       */
      unsigned long *levels[HBITMAP_LEVELS];
+
+    /* The length of each levels[] array. */
+    uint64_t sizes[HBITMAP_LEVELS];
  };
  /* Advance hbi to the next nonzero word and return it.  hbi->pos
@@ -384,6 +387,7 @@ HBitmap *hbitmap_alloc(uint64_t size, int
 granularity)
      hb->granularity = granularity;
      for (i = HBITMAP_LEVELS; i-- > 0; ) {
          size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        hb->sizes[i] = size;
          hb->levels[i] = g_new0(unsigned long, size);
      }
@@ -396,6 +400,49 @@ HBitmap *hbitmap_alloc(uint64_t size, int
 granularity)
      return hb;
  }
+void hbitmap_truncate(HBitmap *hb, uint64_t size)
+{
+    bool truncate;
+    unsigned i;
+    uint64_t num_elements = size;
+
+    /* Size comes in as logical elements, adjust for granularity. */
+    size = (size + (1ULL << hb->granularity) - 1) >> hb->granularity;

util/hbitmap.c HBitmap *hbitmap_alloc(uint64_t size, int · line 420

Quoted 10 lines
util/hbitmap.c +10 -0
+    assert(size <= ((uint64_t)1 << HBITMAP_LOG_MAX_SIZE));
+    truncate = size < hb->size;
+
+    if (size == hb->size) {
+        /* A hard day's work */
+        return;
+    }
+
+    hb->size = size;
+    for (i = HBITMAP_LEVELS; i-- > 0; ) {

By now I've heard that this is a relatively wide-spread pattern; but I still don't like it because I find it pretty hard to figure out that it does what it's supposed to if you've never seen it before (and with "by now I've heard" I mean that I read some blog post where someone mentioned that he didn't like it).

Quoted 5 lines

At least it's not "i --> 0", so I guess I should be fine with it.

(and with "by now" I mean that I'm the one responsible for it not being "-->" in hbitmap_merge())

A copy-paste loop from hbitmap_alloc is why this pattern keeps emerging. It's not necessarily how I'd do it, either.

Quoted 11 lines

util/hbitmap.c HBitmap *hbitmap_alloc(uint64_t size, int granularity)

Quoted 7 lines
util/hbitmap.c +6 -0
+        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        if (hb->sizes[i] == size) {
+            continue;
+        }
+        hb->sizes[i] = size;
+        hb->levels[i] = g_realloc_n(hb->levels[i], size,
 sizeof(unsigned long));

How about initializing the new memory in case of truncate == false? I'm not sure whether to initialize it to 0 or 1, though. Probably 0.

Good call; I had actually thought that g_realloc did initialize the memory, but looking at the docs again: nope.

Quoted 35 lines
Quoted 15 lines
util/hbitmap.c +4 -0
+    }
+    assert(size == 1);
+
+    /* Clear out any "extra space" we may have that the user didn't

request:

util/hbitmap.c +3 -0
+     * It may have garbage data in it, now. */
+    if (truncate) {
+        /* Due to granularity fuzziness, we may accidentally reset

some of
+ * the last bits that are actually valid. So, record the
current value,
+ * reset the "dead range," then re-set the one sector we care
about. */

util/hbitmap.c +2 -0
+        bool status = hbitmap_get(hb, num_elements - 1);
+        uint64_t fix_count = (hb->size << hb->granularity) -
Quoted 2 lines

num_elements;
+ hbitmap_reset(hb, num_elements, fix_count);

This can fail an assertion (or at least I believe it's this function, it may be the hbitmap_set() below, too (but status should be false, so I don't think that)):

./qemu-img create -f raw test.img 64M; echo "{'execute':'qmp_capabilities'}{'execute':'block-dirty-bitmap-add','arguments':{'node':'drv','name':'foo'}}{'execute':'human-monitor-command','arguments':{'command-line':'block_resize drv 32M'}}" | x86_64-softmmu/qemu-system-x86_64 -drive
if=none,id=drv,file=test.img,format=raw -qmp stdio -machine accel=qtest
-nodefaults -display none
Formatting 'test.img', fmt=raw size=67108864
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 2},
"package": ""}, "capabilities": []}}
{"return": {}}
{"return": {}}
qemu-system-x86_64: util/hbitmap.c:149: hbitmap_iter_init: Assertion
`pos < hb->size' failed.

Quoted 6 lines

It's probably because fix_count == 0 and therefore num_elements is not an index into the bitmap (which hbitmap_reset() and subsequently hb_count_between() don't take into account).

Max

Ah, indeed. If you hit a perfect even multiple where there are no bits at all to clean, you wind up with a one-past-the-end, even though the operation should be a no-op.

Quoted 11 lines
Quoted 10 lines
util/hbitmap.c +7 -0
+        if (status) {
+            hbitmap_set(hb, num_elements - 1, 1);
+        }
+    }
+}
+
+
  /**
   * Given HBitmaps A and B, let A := A (BITOR) B.
   * Bitmap B will not be modified.
author
#24 lore raw

PATCH v14 19/19 docs: incremental backup documentation

In reply to John Snow
---
 docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 253 insertions(+)
 create mode 100644 docs/bitmaps.md
docs/bitmaps.md new file +253 -0
new file mode 100644
index 0000000..7cda146
@@ -0,0 +1,253 @@
+# Dirty Bitmaps
+
+* Dirty bitmaps can be created at any time and attached to any node
+(not just complete drives.)
+
+## Dirty Bitmap Names
+
+* A dirty bitmap's name is unique to the node, but bitmaps attached to different
+nodes can share the same name.
+
+## Bitmap Modes
+
+* A Bitmap can be "enabled" (tracking writes, the default) or "disabled"
+(read-only, I/O is ignored.)
+
+* A Bitmap can be "frozen," which means that it is currently in-use by a backup
+operation and cannot be deleted, enabled, disabled, renamed, written to, reset,
+etc.
+
+## Basic QMP Usage
+
+### Supported Commands ###
+
+* block-dirty-bitmap-add
+* block-dirty-bitmap-remove
+* block-dirty-bitmap-enable
+* block-dirty-bitmap-disable
+* block-dirty-bitmap-clear
+
+### Creation
+
+* To create a new bitmap, enabled, on the drive with id=drive0:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0"
+  }
+}
+```
+
+* To create a new bitmap that tracks changes in 32KiB segments:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+    "granularity": "32768"
+  }
+}
+```
+
+### Deletion
+
+* Can be performed on a disabled bitmap, but not a frozen one.
+
+* Because bitmaps are only unique to the node to which they are attached,
+you must specify the node/drive name here, too.
+
+```json
+{ "execute": "block-dirty-bitmap-remove",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```
+
+### Enable/Disable:
+
+* Not very useful in current cases, but potentially useful for debugging in the
+future where we'd like to see what information changed only in a specific
+time period:
+
+* To enable (which is, again, the default state after add)
+
+```json
+{ "execute": "block-dirty-bitmap-enable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```
+
+* To disable:
+
+```json
+{ "execute": "block-dirty-bitmap-disable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```
+
+### Resetting
+
+* Resetting a bitmap will clear all information it holds.
+* An incremental backup created from an empty bitmap will copy no data,
+as if nothing has changed.
+
+```json
+{ "execute": "block-dirty-bitmap-clear",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```
+
+## Transactions
+
+### Justification
+Bitmaps can be safely modified when the VM is paused or halted by using
+the basic QMP commands. For instance, you might perform the following actions:
+
+1. Boot the VM in a paused state.
+2. Create a full drive backup of drive0
+3. Create a new bitmap attached to drive0
+4. resume execution of the VM
+5. Incremental backups are ready to be created.
+
+At this point, the bitmap and drive backup would be correctly in sync,
+and incremental backups made from this point forward would be correctly aligned
+to the full drive backup.
+
+This is not particularly useful if we decide we want to start incremental
+backups after the VM has been running for a while, which will allow us to
+perform actions like the following:
+
+1. Boot the VM and begin execution
+2. Using transactions, perform the following operations:
+    * Create 'bitmap0'
+    * Create a full drive backup of drive0
+3. Incremental backups are now ready to be created.
+
+### Supported Bitmap Transactions
+
+* block-dirty-bitmap-add
+* block-dirty-bitmap-enable
+* block-dirty-bitmap-disable
+* block-dirty-bitmap-clear
+
+The usages are identical to their respective QMP commands, but see below
+for examples.
+
+### Example: New Incremental Backup
+
+As outlined in the justification, perhaps we want to create a new incremental
+backup chain attached to a drive.
+
+```json
+{ "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {"type": "block-dirty-bitmap-add",
+       "data": {"node": "drive0", "name": "bitmap0"} },
+      {"type": "drive-backup",
+       "data": {"device": "drive0", "target": "/path/to/full_backup.img",
+                "sync": "full", "format": "qcow2"} }
+    ]
+  }
+}
+```
+
+### Example: New Incremental Backup Anchor Point
+
+Maybe we just want to create a new full backup with an existing bitmap and
+want to reset the bitmap to track the new chain.
+
+```json
+{ "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {"type": "block-dirty-bitmap-clear",
+       "data": {"node": "drive0", "name": "bitmap0"} },
+      {"type": "drive-backup",
+       "data": {"device": "drive0", "target": "/path/to/new_full_backup.img",
+                "sync": "full", "format": "qcow2"} }
+    ]
+  }
+}
+```
+
+## Incremental Backups
+
+The star of the show.
+
+**Nota Bene!** Only incremental backups of entire drives are supported for now.
+So despite the fact that you can attach a bitmap to any arbitrary node, they are
+only currently useful when attached to the root node. This is because
+drive-backup only supports drives/devices instead of arbitrary nodes.
+
+### Example: First Incremental Backup
+
+1. Create a full backup and sync it to the dirty bitmap, as in the transactional
+examples above; or with the VM offline, manually create a full copy and then
+create a new bitmap before the VM begins execution.
+
+    * Let's assume the full backup is named 'full_backup.img.'
+    * Let's assume the bitmap you created is 'bitmap0' attached to 'drive0'.
+
+2. Create a destination image for the incremental backup that utilizes the
+full backup as a backing image.
+
+    * Let's assume it is named 'incremental.0.img'
+
+```sh
+# qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+```
+
+3. Issue the incremental backup command:
+
+```json
+{ "execute": "drive-backup",
+  "arguments": {
+    "device": "drive0",
+    "bitmap": "bitmap0",
+    "target": "incremental.0.img",
+    "format": "qcow2",
+    "sync": "dirty-bitmap",
+    "mode": "existing"
+  }
+}
+```
+
+### Example: Second Incremental Backup
+
+1. Create a new destination image for the incremental backup that points to the
+previous one, e.g.: 'incremental.1.img'
+
+```bash
+# qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2
+```
+
+2. Issue a new incremental backup command. The only difference here is that we
+have changed the target image below.
+
+```json
+{ "execute": "drive-backup",
+  "arguments": {
+    "device": "drive0",
+    "bitmap": "bitmap0",
+    "target": "incremental.1.img",
+    "format": "qcow2",
+    "sync": "dirty-bitmap",
+    "mode": "existing"
+  }
+}
+```
Signature · 1 line
1.9.3
#25 lore raw

Re: [Qemu-devel] [PATCH v14 19/19] docs: incremental backup documentation

docs/bitmaps.md line 1

Quoted 13 lines · John Snow
---
 docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 253 insertions(+)
 create mode 100644 docs/bitmaps.md
docs/bitmaps.md new file +1 -0
new file mode 100644
index 0000000..7cda146
@@ -0,0 +1,253 @@
+# Dirty Bitmaps

No copyright/license? Might be good to add.

Also, I'm a fan of reviewing docs before code (does the design make sense in isolation, and then did we implement it) rather than last in the series (we may have faithfully documented our code, but that includes baking in any design flaws that reviewers are now blind to because of reading the code)

docs/bitmaps.md line 50

Quoted 8 lines
docs/bitmaps.md new file +8 -0
+* To create a new bitmap that tracks changes in 32KiB segments:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+    "granularity": "32768"

s/"32768"/32768/ (we are using a JSON integer, not string)

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+  }
+}
+```
+
+### Deletion
+
+* Can be performed on a disabled bitmap, but not a frozen one.

docs/bitmaps.md line 67

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+* Because bitmaps are only unique to the node to which they are attached,
+you must specify the node/drive name here, too.
+
+```json
+{ "execute": "block-dirty-bitmap-remove",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

No trailing commas in JSON.

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+}
+```
+
+### Enable/Disable:
+
+* Not very useful in current cases, but potentially useful for debugging in the
+future where we'd like to see what information changed only in a specific

docs/bitmaps.md line 84

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+time period:
+
+* To enable (which is, again, the default state after add)
+
+```json
+{ "execute": "block-dirty-bitmap-enable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

and again.

docs/bitmaps.md line 97

Quoted 13 lines
docs/bitmaps.md new file +13 -0
+}
+```
+
+* To disable:
+
+```json
+{ "execute": "block-dirty-bitmap-disable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```

and again. Also, maybe swap these two, since a bitmap defaults to enabled (that is, you would logically use disable first, then enable to re-enable, when testing these out, if there is no way to create an already-disabled bitmap).

docs/bitmaps.md line 110

Quoted 13 lines
docs/bitmaps.md new file +13 -0
+
+### Resetting
+
+* Resetting a bitmap will clear all information it holds.
+* An incremental backup created from an empty bitmap will copy no data,
+as if nothing has changed.
+
+```json
+{ "execute": "block-dirty-bitmap-clear",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

I'll quit pointing out broken trailing JSON commas, on the assumption that you'll fix all of them rather than stopping here :)

Quoted 4 lines
docs/bitmaps.md new file +4 -0
+}
+```
+
+## Transactions

docs/bitmaps.md line 124

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+### Justification
+Bitmaps can be safely modified when the VM is paused or halted by using
+the basic QMP commands. For instance, you might perform the following actions:
+
+1. Boot the VM in a paused state.
+2. Create a full drive backup of drive0
+3. Create a new bitmap attached to drive0
+4. resume execution of the VM
+5. Incremental backups are ready to be created.

Consistency on using trailing '.'?

Quoted 4 lines
docs/bitmaps.md new file +4 -0
+
+At this point, the bitmap and drive backup would be correctly in sync,
+and incremental backups made from this point forward would be correctly aligned
+to the full drive backup.

docs/bitmaps.md line 138

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+This is not particularly useful if we decide we want to start incremental
+backups after the VM has been running for a while, which will allow us to
+perform actions like the following:
+
+1. Boot the VM and begin execution
+2. Using transactions, perform the following operations:
+    * Create 'bitmap0'
+    * Create a full drive backup of drive0
+3. Incremental backups are now ready to be created.

It's unclear whether point 2 is describing two separate transactions, or one transaction with two steps. If it's just one, I'd word it:

Using a single transaction, perform the following operations:

docs/bitmaps.md line 195

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+
+The star of the show.
+
+**Nota Bene!** Only incremental backups of entire drives are supported for now.
+So despite the fact that you can attach a bitmap to any arbitrary node, they are
+only currently useful when attached to the root node. This is because
+drive-backup only supports drives/devices instead of arbitrary nodes.

Maybe it's time to think about adding a node-backup :) But doesn't have to be this series.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#26 lore raw

Re: [Qemu-devel] [PATCH v14 19/19] docs: incremental backup documentation

Quoted 7 lines · Eric Blake
Quoted 6 lines · John Snow
---
  docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 253 insertions(+)

create mode 100644 docs/bitmaps.md

Quoted 10 lines

docs/bitmaps.md line 1

Quoted 7 lines
docs/bitmaps.md new file +1 -0
new file mode 100644
index 0000000..7cda146
@@ -0,0 +1,253 @@
+# Dirty Bitmaps

No copyright/license? Might be good to add.

OK.

Quoted 6 lines

Also, I'm a fan of reviewing docs before code (does the design make sense in isolation, and then did we implement it) rather than last in the series (we may have faithfully documented our code, but that includes baking in any design flaws that reviewers are now blind to because of reading the code)

Sorry, I definitely did write this *after* and I selfishly put it at the end of the series to avoid disrupting the patch numbers from the previous revision.

Quoted 70 lines

docs/bitmaps.md line 50

Quoted 8 lines
docs/bitmaps.md new file +8 -0
+* To create a new bitmap that tracks changes in 32KiB segments:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+    "granularity": "32768"

s/"32768"/32768/ (we are using a JSON integer, not string)

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+  }
+}
+```
+
+### Deletion
+
+* Can be performed on a disabled bitmap, but not a frozen one.

docs/bitmaps.md line 67

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+* Because bitmaps are only unique to the node to which they are attached,
+you must specify the node/drive name here, too.
+
+```json
+{ "execute": "block-dirty-bitmap-remove",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

No trailing commas in JSON.

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+}
+```
+
+### Enable/Disable:
+
+* Not very useful in current cases, but potentially useful for debugging in the
+future where we'd like to see what information changed only in a specific

docs/bitmaps.md line 84

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+time period:
+
+* To enable (which is, again, the default state after add)
+
+```json
+{ "execute": "block-dirty-bitmap-enable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

and again.

docs/bitmaps.md line 97

Quoted 13 lines
docs/bitmaps.md new file +13 -0
+}
+```
+
+* To disable:
+
+```json
+{ "execute": "block-dirty-bitmap-disable",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }
+}
+```

and again. Also, maybe swap these two, since a bitmap defaults to enabled (that is, you would logically use disable first, then enable to re-enable, when testing these out, if there is no way to create an already-disabled bitmap).

Quoted 17 lines

docs/bitmaps.md line 110

Quoted 13 lines
docs/bitmaps.md new file +13 -0
+
+### Resetting
+
+* Resetting a bitmap will clear all information it holds.
+* An incremental backup created from an empty bitmap will copy no data,
+as if nothing has changed.
+
+```json
+{ "execute": "block-dirty-bitmap-clear",
+  "arguments": {
+    "node": "drive0",
+    "name": "bitmap0",
+  }

I'll quit pointing out broken trailing JSON commas, on the assumption that you'll fix all of them rather than stopping here :)

Yeah, sorry! I have discovered today that I am *awful* at writing out json by hand. I promise the commands work otherwise!

I will fix up the documentation here; if this winds up being the *ONLY* thing wrong with V14, I would prefer this doc be dropped from this series and I will just send a v2 for the doc by itself afterwards.

Quoted 35 lines
Quoted 4 lines
docs/bitmaps.md new file +4 -0
+}
+```
+
+## Transactions

docs/bitmaps.md line 124

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+### Justification
+Bitmaps can be safely modified when the VM is paused or halted by using
+the basic QMP commands. For instance, you might perform the following actions:
+
+1. Boot the VM in a paused state.
+2. Create a full drive backup of drive0
+3. Create a new bitmap attached to drive0
+4. resume execution of the VM
+5. Incremental backups are ready to be created.

Consistency on using trailing '.'?

Quoted 4 lines
docs/bitmaps.md new file +4 -0
+
+At this point, the bitmap and drive backup would be correctly in sync,
+and incremental backups made from this point forward would be correctly aligned
+to the full drive backup.

docs/bitmaps.md line 138

Quoted 10 lines
docs/bitmaps.md new file +10 -0
+
+This is not particularly useful if we decide we want to start incremental
+backups after the VM has been running for a while, which will allow us to
+perform actions like the following:
+
+1. Boot the VM and begin execution
+2. Using transactions, perform the following operations:
+    * Create 'bitmap0'
+    * Create a full drive backup of drive0
+3. Incremental backups are now ready to be created.

It's unclear whether point 2 is describing two separate transactions, or one transaction with two steps. If it's just one, I'd word it:

Quoted 3 lines

Using a single transaction, perform the following operations:

OK.

Quoted 12 lines

docs/bitmaps.md line 195

Quoted 7 lines
docs/bitmaps.md new file +7 -0
+
+The star of the show.
+
+**Nota Bene!** Only incremental backups of entire drives are supported for now.
+So despite the fact that you can attach a bitmap to any arbitrary node, they are
+only currently useful when attached to the root node. This is because
+drive-backup only supports drives/devices instead of arbitrary nodes.

Maybe it's time to think about adding a node-backup :) But doesn't have to be this series.

I will crawl before I walk. And certainly before I start running and flying :)

Thanks,
--JS

#27 lore raw
In reply to John Snow
Quoted 11 lines · John Snow

This series requires: [PATCH v3] blkdebug: fix "once" rule

Salutations, weary traveler. Might I interest you in some incremental backups? Only slightly used, just like new!

This patchset enables the in-memory part of the incremental backup feature. Bitmap persistence and bitmap migration will be addressed in future patchsets by Vladimir Sementsov-Ogievskiy.

This series and most patches were originally by Fam Zheng, though I have corrupted them far beyond their original presentation.

Just a note: Please keep my R-bs on the iotest patches if you need to change the test number (*cough* this is a hint *cough cough*).

Max