Skip to contentSkip to conversation

qapi: block-dirty-bitmap-remove transaction action

PATCH Applied John Snow 3 participants 14 messages Jul 8, 2019 – Jul 26, 2019 Block layer

lore
Share & tools
lore

John Snow

author
#0 lore raw

v3 cover qapi: block-dirty-bitmap-remove transaction action

Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action

===
V3:
===

001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'

- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr

===
V2:
===

It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.

Patches 1 and 3 (formerly 4) included as-is.

John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action

Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254

 block.c                        |   2 +-
 block/dirty-bitmap.c           |  15 +++--
 blockdev.c                     | 105 ++++++++++++++++++++++++++-------
 include/block/dirty-bitmap.h   |   2 +-
 migration/block-dirty-bitmap.c |   2 +-
 qapi/transaction.json          |   2 +
 tests/qemu-iotests/254         |  30 +++++++++-
 tests/qemu-iotests/254.out     |  82 +++++++++++++++++++++++++
 8 files changed, 206 insertions(+), 34 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v3 1/3 blockdev: reduce aio_context locked sections in bitmap add/remove

From: Vladimir Sementsov-Ogievskiy <[email protected]>

Commit 0a6c86d024c52 returned these locks back to add/remove functionality, to protect from intersection of persistent bitmap related IO with other IO. But other bitmap-related functions called here are unrelated to the problem, and there are no needs to keep these calls inside critical sections.

---
 blockdev.c | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)
blockdev.c +13 -17
index 4d141e9a1f..01248252ca 100644
@@ -2811,7 +2811,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
 {
     BlockDriverState *bs;
     BdrvDirtyBitmap *bitmap;
-    AioContext *aio_context = NULL;
 
     if (!name || name[0] == '\0') {
         error_setg(errp, "Bitmap name cannot be empty");
@@ -2847,16 +2846,20 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
     }
 
     if (persistent) {
-        aio_context = bdrv_get_aio_context(bs);
+        AioContext *aio_context = bdrv_get_aio_context(bs);
+        bool ok;
+
         aio_context_acquire(aio_context);
-        if (!bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp)) {
-            goto out;
+        ok = bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp);
+        aio_context_release(aio_context);
+        if (!ok) {
+            return;
         }
     }
 
     bitmap = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
     if (bitmap == NULL) {
-        goto out;
+        return;
     }
 
     if (disabled) {
@@ -2864,10 +2867,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
     }
 
     bdrv_dirty_bitmap_set_persistence(bitmap, persistent);
- out:
-    if (aio_context) {
-        aio_context_release(aio_context);
-    }
 }
 
 void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
@@ -2875,8 +2874,6 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
 {
     BlockDriverState *bs;
     BdrvDirtyBitmap *bitmap;
-    Error *local_err = NULL;
-    AioContext *aio_context = NULL;
 
     bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
     if (!bitmap || !bs) {
@@ -2889,20 +2886,19 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
     }
 
     if (bdrv_dirty_bitmap_get_persistence(bitmap)) {
-        aio_context = bdrv_get_aio_context(bs);
+        AioContext *aio_context = bdrv_get_aio_context(bs);
+        Error *local_err = NULL;
+
         aio_context_acquire(aio_context);
         bdrv_remove_persistent_dirty_bitmap(bs, name, &local_err);
+        aio_context_release(aio_context);
         if (local_err != NULL) {
             error_propagate(errp, local_err);
-            goto out;
+            return;
         }
     }
 
     bdrv_release_dirty_bitmap(bs, bitmap);
- out:
-    if (aio_context) {
-        aio_context_release(aio_context);
-    }
 }
 
 /**
Signature · 1 line
2.21.0
Reviewed-by on v3 1/3
#2 lore raw

Re: [Qemu-devel] [PATCH v3 1/3] blockdev: reduce aio_context locked sections in bitmap add/remove

Quoted 8 lines · John Snow

From: Vladimir Sementsov-Ogievskiy <[email protected]>

Commit 0a6c86d024c52 returned these locks back to add/remove functionality, to protect from intersection of persistent bitmap related IO with other IO. But other bitmap-related functions called here are unrelated to the problem, and there are no needs to keep these calls inside critical sections.

Quoted 6 lines
---
 blockdev.c | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)
author
#3 lore raw

PATCH v3 2/3 qapi: implement block-dirty-bitmap-remove transaction action

In reply to John Snow

It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.

---
 block.c                        |  2 +-
 block/dirty-bitmap.c           | 15 +++----
 blockdev.c                     | 79 +++++++++++++++++++++++++++++++---
 include/block/dirty-bitmap.h   |  2 +-
 migration/block-dirty-bitmap.c |  2 +-
 qapi/transaction.json          |  2 +
 6 files changed, 85 insertions(+), 17 deletions(-)
block.c +1 -1
index c139540f2b..5195d4b910 100644
@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,
     for (bm = bdrv_dirty_bitmap_next(bs, NULL); bm;
          bm = bdrv_dirty_bitmap_next(bs, bm))
     {
-        bdrv_dirty_bitmap_set_migration(bm, false);
+        bdrv_dirty_bitmap_skip_store(bm, false);
     }
 
     ret = refresh_total_sectors(bs, bs->total_sectors);
block/dirty-bitmap.c +7 -8
index 95a9c2a5d8..a308e1f84b 100644
@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {
     bool inconsistent;          /* bitmap is persistent, but inconsistent.
                                    It cannot be used at all in any way, except
                                    a QMP user can remove it. */
-    bool migration;             /* Bitmap is selected for migration, it should
-                                   not be stored on the next inactivation
-                                   (persistent flag doesn't matter until next
-                                   invalidation).*/
+    bool skip_store;            /* We are either migrating or deleting this
+                                 * bitmap; it should not be stored on the next
+                                 * inactivation. */
     QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
 
@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap)
 }
 
 /* Called with BQL taken. */
-void bdrv_dirty_bitmap_set_migration(BdrvDirtyBitmap *bitmap, bool migration)
+void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip)
 {
     qemu_mutex_lock(bitmap->mutex);
-    bitmap->migration = migration;
+    bitmap->skip_store = skip;
     qemu_mutex_unlock(bitmap->mutex);
 }
 
 bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap)
 {
-    return bitmap->persistent && !bitmap->migration;
+    return bitmap->persistent && !bitmap->skip_store;
 }
 
 bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap)
@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
 {
     BdrvDirtyBitmap *bm;
     QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
-        if (bm->persistent && !bm->readonly && !bm->migration) {
+        if (bm->persistent && !bm->readonly && !bm->skip_store) {
             return true;
         }
     }
blockdev.c +73 -6

118 lines not on this page · raw message

include/block/dirty-bitmap.h +1 -1
index 62682eb865..a21d54a090 100644
@@ -83,7 +83,7 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_bitmap_set_busy(BdrvDirtyBitmap *bitmap, bool busy);
 void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
                              HBitmap **backup, Error **errp);
-void bdrv_dirty_bitmap_set_migration(BdrvDirtyBitmap *bitmap, bool migration);
+void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip);
 
 /* Functions that require manual locking.  */
 void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap);
migration/block-dirty-bitmap.c +1 -1
index 4a896a09eb..d650ba4c23 100644
@@ -326,7 +326,7 @@ static int init_dirty_bitmap_migration(void)
 
     /* unset migration flags here, to not roll back it */
     QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
-        bdrv_dirty_bitmap_set_migration(dbms->bitmap, true);
+        bdrv_dirty_bitmap_skip_store(dbms->bitmap, true);
     }
 
     if (QSIMPLEQ_EMPTY(&dirty_bitmap_mig_state.dbms_list)) {
qapi/transaction.json +2 -0
index 95edb78227..da95b804aa 100644
@@ -45,6 +45,7 @@
 #
 # - @abort: since 1.6
 # - @block-dirty-bitmap-add: since 2.5
+# - @block-dirty-bitmap-remove: since 4.1
 # - @block-dirty-bitmap-clear: since 2.5
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
@@ -61,6 +62,7 @@
   'data': {
        'abort': 'Abort',
        'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd',
+       'block-dirty-bitmap-remove': 'BlockDirtyBitmap',
        'block-dirty-bitmap-clear': 'BlockDirtyBitmap',
        'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
Signature · 1 line
2.21.0
Reviewed-by on v3 2/3
#4 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action

Quoted 15 lines · John Snow

It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.

---
 block.c                        |  2 +-
 block/dirty-bitmap.c           | 15 +++----
 blockdev.c                     | 79 +++++++++++++++++++++++++++++++---
 include/block/dirty-bitmap.h   |  2 +-
 migration/block-dirty-bitmap.c |  2 +-
 qapi/transaction.json          |  2 +
 6 files changed, 85 insertions(+), 17 deletions(-)

[...]

qapi/transaction.json line 48

Quoted 9 lines
qapi/transaction.json +1 -0
index 95edb78227..da95b804aa 100644
@@ -45,6 +45,7 @@
 #
 # - @abort: since 1.6
 # - @block-dirty-bitmap-add: since 2.5
+# - @block-dirty-bitmap-remove: since 4.1

Optimistic.

But anyway:

Quoted 3 lines
qapi/transaction.json +0 -0
 # - @block-dirty-bitmap-clear: since 2.5
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
author
#5 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action

Quoted 31 lines · Max Reitz
Quoted 15 lines · John Snow

It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.

---
 block.c                        |  2 +-
 block/dirty-bitmap.c           | 15 +++----
 blockdev.c                     | 79 +++++++++++++++++++++++++++++++---
 include/block/dirty-bitmap.h   |  2 +-
 migration/block-dirty-bitmap.c |  2 +-
 qapi/transaction.json          |  2 +
 6 files changed, 85 insertions(+), 17 deletions(-)

[...]

qapi/transaction.json line 48

Quoted 9 lines
qapi/transaction.json +1 -0
index 95edb78227..da95b804aa 100644
@@ -45,6 +45,7 @@
 #
 # - @abort: since 1.6
 # - @block-dirty-bitmap-add: since 2.5
+# - @block-dirty-bitmap-remove: since 4.1

Optimistic.

Quoted 8 lines

But anyway:

Quoted 3 lines
qapi/transaction.json +0 -0
 # - @block-dirty-bitmap-clear: since 2.5
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0

Hah, oops. OK, I'll adjust that (and your comment in patch 3) and stage them.

Thank you!

#6 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action

In reply to John Snow
Quoted 105 lines · John Snow

It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.

---
  block.c                        |  2 +-
  block/dirty-bitmap.c           | 15 +++----
  blockdev.c                     | 79 +++++++++++++++++++++++++++++++---
  include/block/dirty-bitmap.h   |  2 +-
  migration/block-dirty-bitmap.c |  2 +-
  qapi/transaction.json          |  2 +
  6 files changed, 85 insertions(+), 17 deletions(-)
block.c +1 -1
index c139540f2b..5195d4b910 100644
@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,
      for (bm = bdrv_dirty_bitmap_next(bs, NULL); bm;
           bm = bdrv_dirty_bitmap_next(bs, bm))
      {
-        bdrv_dirty_bitmap_set_migration(bm, false);
+        bdrv_dirty_bitmap_skip_store(bm, false);
      }
 
      ret = refresh_total_sectors(bs, bs->total_sectors);
block/dirty-bitmap.c +7 -8
index 95a9c2a5d8..a308e1f84b 100644
@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {
      bool inconsistent;          /* bitmap is persistent, but inconsistent.
                                     It cannot be used at all in any way, except
                                     a QMP user can remove it. */
-    bool migration;             /* Bitmap is selected for migration, it should
-                                   not be stored on the next inactivation
-                                   (persistent flag doesn't matter until next
-                                   invalidation).*/
+    bool skip_store;            /* We are either migrating or deleting this
+                                 * bitmap; it should not be stored on the next
+                                 * inactivation. */
      QLIST_ENTRY(BdrvDirtyBitmap) list;
  };
 
@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap)
  }
 
  /* Called with BQL taken. */
-void bdrv_dirty_bitmap_set_migration(BdrvDirtyBitmap *bitmap, bool migration)
+void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip)
  {
      qemu_mutex_lock(bitmap->mutex);
-    bitmap->migration = migration;
+    bitmap->skip_store = skip;
      qemu_mutex_unlock(bitmap->mutex);
  }
 
  bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap)
  {
-    return bitmap->persistent && !bitmap->migration;
+    return bitmap->persistent && !bitmap->skip_store;
  }
 
  bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap)
@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
  {
      BdrvDirtyBitmap *bm;
      QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
-        if (bm->persistent && !bm->readonly && !bm->migration) {
+        if (bm->persistent && !bm->readonly && !bm->skip_store) {
              return true;
          }
      }
blockdev.c +21 -0
index 01248252ca..800b3dcb42 100644
@@ -2134,6 +2134,51 @@ static void block_dirty_bitmap_merge_prepare(BlkActionState *common,
                                                  errp);
  }
 
+static BdrvDirtyBitmap *do_block_dirty_bitmap_remove(
+        const char *node, const char *name, bool release,
+        BlockDriverState **bitmap_bs, Error **errp);
+
+static void block_dirty_bitmap_remove_prepare(BlkActionState *common,
+                                              Error **errp)
+{
+    BlockDirtyBitmap *action;
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (action_check_completion_mode(common, errp) < 0) {
+        return;
+    }
+
+    action = common->action->u.block_dirty_bitmap_remove.data;
+
+    state->bitmap = do_block_dirty_bitmap_remove(action->node, action->name,
+                                                 false, &state->bs, errp);
+    if (state->bitmap) {
+        bdrv_dirty_bitmap_skip_store(state->bitmap, true);

blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2167

Quoted 10 lines
blockdev.c +10 -0
+        bdrv_dirty_bitmap_set_busy(state->bitmap, true);
+    }
+}
+
+static void block_dirty_bitmap_remove_abort(BlkActionState *common)
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->bitmap) {

Hmm, interesting, I thought, abort should not be called, if prepare failed, so the following may be done unconditionally?

blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2172

Quoted 5 lines
blockdev.c +5 -0
+        bdrv_dirty_bitmap_skip_store(state->bitmap, false);
+        bdrv_dirty_bitmap_set_busy(state->bitmap, false);
+    }
+}
+

[..]

OK, I agree, good idea to reuse BUSY and migration field here and avoid new API. Now release-prepare is less "release", but I don't see any real problems with it. Maybe, we need something to be noted in docs.

Hmm, as we are not in a hurry now, we may wait for Nikolay to check this on his scenarios.

Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action

Quoted 111 lines · Vladimir Sementsov-Ogievskiy
Quoted 110 lines · John Snow

It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.

---
  block.c                        |  2 +-
  block/dirty-bitmap.c           | 15 +++----
  blockdev.c                     | 79 +++++++++++++++++++++++++++++++---
  include/block/dirty-bitmap.h   |  2 +-
  migration/block-dirty-bitmap.c |  2 +-
  qapi/transaction.json          |  2 +
  6 files changed, 85 insertions(+), 17 deletions(-)
block.c +1 -1
index c139540f2b..5195d4b910 100644
@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,
      for (bm = bdrv_dirty_bitmap_next(bs, NULL); bm;
           bm = bdrv_dirty_bitmap_next(bs, bm))
      {
-        bdrv_dirty_bitmap_set_migration(bm, false);
+        bdrv_dirty_bitmap_skip_store(bm, false);
      }
 
      ret = refresh_total_sectors(bs, bs->total_sectors);
block/dirty-bitmap.c +7 -8
index 95a9c2a5d8..a308e1f84b 100644
@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {
      bool inconsistent;          /* bitmap is persistent, but inconsistent.
                                     It cannot be used at all in any way, except
                                     a QMP user can remove it. */
-    bool migration;             /* Bitmap is selected for migration, it should
-                                   not be stored on the next inactivation
-                                   (persistent flag doesn't matter until next
-                                   invalidation).*/
+    bool skip_store;            /* We are either migrating or deleting this
+                                 * bitmap; it should not be stored on the next
+                                 * inactivation. */
      QLIST_ENTRY(BdrvDirtyBitmap) list;
  };
 
@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap)
  }
 
  /* Called with BQL taken. */
-void bdrv_dirty_bitmap_set_migration(BdrvDirtyBitmap *bitmap, bool migration)
+void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip)
  {
      qemu_mutex_lock(bitmap->mutex);
-    bitmap->migration = migration;
+    bitmap->skip_store = skip;
      qemu_mutex_unlock(bitmap->mutex);
  }
 
  bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap)
  {
-    return bitmap->persistent && !bitmap->migration;
+    return bitmap->persistent && !bitmap->skip_store;
  }
 
  bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap)
@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
  {
      BdrvDirtyBitmap *bm;
      QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
-        if (bm->persistent && !bm->readonly && !bm->migration) {
+        if (bm->persistent && !bm->readonly && !bm->skip_store) {
              return true;
          }
      }
blockdev.c +26 -0
index 01248252ca..800b3dcb42 100644
@@ -2134,6 +2134,51 @@ static void block_dirty_bitmap_merge_prepare(BlkActionState *common,
                                                  errp);
  }
 
+static BdrvDirtyBitmap *do_block_dirty_bitmap_remove(
+        const char *node, const char *name, bool release,
+        BlockDriverState **bitmap_bs, Error **errp);
+
+static void block_dirty_bitmap_remove_prepare(BlkActionState *common,
+                                              Error **errp)
+{
+    BlockDirtyBitmap *action;
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (action_check_completion_mode(common, errp) < 0) {
+        return;
+    }
+
+    action = common->action->u.block_dirty_bitmap_remove.data;
+
+    state->bitmap = do_block_dirty_bitmap_remove(action->node, action->name,
+                                                 false, &state->bs, errp);
+    if (state->bitmap) {
+        bdrv_dirty_bitmap_skip_store(state->bitmap, true);
+        bdrv_dirty_bitmap_set_busy(state->bitmap, true);
+    }
+}
+
+static void block_dirty_bitmap_remove_abort(BlkActionState *common)
Quoted 9 lines

blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2156

Quoted 5 lines
blockdev.c +5 -0
+{
+    BlockDirtyBitmapState *state = DO_UPCAST(BlockDirtyBitmapState,
+                                             common, common);
+
+    if (state->bitmap) {

Hmm, interesting, I thought, abort should not be called, if prepare failed, so the following may be done unconditionally?

I don't think so:

qmp_transaction:

	...
        state->ops->prepare(state, &local_err);
        if (local_err) {
            error_propagate(errp, local_err);
	    goto delete_and_fail;
	}
	...
delete_and_fail:
    ...
    QTAILQ_FOREACH_REVERSE(state, &snap_bdrv_states, entry) {
        if (state->ops->abort) {
            state->ops->abort(state);
        }
    }
Quoted 8 lines
Quoted 5 lines
blockdev.c +5 -0
+        bdrv_dirty_bitmap_skip_store(state->bitmap, false);
+        bdrv_dirty_bitmap_set_busy(state->bitmap, false);
+    }
+}
+

[..]

Quoted 7 lines

OK, I agree, good idea to reuse BUSY and migration field here and avoid new API. Now release-prepare is less "release", but I don't see any real problems with it. Maybe, we need something to be noted in docs.

Hmm, as we are not in a hurry now, we may wait for Nikolay to check this on his scenarios.

OK, you have time. These patches are all sitting in my bitmaps branch and I haven't sent the PR for them yet.

--js

author
#8 lore raw

PATCH v3 3/3 iotests: test bitmap moving inside 254

In reply to John Snow

From: Vladimir Sementsov-Ogievskiy <[email protected]>

Test persistent bitmap copying with and without removal of original bitmap.

---
 tests/qemu-iotests/254     | 30 +++++++++++++-
 tests/qemu-iotests/254.out | 82 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 110 insertions(+), 2 deletions(-)
tests/qemu-iotests/254 +28 -2
index 8edba91c5d..9a57bccc26 100755
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 #
-# Test external snapshot with bitmap copying.
+# Test external snapshot with bitmap copying and moving.
 #
 # Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved.
 #
@@ -32,6 +32,10 @@ vm = iotests.VM().add_drive(disk, opts='node-name=base')
 vm.launch()
 
 vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap0')
+vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap1',
+           persistent=True)
+vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap2',
+           persistent=True)
 
 vm.hmp_qemu_io('drive0', 'write 0 512K')
 
@@ -39,16 +43,38 @@ vm.qmp_log('transaction', indent=2, actions=[
     {'type': 'blockdev-snapshot-sync',
      'data': {'device': 'drive0', 'snapshot-file': top,
               'snapshot-node-name': 'snap'}},
+
+    # copy non-persistent bitmap0
     {'type': 'block-dirty-bitmap-add',
      'data': {'node': 'snap', 'name': 'bitmap0'}},
     {'type': 'block-dirty-bitmap-merge',
      'data': {'node': 'snap', 'target': 'bitmap0',
-              'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}}
+              'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}},
+
+    # copy persistent bitmap1, original will be saved to base image
+    {'type': 'block-dirty-bitmap-add',
+     'data': {'node': 'snap', 'name': 'bitmap1', 'persistent': True}},
+    {'type': 'block-dirty-bitmap-merge',
+     'data': {'node': 'snap', 'target': 'bitmap1',
+              'bitmaps': [{'node': 'base', 'name': 'bitmap1'}]}},
+
+    # move persistent bitmap1, original will be removed and not saved
+    # to base image
+    {'type': 'block-dirty-bitmap-add',
+     'data': {'node': 'snap', 'name': 'bitmap2', 'persistent': True}},
+    {'type': 'block-dirty-bitmap-merge',
+     'data': {'node': 'snap', 'target': 'bitmap2',
+              'bitmaps': [{'node': 'base', 'name': 'bitmap2'}]}},
+    {'type': 'block-dirty-bitmap-remove',
+     'data': {'node': 'base', 'name': 'bitmap2'}}
 ], filters=[iotests.filter_qmp_testfiles])
 
 result = vm.qmp('query-block')['return'][0]
 log("query-block: device = {}, node-name = {}, dirty-bitmaps:".format(
     result['device'], result['inserted']['node-name']))
 log(result['dirty-bitmaps'], indent=2)
+log("\nbitmaps in backing image:")
+log(result['inserted']['image']['backing-image']['format-specific'] \
+    ['data']['bitmaps'], indent=2)
 
 vm.shutdown()
tests/qemu-iotests/254.out +82 -0

102 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v3 3/3
#9 lore raw

Re: [Qemu-devel] [PATCH v3 3/3] iotests: test bitmap moving inside 254

Quoted 16 lines · John Snow

From: Vladimir Sementsov-Ogievskiy <[email protected]>

Test persistent bitmap copying with and without removal of original bitmap.

---
 tests/qemu-iotests/254     | 30 +++++++++++++-
 tests/qemu-iotests/254.out | 82 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 110 insertions(+), 2 deletions(-)
tests/qemu-iotests/254 +0 -0
index 8edba91c5d..9a57bccc26 100755

[...]

Quoted 11 lines
tests/qemu-iotests/254 +2 -1
@@ -39,16 +43,38 @@ vm.qmp_log('transaction', indent=2, actions=[
     {'type': 'blockdev-snapshot-sync',
      'data': {'device': 'drive0', 'snapshot-file': top,
               'snapshot-node-name': 'snap'}},
+
+    # copy non-persistent bitmap0
     {'type': 'block-dirty-bitmap-add',
      'data': {'node': 'snap', 'name': 'bitmap0'}},
     {'type': 'block-dirty-bitmap-merge',
      'data': {'node': 'snap', 'target': 'bitmap0',
-              'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}}

tests/qemu-iotests/254 vm.qmp_log('transaction', indent=2, actions=[ · line 61

Quoted 10 lines
tests/qemu-iotests/254 +10 -0
+              'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}},
+
+    # copy persistent bitmap1, original will be saved to base image
+    {'type': 'block-dirty-bitmap-add',
+     'data': {'node': 'snap', 'name': 'bitmap1', 'persistent': True}},
+    {'type': 'block-dirty-bitmap-merge',
+     'data': {'node': 'snap', 'target': 'bitmap1',
+              'bitmaps': [{'node': 'base', 'name': 'bitmap1'}]}},
+
+    # move persistent bitmap1, original will be removed and not saved

*bitmap2

Quoted 9 lines
tests/qemu-iotests/254 +8 -0
+    # to base image
+    {'type': 'block-dirty-bitmap-add',
+     'data': {'node': 'snap', 'name': 'bitmap2', 'persistent': True}},
+    {'type': 'block-dirty-bitmap-merge',
+     'data': {'node': 'snap', 'target': 'bitmap2',
+              'bitmaps': [{'node': 'base', 'name': 'bitmap2'}]}},
+    {'type': 'block-dirty-bitmap-remove',
+     'data': {'node': 'base', 'name': 'bitmap2'}}
 ], filters=[iotests.filter_qmp_testfiles])
author Applied
#10 lore raw
In reply to John Snow
Quoted 32 lines · John Snow

Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action

===
V3:
===

001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'

- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr

===
V2:
===

It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.

Patches 1 and 3 (formerly 4) included as-is.

John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action

Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254

Quoted 10 lines
 block.c                        |   2 +-
 block/dirty-bitmap.c           |  15 +++--
 blockdev.c                     | 105 ++++++++++++++++++++++++++-------
 include/block/dirty-bitmap.h   |   2 +-
 migration/block-dirty-bitmap.c |   2 +-
 qapi/transaction.json          |   2 +
 tests/qemu-iotests/254         |  30 +++++++++-
 tests/qemu-iotests/254.out     |  82 +++++++++++++++++++++++++
 8 files changed, 206 insertions(+), 34 deletions(-)

Thanks, applied to my bitmaps tree:

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js

(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)

#11 lore raw
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow

Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action

===
V3:
===

001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'

- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr

===
V2:
===

It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.

Patches 1 and 3 (formerly 4) included as-is.

John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action

Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254

Quoted 10 lines
  block.c                        |   2 +-
  block/dirty-bitmap.c           |  15 +++--
  blockdev.c                     | 105 ++++++++++++++++++++++++++-------
  include/block/dirty-bitmap.h   |   2 +-
  migration/block-dirty-bitmap.c |   2 +-
  qapi/transaction.json          |   2 +
  tests/qemu-iotests/254         |  30 +++++++++-
  tests/qemu-iotests/254.out     |  82 +++++++++++++++++++++++++
  8 files changed, 206 insertions(+), 34 deletions(-)

Thanks, applied to my bitmaps tree:

Quoted 9 lines

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js

(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)

And finally I'm here :)

Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.

Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.

Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..

Anyway, it's cool, thanks!

Signature · 2 lines
Best regards,
Vladimir
author
#12 lore raw
Quoted 69 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow

Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action

===
V3:
===

001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'

- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr

===
V2:
===

It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.

Patches 1 and 3 (formerly 4) included as-is.

John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action

Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254

Quoted 10 lines
  block.c                        |   2 +-
  block/dirty-bitmap.c           |  15 +++--
  blockdev.c                     | 105 ++++++++++++++++++++++++++-------
  include/block/dirty-bitmap.h   |   2 +-
  migration/block-dirty-bitmap.c |   2 +-
  qapi/transaction.json          |   2 +
  tests/qemu-iotests/254         |  30 +++++++++-
  tests/qemu-iotests/254.out     |  82 +++++++++++++++++++++++++
  8 files changed, 206 insertions(+), 34 deletions(-)

Thanks, applied to my bitmaps tree:

Quoted 9 lines

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js

(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)

And finally I'm here :)

Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.

Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.

Quoted 6 lines

Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..

Anyway, it's cool, thanks!

Wow!

Congratulations Vladimir!

#13 lore raw
Quoted 76 lines · John Snow
Quoted 73 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow

Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action

===
V3:
===

001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'

- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr

===
V2:
===

It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.

Patches 1 and 3 (formerly 4) included as-is.

John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action

Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254

Quoted 10 lines
   block.c                        |   2 +-
   block/dirty-bitmap.c           |  15 +++--
   blockdev.c                     | 105 ++++++++++++++++++++++++++-------
   include/block/dirty-bitmap.h   |   2 +-
   migration/block-dirty-bitmap.c |   2 +-
   qapi/transaction.json          |   2 +
   tests/qemu-iotests/254         |  30 +++++++++-
   tests/qemu-iotests/254.out     |  82 +++++++++++++++++++++++++
   8 files changed, 206 insertions(+), 34 deletions(-)

Thanks, applied to my bitmaps tree:

Quoted 9 lines

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js

(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)

And finally I'm here :)

Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.

Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.

Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..

Quoted 7 lines
Quoted 2 lines

Anyway, it's cool, thanks!

Wow!

Congratulations Vladimir!

Thank you!

Signature · 2 lines
Best regards,
Vladimir