Skip to contentSkip to conversation

migration: Add block-bitmap-mapping parameter

PATCH Superseded Max Reitz 3 participants 9 messages Jul 22, 2020 – Aug 13, 2020

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v3 cover migration: Add block-bitmap-mapping parameter

RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html

Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3

Hi,

This new migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

v3:
- Patch 1:
- Add notes on the fact that the destination won’t ever know about any bitmaps that aren’t sent, so you can freely drop what you want, and you’re completely free in renaming bitmaps and putting them on “other” nodes (whatever “other” node means in the context of migration, because that’s kind of one of the problems this series is trying to solve: The fact that you can’t trivially match nodes between source and destination)
- Fix an assertion

- Patch 2: s/pass/time.sleep(0.2)/

- Patch 3:
- Add copyright line
- Use format string instead of %
- s/pass/time.sleep(0.1)/
- s/wait_for_runstate/wait_migration/ on the destination to wait for the migration to actually complete
- Replace the “info migrate_parameters” parsing code by a multiline regex
- Test what happens when the destination has a mapping that isn’t used because there are not bitmaps to be transferred (which breaks the assertion in patch 1 as it was in v2)
- Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the destination instead of the source

git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter' 002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()' 003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'

Max Reitz (3):
migration: Add block-bitmap-mapping parameter iotests.py: Add wait_for_runstate()
iotests: Test node/bitmap aliases during migration

 qapi/migration.json            | 104 ++++++-
 migration/migration.h          |   3 +
 migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
 migration/migration.c          |  30 ++
 monitor/hmp-cmds.c             |  30 ++
 tests/qemu-iotests/300         | 515 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/300.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |   5 +
 9 files changed, 1011 insertions(+), 55 deletions(-)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v3 1/3 migration: Add block-bitmap-mapping parameter

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

---
 qapi/migration.json            | 104 ++++++++-
 migration/migration.h          |   3 +
 migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++-----
 migration/migration.c          |  30 +++
 monitor/hmp-cmds.c             |  30 +++
 5 files changed, 485 insertions(+), 55 deletions(-)
qapi/migration.json +101 -3
index d5000558c6..d7e953cd73 100644
@@ -507,6 +507,44 @@
   'data': [ 'none', 'zlib',
             { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
 
+##
+# @BitmapMigrationBitmapAlias:
+#
+# @name: The name of the bitmap.
+#
+# @alias: An alias name for migration (for example the bitmap name on
+#         the opposite site).
+#
+# Since: 5.1
+##
+{ 'struct': 'BitmapMigrationBitmapAlias',
+  'data': {
+      'name': 'str',
+      'alias': 'str'
+  } }
+
+##
+# @BitmapMigrationNodeAlias:
+#
+# Maps a block node name and the bitmaps it has to aliases for dirty
+# bitmap migration.
+#
+# @node-name: A block node name.
+#
+# @alias: An alias block node name for migration (for example the
+#         node name on the opposite site).
+#
+# @bitmaps: Mappings for the bitmaps on this node.
+#
+# Since: 5.1
+##
+{ 'struct': 'BitmapMigrationNodeAlias',
+  'data': {
+      'node-name': 'str',
+      'alias': 'str',
+      'bitmaps': [ 'BitmapMigrationBitmapAlias' ]
+  } }
+
 ##
 # @MigrationParameter:
 #
@@ -641,6 +679,25 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, and on the destination also
+#          complete: On the source, bitmaps on nodes where either the
+#          bitmap or the node is not mapped will be ignored.  In
+#          contrast, on the destination, receiving a bitmap (by alias)
+#          from a node (by alias) when either alias is not mapped will
+#          result in an error.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 { 'enum': 'MigrationParameter',
@@ -655,7 +712,8 @@
            'multifd-channels',
            'xbzrle-cache-size', 'max-postcopy-bandwidth',
            'max-cpu-throttle', 'multifd-compression',
-           'multifd-zlib-level' ,'multifd-zstd-level' ] }
+           'multifd-zlib-level' ,'multifd-zstd-level',
+           'block-bitmap-mapping' ] }
 
 ##
 # @MigrateSetParameters:
@@ -781,6 +839,25 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, and on the destination also
+#          complete: On the source, bitmaps on nodes where either the
+#          bitmap or the node is not mapped will be ignored.  In
+#          contrast, on the destination, receiving a bitmap (by alias)
+#          from a node (by alias) when either alias is not mapped will
+#          result in an error.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 # TODO either fuse back into MigrationParameters, or make
@@ -811,7 +888,8 @@
             '*max-cpu-throttle': 'int',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'int',
-            '*multifd-zstd-level': 'int' } }
+            '*multifd-zstd-level': 'int',
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
 
 ##
 # @migrate-set-parameters:
@@ -957,6 +1035,25 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, and on the destination also
+#          complete: On the source, bitmaps on nodes where either the
+#          bitmap or the node is not mapped will be ignored.  In
+#          contrast, on the destination, receiving a bitmap (by alias)
+#          from a node (by alias) when either alias is not mapped will
+#          result in an error.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 { 'struct': 'MigrationParameters',
@@ -985,7 +1082,8 @@
             '*max-cpu-throttle': 'uint8',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
-            '*multifd-zstd-level': 'uint8' } }
+            '*multifd-zstd-level': 'uint8',
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
 
 ##
 # @query-migrate-parameters:
migration/migration.h +3 -0
index f617960522..038c318b92 100644
@@ -336,6 +336,9 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
 
 void dirty_bitmap_mig_before_vm_start(void);
 void init_dirty_bitmap_incoming_migration(void);
+bool check_dirty_bitmap_mig_alias_map(const BitmapMigrationNodeAliasList *bbm,
+                                      Error **errp);
+
 void migrate_add_address(SocketAddress *address);
 
 int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
migration/block-dirty-bitmap.c +321 -52

548 lines not on this page · raw message

migration/migration.c +30 -0
index 2ed9923227..bcf7c3bf76 100644
@@ -36,6 +36,7 @@
 #include "block/block.h"
 #include "qapi/error.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-migration.h"
 #include "qapi/qapi-visit-sockets.h"
 #include "qapi/qapi-commands-migration.h"
 #include "qapi/qapi-events-migration.h"
@@ -832,6 +833,13 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->has_announce_step = true;
     params->announce_step = s->parameters.announce_step;
 
+    if (s->parameters.has_block_bitmap_mapping) {
+        params->has_block_bitmap_mapping = true;
+        params->block_bitmap_mapping =
+            QAPI_CLONE(BitmapMigrationNodeAliasList,
+                       s->parameters.block_bitmap_mapping);
+    }
+
     return params;
 }
 
@@ -1297,6 +1305,13 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
                    "is invalid, it must be in the range of 1 to 10000 ms");
        return false;
     }
+
+    if (params->has_block_bitmap_mapping &&
+        !check_dirty_bitmap_mig_alias_map(params->block_bitmap_mapping, errp)) {
+        error_prepend(errp, "Invalid mapping given for block-bitmap-mapping: ");
+        return false;
+    }
+
     return true;
 }
 
@@ -1391,6 +1406,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_announce_step) {
         dest->announce_step = params->announce_step;
     }
+
+    if (params->has_block_bitmap_mapping) {
+        dest->has_block_bitmap_mapping = true;
+        dest->block_bitmap_mapping = params->block_bitmap_mapping;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1503,6 +1523,16 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
     if (params->has_announce_step) {
         s->parameters.announce_step = params->announce_step;
     }
+
+    if (params->has_block_bitmap_mapping) {
+        qapi_free_BitmapMigrationNodeAliasList(
+            s->parameters.block_bitmap_mapping);
+
+        s->parameters.has_block_bitmap_mapping = true;
+        s->parameters.block_bitmap_mapping =
+            QAPI_CLONE(BitmapMigrationNodeAliasList,
+                       params->block_bitmap_mapping);
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
monitor/hmp-cmds.c +30 -0
index ae4b6a4246..7711726fd2 100644
@@ -469,6 +469,32 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: '%s'\n",
             MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
             params->tls_authz);
+
+        if (params->has_block_bitmap_mapping) {
+            const BitmapMigrationNodeAliasList *bmnal;
+
+            monitor_printf(mon, "%s:\n",
+                           MigrationParameter_str(
+                               MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING));
+
+            for (bmnal = params->block_bitmap_mapping;
+                 bmnal;
+                 bmnal = bmnal->next)
+            {
+                const BitmapMigrationNodeAlias *bmna = bmnal->value;
+                const BitmapMigrationBitmapAliasList *bmbal;
+
+                monitor_printf(mon, "  '%s' -> '%s'\n",
+                               bmna->node_name, bmna->alias);
+
+                for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+                    const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+
+                    monitor_printf(mon, "    '%s' -> '%s'\n",
+                                   bmba->name, bmba->alias);
+                }
+            }
+        }
     }
 
     qapi_free_MigrationParameters(params);
@@ -1384,6 +1410,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_announce_step = true;
         visit_type_size(v, param, &p->announce_step, &err);
         break;
+    case MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING:
+        error_setg(&err, "The block-bitmap-mapping parameter can only be set "
+                   "through QMP");
+        break;
     default:
         assert(0);
     }
Signature · 1 line
2.26.2
#2 lore raw
Quoted 25 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

---
  qapi/migration.json            | 104 ++++++++-
  migration/migration.h          |   3 +
  migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++-----
  migration/migration.c          |  30 +++
  monitor/hmp-cmds.c             |  30 +++
  5 files changed, 485 insertions(+), 55 deletions(-)
qapi/migration.json +0 -0
index d5000558c6..d7e953cd73 100644
@@ -507,6 +507,44 @@
    'data': [ 'none', 'zlib',
              { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }

qapi/migration.json line 518

Quoted 10 lines
qapi/migration.json +9 -0
 
+##
+# @BitmapMigrationBitmapAlias:
+#
+# @name: The name of the bitmap.
+#
+# @alias: An alias name for migration (for example the bitmap name on
+#         the opposite site).
+#
+# Since: 5.1

5.2, now, but I can touch that up if it is the only problem raised.

qapi/migration.json line 691

Quoted 10 lines
qapi/migration.json +10 -0
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, and on the destination also
+#          complete: On the source, bitmaps on nodes where either the
+#          bitmap or the node is not mapped will be ignored.  In
+#          contrast, on the destination, receiving a bitmap (by alias)
+#          from a node (by alias) when either alias is not mapped will
+#          result in an error.

Grammar is a bit odd and it feels repetitive. How about:

The mapping must not provide more than one alias for a bitmap, nor reuse the same alias across multiple bitmaps in the same node. On the source, an omitted node or bitmap within a node will ignore those bitmaps. In contrast, on the destination, receiving a bitmap (by alias) from a node (by alias) when either alias is not mapped will result in an error.

qapi/migration.json line 699

Quoted 8 lines
qapi/migration.json +8 -0
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)

Looks good.

qapi/migration.json line 845

Quoted 8 lines
qapi/migration.json +4 -0
@@ -781,6 +839,25 @@
  #          will consume more CPU.
  #          Defaults to 1. (Since 5.0)
  #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.

Ah, the joys of duplicated text.

qapi/migration.json typedef struct DirtyBitmapMigState { · line 135

Quoted 9 lines
qapi/migration.json +3 -1
+++ b/migration/block-dirty-bitmap.c
 
@@ -128,7 +131,8 @@ typedef struct DirtyBitmapMigState {
 
  typedef struct DirtyBitmapLoadState {
      uint32_t flags;
-    char node_name[256];
+    char node_alias[256];
+    char bitmap_alias[256];

Do we properly check that alias names will never overflow?

Quoted 11 lines
qapi/migration.json +11 -0
+static GHashTable *construct_alias_map(const BitmapMigrationNodeAliasList *bbm,
+                                       bool name_to_alias,
+                                       Error **errp)
+{
+    GHashTable *alias_map;
+
+    alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
+                                      g_free, free_alias_map_inner_node);
+
+    for (; bbm; bbm = bbm->next) {
+        const BitmapMigrationNodeAlias *bmna = bbm->value;

qapi/migration.json typedef struct DirtyBitmapMigState {

Quoted 10 lines
qapi/migration.json +10 -0
+        const BitmapMigrationBitmapAliasList *bmbal;
+        AliasMapInnerNode *amin;
+        GHashTable *bitmaps_map;
+        const char *node_map_from, *node_map_to;
+
+        if (!id_wellformed(bmna->alias)) {
+            error_setg(errp, "The node alias %s is not well-formed",
+                       bmna->alias);
+            goto fail;
+        }

Maybe id_wellformed should enforce lengths? Otherwise, I'm not seeing a length limit applied during the mapping process.

Modulo that, I think we're ready to go.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#3 lore raw
Quoted 29 lines · Eric Blake
Quoted 28 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

---
  qapi/migration.json            | 104 ++++++++- migration/migration.h          |   3 + migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++----- migration/migration.c          |  30 +++ monitor/hmp-cmds.c             |  30 +++

  5 files changed, 485 insertions(+), 55 deletions(-)
qapi/migration.json +0 -0
index d5000558c6..d7e953cd73 100644
@@ -507,6 +507,44 @@
     'data': [ 'none', 'zlib',
               { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }

  +##

qapi/migration.json +2 -0
+# @BitmapMigrationBitmapAlias:
+#
Quoted 8 lines

qapi/migration.json line 518

Quoted 6 lines
qapi/migration.json +6 -0
+# @name: The name of the bitmap.
+#
+# @alias: An alias name for migration (for example the bitmap name on
+#         the opposite site).
+#
+# Since: 5.1

5.2, now, but I can touch that up if it is the only problem raised.

Yes, somehow I missed this, and amended it to 5.2 elsewhere...

Quoted 13 lines

qapi/migration.json line 691

Quoted 10 lines
qapi/migration.json +10 -0
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, and on the destination also
+#          complete: On the source, bitmaps on nodes where either the
+#          bitmap or the node is not mapped will be ignored.  In
+#          contrast, on the destination, receiving a bitmap (by alias)
+#          from a node (by alias) when either alias is not mapped will
+#          result in an error.

Grammar is a bit odd and it feels repetitive.  How about:

Quoted 2 lines

The mapping must not provide more than one alias for a bitmap, nor reuse the same alias across multiple bitmaps in the same node.

This describes an injective function, as in, one-to-one.

Quoted 4 lines

On the source,
an omitted node or bitmap within a node will ignore those bitmaps. In contrast, on the destination, receiving a bitmap (by alias) from a node (by alias) when either alias is not mapped will result in an error.

This will need a rewrite anyway. Because of Vladimir’s patches you merged in rc2, it looks like it makes more sense now to ignore unknown aliases on the destination. So I’ll have to think up something new anyway.

Quoted 25 lines

qapi/migration.json line 699

Quoted 8 lines
qapi/migration.json +8 -0
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)

Looks good.

qapi/migration.json line 845

Quoted 8 lines
qapi/migration.json +4 -0
@@ -781,6 +839,25 @@
   #          will consume more CPU.
   #          Defaults to 1. (Since 5.0)
   #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.

Ah, the joys of duplicated text.

Quoted 1 line
qapi/migration.json +1 -0
+++ b/migration/block-dirty-bitmap.c
Quoted 8 lines

qapi/migration.json typedef struct DirtyBitmapMigState { · line 134

Quoted 6 lines
qapi/migration.json +2 -1
@@ -128,7 +131,8 @@ typedef struct DirtyBitmapMigState {
     typedef struct DirtyBitmapLoadState {
       uint32_t flags;
-    char node_name[256];
+    char node_alias[256];
+    char bitmap_alias[256];

Do we properly check that alias names will never overflow?

Hm, well. There are assertions guarding against that, but they’re assertions.

That’s an existing problem, actually. If you give a bitmap a name longer than 255 bytes, you run into the same failed assertion.

Quoted 18 lines
Quoted 18 lines
qapi/migration.json +8 -0
+static GHashTable *construct_alias_map(const
 BitmapMigrationNodeAliasList *bbm,
+                                       bool name_to_alias,
+                                       Error **errp)
+{
+    GHashTable *alias_map;
+
+    alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
+                                      g_free,

free_alias_map_inner_node);

qapi/migration.json +8 -0
+
+    for (; bbm; bbm = bbm->next) {
+        const BitmapMigrationNodeAlias *bmna = bbm->value;
+        const BitmapMigrationBitmapAliasList *bmbal;
+        AliasMapInnerNode *amin;
+        GHashTable *bitmaps_map;
+        const char *node_map_from, *node_map_to;
+
Quoted 8 lines
Quoted 5 lines

+        if (!id_wellformed(bmna->alias)) {
+            error_setg(errp, "The node alias %s is not well-formed", +                       bmna->alias);
+            goto fail;
+        }

Maybe id_wellformed should enforce lengths?  Otherwise, I'm not seeing a length limit applied during the mapping process.

The limit is applied by qemu_put_counted_string(), but it’s enforced via assertion, so, yeah. I’ll fix it.

(That’s going to be the easy part. The harder part is fixing the pre-existing issue, because we only see bitmap names once migration starts. Thanks to the new permissive error policy (as in, no configuration problem should ever produce errors during bitmap migration), we then need to drop such bitmaps.)

Quoted 1 line

Modulo that, I think we're ready to go.

Unfortunately, no. As Vladimir has pointed out, this will require a hefty rebase now.

Max

author
#4 lore raw

PATCH v3 2/3 iotests.py: Add wait_for_runstate()

In reply to Max Reitz
---
 tests/qemu-iotests/iotests.py | 5 +++++
 1 file changed, 5 insertions(+)
tests/qemu-iotests/iotests.py +5 -0
index 3590ed78a0..20645a6e7d 100644
@@ -28,6 +28,7 @@ import signal
 import struct
 import subprocess
 import sys
+import time
 from typing import (Any, Callable, Dict, Iterable,
                     List, Optional, Sequence, Tuple, TypeVar)
 import unittest
@@ -803,6 +804,10 @@ class VM(qtest.QEMUQtestMachine):
                    'Found node %s under %s (but expected %s)' % \
                    (node['name'], path, expected_node)
 
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] != runstate:
+            time.sleep(0.2)
+
 index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
 
 class QMPTestCase(unittest.TestCase):
Signature · 1 line
2.26.2
author
#5 lore raw

PATCH v3 3/3 iotests: Test node/bitmap aliases during migration

In reply to Max Reitz
---
 tests/qemu-iotests/300     | 515 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/300.out |   5 +
 tests/qemu-iotests/group   |   1 +
 3 files changed, 521 insertions(+)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
tests/qemu-iotests/300 new file +515 -0
new file mode 100755
index 0000000000..6d684e882a
@@ -0,0 +1,515 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# Tests for dirty bitmaps migration with node aliases
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import random
+import re
+import time
+from typing import Dict, List, Optional, Union
+import iotests
+import qemu
+
+BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
+
+assert iotests.sock_dir is not None
+mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+    src_node_name: str = ''
+    dst_node_name: str = ''
+    src_bmap_name: str = ''
+    dst_bmap_name: str = ''
+
+    def setUp(self) -> None:
+        self.vm_a = iotests.VM(path_suffix='-a')
+        self.vm_a.add_blockdev(f'node-name={self.src_node_name},' \
+                               'driver=null-co')
+        self.vm_a.launch()
+
+        self.vm_b = iotests.VM(path_suffix='-b')
+        self.vm_b.add_blockdev(f'node-name={self.dst_node_name},' \
+                               'driver=null-co')
+        self.vm_b.add_incoming(f'unix:{mig_sock}')
+        self.vm_b.launch()
+
+        result = self.vm_a.qmp('block-dirty-bitmap-add',
+                               node=self.src_node_name,
+                               name=self.src_bmap_name)
+        self.assert_qmp(result, 'return', {})
+
+        # Dirty some random megabytes
+        for _ in range(9):
+            mb_ofs = random.randrange(1024)
+            self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')
+
+        result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
+                               node=self.src_node_name,
+                               name=self.src_bmap_name)
+        self.bitmap_hash_reference = result['return']['sha256']
+
+        caps = [{'capability': name, 'state': True} \
+                for name in ('dirty-bitmaps', 'events')]
+
+        for vm in (self.vm_a, self.vm_b):
+            result = vm.qmp('migrate-set-capabilities', capabilities=caps)
+            self.assert_qmp(result, 'return', {})
+
+    def tearDown(self) -> None:
+        self.vm_a.shutdown()
+        self.vm_b.shutdown()
+        try:
+            os.remove(mig_sock)
+        except OSError:
+            pass
+
+    def check_bitmap(self, bitmap_name_valid: bool) -> None:
+        result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
+                               node=self.dst_node_name,
+                               name=self.dst_bmap_name)
+        if bitmap_name_valid:
+            self.assert_qmp(result, 'return/sha256',
+                            self.bitmap_hash_reference)
+        else:
+            self.assert_qmp(result, 'error/desc',
+                            f"Dirty bitmap '{self.dst_bmap_name}' not found")
+
+    def migrate(self, migration_success: bool = True,
+                bitmap_name_valid: bool = True) -> None:
+        result = self.vm_a.qmp('migrate', uri=f'unix:{mig_sock}')
+        self.assert_qmp(result, 'return', {})
+
+        status = None
+        while status not in ('completed', 'failed'):
+            status = self.vm_a.event_wait('MIGRATION')['data']['status']
+
+        self.assertEqual(status == 'completed', migration_success)
+        if status == 'failed':
+            # Wait until the migration has been cleaned up
+            # (Otherwise, bdrv_close_all() will abort because the
+            # dirty bitmap migration code still holds a reference to
+            # the BDS)
+            # (Unfortunately, there does not appear to be a nicer way
+            # of waiting until a failed migration has been cleaned up)
+            timeout_msg = 'Timeout waiting for migration to be cleaned up'
+            with iotests.Timeout(30, timeout_msg):
+                while os.path.exists(mig_sock):
+                    time.sleep(0.1)
+
+                # Dropping src_node_name will only work once the
+                # bitmap migration code has released it
+                while 'error' in self.vm_a.qmp('blockdev-del',
+                                               node_name=self.src_node_name):
+                    time.sleep(0.1)
+
+            return
+
+        self.vm_a.wait_for_runstate('postmigrate')
+        self.vm_b.wait_migration('running')
+
+        self.check_bitmap(bitmap_name_valid)
+
+    @staticmethod
+    def mapping(node_name: str, node_alias: str,
+                bitmap_name: str, bitmap_alias: str) \
+        -> BlockBitmapMapping:
+        return [{
+            'node-name': node_name,
+            'alias': node_alias,
+            'bitmaps': [{
+                'name': bitmap_name,
+                'alias': bitmap_alias
+            }]
+        }]
+
+    def set_mapping(self, vm: iotests.VM, mapping: BlockBitmapMapping,
+                    error: Optional[str] = None) \
+        -> None:
+        """
+        Invoke migrate-set-parameters on @vm to set the given @mapping.
+        Check for success if @error is None, or verify the error message
+        if it is not.
+        On success, verify that "info migrate_parameters" on HMP returns
+        our mapping.  (Just to check its formatting code.)
+        """
+        result = vm.qmp('migrate-set-parameters',
+                        block_bitmap_mapping=mapping)
+
+        if error is None:
+            self.assert_qmp(result, 'return', {})
+
+            result = vm.qmp('human-monitor-command',
+                            command_line='info migrate_parameters')
+
+            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
+                                    result['return'], flags=re.MULTILINE)
+
+            self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
+                             self.to_hmp_mapping(mapping))
+        else:
+            self.assert_qmp(result, 'error/desc', error)
+
+    @staticmethod
+    def to_hmp_mapping(mapping: BlockBitmapMapping) -> str:
+        result = 'block-bitmap-mapping:\n'
+
+        for node in mapping:
+            result += f"  '{node['node-name']}' -> '{node['alias']}'\n"
+
+            assert isinstance(node['bitmaps'], list)
+            for bitmap in node['bitmaps']:
+                result += f"    '{bitmap['name']}' -> '{bitmap['alias']}'\n"
+
+        return result
+
+
+class TestAliasMigration(TestDirtyBitmapMigration):
+    src_node_name = 'node0'
+    dst_node_name = 'node0'
+    src_bmap_name = 'bmap0'
+    dst_bmap_name = 'bmap0'
+
+    def test_migration_without_alias(self) -> None:
+        self.migrate(self.src_node_name == self.dst_node_name,
+                     self.src_bmap_name == self.dst_bmap_name)
+
+        # Expect abnormal shutdown of the destination VM on migration failure
+        if self.src_node_name != self.dst_node_name:
+            try:
+                self.vm_b.shutdown()
+            except qemu.machine.AbnormalShutdown:
+                pass
+
+    def test_alias_on_src_migration(self) -> None:
+        mapping = self.mapping(self.src_node_name, self.dst_node_name,
+                               self.src_bmap_name, self.dst_bmap_name)
+
+        self.set_mapping(self.vm_a, mapping)
+        self.migrate()
+
+    def test_alias_on_dst_migration(self) -> None:
+        mapping = self.mapping(self.dst_node_name, self.src_node_name,
+                               self.dst_bmap_name, self.src_bmap_name)
+
+        self.set_mapping(self.vm_b, mapping)
+        self.migrate()
+
+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias',
+                               self.src_bmap_name, 'bmap-alias')
+
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',
+                               self.dst_bmap_name, 'bmap-alias')
+
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+
+
+class TestNodeAliasMigration(TestAliasMigration):
+    src_node_name = 'node-src'
+    dst_node_name = 'node-dst'
+
+
+class TestBitmapAliasMigration(TestAliasMigration):
+    src_bmap_name = 'bmap-src'
+    dst_bmap_name = 'bmap-dst'
+
+
+class TestFullAliasMigration(TestAliasMigration):
+    src_node_name = 'node-src'
+    dst_node_name = 'node-dst'
+    src_bmap_name = 'bmap-src'
+    dst_bmap_name = 'bmap-dst'
+
+
+class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
+    src_node_name = 'node0'
+    dst_node_name = 'node0'
+    src_bmap_name = 'bmap0'
+    dst_bmap_name = 'bmap0'
+
+    """
+    Note that mapping nodes or bitmaps that do not exist is not an error.
+    """
+
+    def test_non_injective_node_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [
+            {
+                'node-name': 'node0',
+                'alias': 'common-alias',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            },
+            {
+                'node-name': 'node1',
+                'alias': 'common-alias',
+                'bitmaps': [{
+                    'name': 'bmap1',
+                    'alias': 'bmap-alias1'
+                }]
+            }
+        ]
+
+        self.set_mapping(self.vm_a, mapping,
+                         'Invalid mapping given for block-bitmap-mapping: ' \
+                         'The node alias common-alias is used twice')
+
+    def test_non_injective_bitmap_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': 'node0',
+            'alias': 'node-alias0',
+            'bitmaps': [
+                {
+                    'name': 'bmap0',
+                    'alias': 'common-alias'
+                },
+                {
+                    'name': 'bmap1',
+                    'alias': 'common-alias'
+                }
+            ]
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         'Invalid mapping given for block-bitmap-mapping: ' \
+                         'The bitmap alias node-alias0/common-alias is used ' \
+                         'twice')
+
+    def test_ambiguous_node_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [
+            {
+                'node-name': 'node0',
+                'alias': 'node-alias0',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            },
+            {
+                'node-name': 'node0',
+                'alias': 'node-alias1',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            }
+        ]
+
+        self.set_mapping(self.vm_a, mapping,
+                         'Invalid mapping given for block-bitmap-mapping: ' \
+                         'The node name node0 is mapped twice')
+
+    def test_ambiguous_bitmap_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': 'node0',
+            'alias': 'node-alias0',
+            'bitmaps': [
+                {
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                },
+                {
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias1'
+                }
+            ]
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         'Invalid mapping given for block-bitmap-mapping: ' \
+                         'The bitmap node0/bmap0 is mapped twice')
+
+    def test_migratee_node_is_not_mapped_on_src(self) -> None:
+        self.set_mapping(self.vm_a, [])
+        # Should just ignore all bitmaps on unmapped nodes
+        self.migrate(True, False)
+
+    def test_migratee_node_is_not_mapped_on_dst(self) -> None:
+        self.set_mapping(self.vm_b, [])
+        self.migrate(False, False)
+
+        # Expect abnormal shutdown of the destination VM on migration failure
+        try:
+            self.vm_b.shutdown()
+        except qemu.machine.AbnormalShutdown:
+            pass
+
+        self.assertIn(f"Unknown node alias '{self.src_node_name}'",
+                      self.vm_b.get_log())
+
+    def test_migratee_bitmap_is_not_mapped_on_src(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.src_node_name,
+            'alias': self.dst_node_name,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_a, mapping)
+        # Should just ignore all unmapped bitmaps
+        self.migrate(True, False)
+
+    def test_migratee_bitmap_is_not_mapped_on_dst(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.dst_node_name,
+            'alias': self.src_node_name,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_b, mapping)
+        self.migrate(False, False)
+
+        # Expect abnormal shutdown of the destination VM on migration failure
+        try:
+            self.vm_b.shutdown()
+        except qemu.machine.AbnormalShutdown:
+            pass
+
+        self.assertIn(f"Unknown bitmap alias '{self.src_bmap_name}' on node " \
+                      f"'{self.dst_node_name}' (alias '{self.src_node_name}')",
+                      self.vm_b.get_log())
+
+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps
+        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination
+        self.set_mapping(self.vm_b, [])
+
+        # The fact that there is a mapping on B without any bitmaps
+        # being received should be fine, not fatal
+        self.migrate(True, False)
+
+    def test_non_wellformed_node_alias(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.src_bmap_name,
+            'alias': '123-foo',
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         'Invalid mapping given for block-bitmap-mapping: ' \
+                         'The node alias 123-foo is not well-formed')
+
+
+class TestCrossAliasMigration(TestDirtyBitmapMigration):
+    """
+    Swap aliases, both to see that qemu does not get confused, and
+    that we can migrate multiple things at once.
+
+    So we migrate this:
+      node-a.bmap-a -> node-b.bmap-b
+      node-a.bmap-b -> node-b.bmap-a
+      node-b.bmap-a -> node-a.bmap-b
+      node-b.bmap-b -> node-a.bmap-a
+    """
+
+    src_node_name = 'node-a'
+    dst_node_name = 'node-b'
+    src_bmap_name = 'bmap-a'
+    dst_bmap_name = 'bmap-b'
+
+    def setUp(self) -> None:
+        TestDirtyBitmapMigration.setUp(self)
+
+        # Now create another block device and let both have two bitmaps each
+        result = self.vm_a.qmp('blockdev-add',
+                               node_name='node-b', driver='null-co')
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm_b.qmp('blockdev-add',
+                               node_name='node-a', driver='null-co')
+        self.assert_qmp(result, 'return', {})
+
+        bmaps_to_add = (('node-a', 'bmap-b'),
+                        ('node-b', 'bmap-a'),
+                        ('node-b', 'bmap-b'))
+
+        for (node, bmap) in bmaps_to_add:
+            result = self.vm_a.qmp('block-dirty-bitmap-add',
+                                   node=node, name=bmap)
+            self.assert_qmp(result, 'return', {})
+
+    @staticmethod
+    def cross_mapping() -> BlockBitmapMapping:
+        return [
+            {
+                'node-name': 'node-a',
+                'alias': 'node-b',
+                'bitmaps': [
+                    {
+                        'name': 'bmap-a',
+                        'alias': 'bmap-b'
+                    },
+                    {
+                        'name': 'bmap-b',
+                        'alias': 'bmap-a'
+                    }
+                ]
+            },
+            {
+                'node-name': 'node-b',
+                'alias': 'node-a',
+                'bitmaps': [
+                    {
+                        'name': 'bmap-b',
+                        'alias': 'bmap-a'
+                    },
+                    {
+                        'name': 'bmap-a',
+                        'alias': 'bmap-b'
+                    }
+                ]
+            }
+        ]
+
+    def verify_dest_has_all_bitmaps(self) -> None:
+        bitmaps = self.vm_b.query_bitmaps()
+
+        # Extract and sort bitmap names
+        for node in bitmaps:
+            bitmaps[node] = sorted((bmap['name'] for bmap in bitmaps[node]))
+
+        self.assertEqual(bitmaps,
+                         {'node-a': ['bmap-a', 'bmap-b'],
+                          'node-b': ['bmap-a', 'bmap-b']})
+
+    def test_alias_on_src(self) -> None:
+        self.set_mapping(self.vm_a, self.cross_mapping())
+
+        # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+        # that is enough
+        self.migrate()
+
+        self.verify_dest_has_all_bitmaps()
+
+    def test_alias_on_dst(self) -> None:
+        self.set_mapping(self.vm_b, self.cross_mapping())
+
+        # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+        # that is enough
+        self.migrate()
+
+        self.verify_dest_has_all_bitmaps()
+
+
+if __name__ == '__main__':
+    iotests.main(supported_protocols=['file'])
tests/qemu-iotests/300.out new file +5 -0
new file mode 100644
index 0000000000..af8dac10f9
@@ -0,0 +1,5 @@
+............................
+----------------------------------------------------------------------
+Ran 28 tests
+
+OK
tests/qemu-iotests/group +1 -0
index e8d8747887..0f3907e7ad 100644
@@ -308,4 +308,5 @@
 296 rw
 297 meta
 299 auto quick
+300 migration
 301 backing quick
Signature · 1 line
2.26.2
author
#6 lore raw
In reply to Max Reitz

Ping – seems like everyone found v2 more or less acceptable bar the failing assertion in patch 1, and some aspects of the test. How about v3, are there any objections?

Quoted 78 lines · Max Reitz

RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html

Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3

Hi,

This new migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

v3:
- Patch 1:
- Add notes on the fact that the destination won’t ever know about any bitmaps that aren’t sent, so you can freely drop what you want, and you’re completely free in renaming bitmaps and putting them on “other” nodes (whatever “other” node means in the context of migration, because that’s kind of one of the problems this series is trying to solve: The fact that you can’t trivially match nodes between source and destination)
- Fix an assertion

- Patch 2: s/pass/time.sleep(0.2)/

- Patch 3:
- Add copyright line
- Use format string instead of %
- s/pass/time.sleep(0.1)/
- s/wait_for_runstate/wait_migration/ on the destination to wait for the migration to actually complete
- Replace the “info migrate_parameters” parsing code by a multiline regex
- Test what happens when the destination has a mapping that isn’t used because there are not bitmaps to be transferred (which breaks the assertion in patch 1 as it was in v2)
- Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the destination instead of the source

git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter' 002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()' 003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'

Max Reitz (3):
migration: Add block-bitmap-mapping parameter iotests.py: Add wait_for_runstate()
iotests: Test node/bitmap aliases during migration

 qapi/migration.json            | 104 ++++++-
 migration/migration.h          |   3 +
 migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
 migration/migration.c          |  30 ++
 monitor/hmp-cmds.c             |  30 ++
 tests/qemu-iotests/300         | 515 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/300.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |   5 +
 9 files changed, 1011 insertions(+), 55 deletions(-)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
#7 lore raw

Now this doesn't apply, as code changed a lot after my series
"[PATCH v4 for-5.1 00/21] Fix error handling during bitmap postcopy"
merged first.

I feel my responsibility for the mess with these series, so if you want I can try to rebase and post v4 of this one myself :)

Quoted 85 lines · Max Reitz

Ping – seems like everyone found v2 more or less acceptable bar the failing assertion in patch 1, and some aspects of the test. How about v3, are there any objections?

Quoted 78 lines · Max Reitz

RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html

Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3

Hi,

This new migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

v3:
- Patch 1:
- Add notes on the fact that the destination won’t ever know about any bitmaps that aren’t sent, so you can freely drop what you want, and you’re completely free in renaming bitmaps and putting them on “other” nodes (whatever “other” node means in the context of migration, because that’s kind of one of the problems this series is trying to solve: The fact that you can’t trivially match nodes between source and destination)
- Fix an assertion

- Patch 2: s/pass/time.sleep(0.2)/

- Patch 3:
- Add copyright line
- Use format string instead of %
- s/pass/time.sleep(0.1)/
- s/wait_for_runstate/wait_migration/ on the destination to wait for the migration to actually complete
- Replace the “info migrate_parameters” parsing code by a multiline regex
- Test what happens when the destination has a mapping that isn’t used because there are not bitmaps to be transferred (which breaks the assertion in patch 1 as it was in v2)
- Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the destination instead of the source

git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter' 002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()' 003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'

Max Reitz (3):
migration: Add block-bitmap-mapping parameter iotests.py: Add wait_for_runstate()
iotests: Test node/bitmap aliases during migration

  qapi/migration.json            | 104 ++++++-
  migration/migration.h          |   3 +
  migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
  migration/migration.c          |  30 ++
  monitor/hmp-cmds.c             |  30 ++
  tests/qemu-iotests/300         | 515 +++++++++++++++++++++++++++++++++
  tests/qemu-iotests/300.out     |   5 +
  tests/qemu-iotests/group       |   1 +
  tests/qemu-iotests/iotests.py  |   5 +
  9 files changed, 1011 insertions(+), 55 deletions(-)

create mode 100755 tests/qemu-iotests/300
create mode 100644 tests/qemu-iotests/300.out

Signature · 2 lines
Best regards,
Vladimir
author
#8 lore raw
Quoted 3 lines · Vladimir Sementsov-Ogievskiy

Now this doesn't apply, as code changed a lot after my series "[PATCH v4 for-5.1 00/21] Fix error handling during bitmap postcopy" merged first.

Yeah, well. Yeah.

Quoted 2 lines

I feel my responsibility for the mess with these series, so if you want I can try to rebase and post v4 of this one myself :)

I feel like rebasing is simpler than reviewing, so I’ll be egoistic and choose the rebaser’s side.

Max