Skip to contentSkip to conversation

migration: Add block-bitmap-mapping parameter

PATCH Superseded Max Reitz 4 participants 23 messages Aug 18, 2020 – Aug 21, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v4 cover migration: Add block-bitmap-mapping parameter

RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html
v3: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01385.html

Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v4
Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v4

Hi,

This new migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

v4:
- Patch 1:
- Rebased
- %s/5.1/5.2/
- The destination side will now ignore unmapped aliases, too (though they are reported to stderr)
- Check that the aliases given will fit in the migration stream (i.e. are shorter than 256 bytes)
- On that note, fix a pre-existing bug where a bitmap with a name longer than 255 bytes would make qemu_put_counted_string() fail an assertion

- Patch 2:
- import time (apparently, I forgot that in v3...?)

- Patch 3:
- Added; I need this now

- Patch 4:
- Migration rarely ever really fails now (just one case, where a non-aliased bitmap name is too long, which can only be detected when beginning migration)

- Thus, we have to read most error messages from the VM log now, for which I’ve added a new function (verify_dest_error)

- Added tests for overly long bitmap names and node/bitmap aliases

git-backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/4:[0167] [FC] 'migration: Add block-bitmap-mapping parameter' 002/4:[0001] [FC] 'iotests.py: Add wait_for_runstate()' 003/4:[down] 'iotests.py: Let wait_migration() return on failure' 004/4:[0232] [FC] 'iotests: Test node/bitmap aliases during migration'

Max Reitz (4):
migration: Add block-bitmap-mapping parameter iotests.py: Add wait_for_runstate() iotests.py: Let wait_migration() return on failure
iotests: Test node/bitmap aliases during migration

 qapi/migration.json            | 101 +++++-
 migration/migration.h          |   3 +
 migration/block-dirty-bitmap.c | 409 +++++++++++++++++++---
 migration/migration.c          |  30 ++
 monitor/hmp-cmds.c             |  30 ++
 tests/qemu-iotests/300         | 595 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/300.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |  22 +-
 9 files changed, 1134 insertions(+), 62 deletions(-)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v4 1/4 migration: Add block-bitmap-mapping parameter

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

While touching this code, fix a bug where bitmap names longer than 255 bytes would fail an assertion in qemu_put_counted_string().

---
 qapi/migration.json            | 101 +++++++-
 migration/migration.h          |   3 +
 migration/block-dirty-bitmap.c | 409 ++++++++++++++++++++++++++++-----
 migration/migration.c          |  30 +++
 monitor/hmp-cmds.c             |  30 +++
 5 files changed, 517 insertions(+), 56 deletions(-)
qapi/migration.json +98 -3
index ea53b23dca..0c4ae102b1 100644
@@ -508,6 +508,44 @@
   'data': [ 'none', 'zlib',
             { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
 
+##
+# @BitmapMigrationBitmapAlias:
+#
+# @name: The name of the bitmap.
+#
+# @alias: An alias name for migration (for example the bitmap name on
+#         the opposite site).
+#
+# Since: 5.2
+##
+{ 'struct': 'BitmapMigrationBitmapAlias',
+  'data': {
+      'name': 'str',
+      'alias': 'str'
+  } }
+
+##
+# @BitmapMigrationNodeAlias:
+#
+# Maps a block node name and the bitmaps it has to aliases for dirty
+# bitmap migration.
+#
+# @node-name: A block node name.
+#
+# @alias: An alias block node name for migration (for example the
+#         node name on the opposite site).
+#
+# @bitmaps: Mappings for the bitmaps on this node.
+#
+# Since: 5.2
+##
+{ 'struct': 'BitmapMigrationNodeAlias',
+  'data': {
+      'node-name': 'str',
+      'alias': 'str',
+      'bitmaps': [ 'BitmapMigrationBitmapAlias' ]
+  } }
+
 ##
 # @MigrationParameter:
 #
@@ -642,6 +680,24 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, but not necessarily
+#          complete: On the source, unmapped bitmaps and all bitmaps
+#          on unmapped nodes will be ignored.  On the destination,
+#          all unmapped aliases in the incoming migration stream will
+#          be reported, but they will not result in failure.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 { 'enum': 'MigrationParameter',
@@ -656,7 +712,8 @@
            'multifd-channels',
            'xbzrle-cache-size', 'max-postcopy-bandwidth',
            'max-cpu-throttle', 'multifd-compression',
-           'multifd-zlib-level' ,'multifd-zstd-level' ] }
+           'multifd-zlib-level' ,'multifd-zstd-level',
+           'block-bitmap-mapping' ] }
 
 ##
 # @MigrateSetParameters:
@@ -782,6 +839,24 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, but not necessarily
+#          complete: On the source, unmapped bitmaps and all bitmaps
+#          on unmapped nodes will be ignored.  On the destination,
+#          all unmapped aliases in the incoming migration stream will
+#          be reported, but they will not result in failure.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 # TODO either fuse back into MigrationParameters, or make
@@ -812,7 +887,8 @@
             '*max-cpu-throttle': 'int',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'int',
-            '*multifd-zstd-level': 'int' } }
+            '*multifd-zstd-level': 'int',
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
 
 ##
 # @migrate-set-parameters:
@@ -958,6 +1034,24 @@
 #          will consume more CPU.
 #          Defaults to 1. (Since 5.0)
 #
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, but not necessarily
+#          complete: On the source, unmapped bitmaps and all bitmaps
+#          on unmapped nodes will be ignored.  On the destination,
+#          all unmapped aliases in the incoming migration stream will
+#          be reported, but they will not result in failure.
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
 # Since: 2.4
 ##
 { 'struct': 'MigrationParameters',
@@ -986,7 +1080,8 @@
             '*max-cpu-throttle': 'uint8',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
-            '*multifd-zstd-level': 'uint8' } }
+            '*multifd-zstd-level': 'uint8',
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
 
 ##
 # @query-migrate-parameters:
migration/migration.h +3 -0
index 6c6a931d0d..2ed55c4aef 100644
@@ -337,6 +337,9 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
 void dirty_bitmap_mig_before_vm_start(void);
 void dirty_bitmap_mig_cancel_outgoing(void);
 void dirty_bitmap_mig_cancel_incoming(void);
+bool check_dirty_bitmap_mig_alias_map(const BitmapMigrationNodeAliasList *bbm,
+                                      Error **errp);
+
 void migrate_add_address(SocketAddress *address);
 
 int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
migration/block-dirty-bitmap.c +356 -53

591 lines not on this page · raw message

migration/migration.c +30 -0
index 8fe36339db..dbd4afa1e8 100644
@@ -36,6 +36,7 @@
 #include "block/block.h"
 #include "qapi/error.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-migration.h"
 #include "qapi/qapi-visit-sockets.h"
 #include "qapi/qapi-commands-migration.h"
 #include "qapi/qapi-events-migration.h"
@@ -843,6 +844,13 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->has_announce_step = true;
     params->announce_step = s->parameters.announce_step;
 
+    if (s->parameters.has_block_bitmap_mapping) {
+        params->has_block_bitmap_mapping = true;
+        params->block_bitmap_mapping =
+            QAPI_CLONE(BitmapMigrationNodeAliasList,
+                       s->parameters.block_bitmap_mapping);
+    }
+
     return params;
 }
 
@@ -1308,6 +1316,13 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
                    "is invalid, it must be in the range of 1 to 10000 ms");
        return false;
     }
+
+    if (params->has_block_bitmap_mapping &&
+        !check_dirty_bitmap_mig_alias_map(params->block_bitmap_mapping, errp)) {
+        error_prepend(errp, "Invalid mapping given for block-bitmap-mapping: ");
+        return false;
+    }
+
     return true;
 }
 
@@ -1402,6 +1417,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_announce_step) {
         dest->announce_step = params->announce_step;
     }
+
+    if (params->has_block_bitmap_mapping) {
+        dest->has_block_bitmap_mapping = true;
+        dest->block_bitmap_mapping = params->block_bitmap_mapping;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1514,6 +1534,16 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
     if (params->has_announce_step) {
         s->parameters.announce_step = params->announce_step;
     }
+
+    if (params->has_block_bitmap_mapping) {
+        qapi_free_BitmapMigrationNodeAliasList(
+            s->parameters.block_bitmap_mapping);
+
+        s->parameters.has_block_bitmap_mapping = true;
+        s->parameters.block_bitmap_mapping =
+            QAPI_CLONE(BitmapMigrationNodeAliasList,
+                       params->block_bitmap_mapping);
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
monitor/hmp-cmds.c +30 -0
index ae4b6a4246..7711726fd2 100644
@@ -469,6 +469,32 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: '%s'\n",
             MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
             params->tls_authz);
+
+        if (params->has_block_bitmap_mapping) {
+            const BitmapMigrationNodeAliasList *bmnal;
+
+            monitor_printf(mon, "%s:\n",
+                           MigrationParameter_str(
+                               MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING));
+
+            for (bmnal = params->block_bitmap_mapping;
+                 bmnal;
+                 bmnal = bmnal->next)
+            {
+                const BitmapMigrationNodeAlias *bmna = bmnal->value;
+                const BitmapMigrationBitmapAliasList *bmbal;
+
+                monitor_printf(mon, "  '%s' -> '%s'\n",
+                               bmna->node_name, bmna->alias);
+
+                for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+                    const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+
+                    monitor_printf(mon, "    '%s' -> '%s'\n",
+                                   bmba->name, bmba->alias);
+                }
+            }
+        }
     }
 
     qapi_free_MigrationParameters(params);
@@ -1384,6 +1410,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_announce_step = true;
         visit_type_size(v, param, &p->announce_step, &err);
         break;
+    case MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING:
+        error_setg(&err, "The block-bitmap-mapping parameter can only be set "
+                   "through QMP");
+        break;
     default:
         assert(0);
     }
Signature · 1 line
2.26.2
Reviewed-by on v4 1/4
#2 lore raw
Quoted 26 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

While touching this code, fix a bug where bitmap names longer than 255 bytes would fail an assertion in qemu_put_counted_string().

---

qapi/migration.json +11 -0
+##
+# @BitmapMigrationNodeAlias:
+#
+# Maps a block node name and the bitmaps it has to aliases for dirty
+# bitmap migration.
+#
+# @node-name: A block node name.
+#
+# @alias: An alias block node name for migration (for example the
+#         node name on the opposite site).
+#

qapi/migration.json line 547

Quoted 10 lines
qapi/migration.json +10 -0
+# @bitmaps: Mappings for the bitmaps on this node.
+#
+# Since: 5.2
+##
+{ 'struct': 'BitmapMigrationNodeAlias',
+  'data': {
+      'node-name': 'str',
+      'alias': 'str',
+      'bitmaps': [ 'BitmapMigrationBitmapAlias' ]
+  } }

Possible change: should 'alias' be optional (if absent, it defaults to 'node-name')? But that can be done on top, if we like it.

migration/block-dirty-bitmap.c typedef struct DBMState { · line 217

Quoted 7 lines
migration/block-dirty-bitmap.c +7 -0
+static GHashTable *construct_alias_map(const BitmapMigrationNodeAliasList *bbm,
+                                       bool name_to_alias,
+                                       Error **errp)
+{
+    GHashTable *alias_map;
+    size_t max_node_name_len =
+        sizeof(((BlockDriverState *)NULL)->node_name) - 1;

Looks a bit nicer as = sizeof_field(BlockDriverState, node_name) - 1.

Quoted 8 lines
migration/block-dirty-bitmap.c +8 -0
+
+    alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
+                                      g_free, free_alias_map_inner_node);
+
+    for (; bbm; bbm = bbm->next) {
+        const BitmapMigrationNodeAlias *bmna = bbm->value;
+        const BitmapMigrationBitmapAliasList *bmbal;
+        AliasMapInnerNode *amin;

migration/block-dirty-bitmap.c typedef struct DBMState { · line 235

Quoted 10 lines
migration/block-dirty-bitmap.c +10 -0
+        GHashTable *bitmaps_map;
+        const char *node_map_from, *node_map_to;
+
+        if (!id_wellformed(bmna->alias)) {
+            error_setg(errp, "The node alias '%s' is not well-formed",
+                       bmna->alias);
+            goto fail;
+        }
+
+        if (strlen(bmna->alias) > 255) {

Magic number. UINT8_MAX seems better (since the limit really is due to our migration format limiting to one byte).

...

migration/block-dirty-bitmap.c typedef struct DBMState { · line 282

Quoted 7 lines
migration/block-dirty-bitmap.c +7 -0
+        g_hash_table_insert(alias_map, g_strdup(node_map_from), amin);
+
+        for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+            const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+            const char *bmap_map_from, *bmap_map_to;
+
+            if (strlen(bmba->alias) > 255) {

and again

migration/block-dirty-bitmap.c typedef struct DBMState { · line 287

Quoted 5 lines
migration/block-dirty-bitmap.c +5 -0
+                error_setg(errp,
+                           "The bitmap alias '%s' is longer than 255 bytes",
+                           bmba->alias);
+                goto fail;
+            }

Thanks for adding in the length checking since last revision!

Quoted 6 lines
migration/block-dirty-bitmap.c +2 -0
@@ -326,12 +538,29 @@ static int add_bitmaps_to_list(DBMSaveState *s, BlockDriverState *bs,
              return -1;
          }
 
+        if (bitmap_aliases) {
+            bitmap_alias = g_hash_table_lookup(bitmap_aliases, bitmap_name);

migration/block-dirty-bitmap.c static int add_bitmaps_to_list(DBMSaveState *s, BlockDriverState *bs, · line 552

Quoted 10 lines
migration/block-dirty-bitmap.c +10 -0
+            if (!bitmap_alias) {
+                /* Skip bitmaps with no alias */
+                continue;
+            }
+        } else {
+            if (strlen(bitmap_name) > 255) {
+                error_report("Cannot migrate bitmap '%s' on node '%s': "
+                             "Name is longer than 255 bytes",
+                             bitmap_name, bs_name);
+                return -1;

Another one.

I'm happy to make those touchups, and put this on my bitmaps queue for a pull request as soon as Paolo's meson stuff stabilizes.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#3 lore raw
Quoted 32 lines · Eric Blake
Quoted 31 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

While touching this code, fix a bug where bitmap names longer than 255 bytes would fail an assertion in qemu_put_counted_string().

---

qapi/migration.json +16 -0
+##
+# @BitmapMigrationNodeAlias:
+#
+# Maps a block node name and the bitmaps it has to aliases for dirty
+# bitmap migration.
+#
+# @node-name: A block node name.
+#
+# @alias: An alias block node name for migration (for example the
+#         node name on the opposite site).
+#
+# @bitmaps: Mappings for the bitmaps on this node.
+#
+# Since: 5.2
+##
+{ 'struct': 'BitmapMigrationNodeAlias',
Quoted 8 lines
Quoted 5 lines

+  'data': {
+      'node-name': 'str',
+      'alias': 'str',
+      'bitmaps': [ 'BitmapMigrationBitmapAlias' ] +  } }

Possible change: should 'alias' be optional (if absent, it defaults to 'node-name')?  But that can be done on top, if we like it.

I don’t know whether anyone would make use of that, though, so I’d defer it until someone wants it.

Quoted 10 lines
Quoted 8 lines

+static GHashTable *construct_alias_map(const BitmapMigrationNodeAliasList *bbm,
+                                       bool name_to_alias, +                                       Error **errp) +{
+    GHashTable *alias_map;
+    size_t max_node_name_len =
+        sizeof(((BlockDriverState *)NULL)->node_name) - 1;

Looks a bit nicer as = sizeof_field(BlockDriverState, node_name) - 1.

Oh, I didn’t know we had that.

Quoted 14 lines
Quoted 14 lines

+
+    alias_map = g_hash_table_new_full(g_str_hash, g_str_equal, +                                      g_free, free_alias_map_inner_node);

migration/block-dirty-bitmap.c +10 -0
+
+    for (; bbm; bbm = bbm->next) {
+        const BitmapMigrationNodeAlias *bmna = bbm->value;
+        const BitmapMigrationBitmapAliasList *bmbal;
+        AliasMapInnerNode *amin;
+        GHashTable *bitmaps_map;
+        const char *node_map_from, *node_map_to;
+
+        if (!id_wellformed(bmna->alias)) {
+            error_setg(errp, "The node alias '%s' is not well-formed",
Quoted 8 lines
Quoted 5 lines

+                       bmna->alias);
+            goto fail;
+        }
+
+        if (strlen(bmna->alias) > 255) {

Magic number.  UINT8_MAX seems better (since the limit really is due to our migration format limiting to one byte).

Well, yes, but qemu_put_counted_string() uses that magic number, too. *shrug*

Quoted 41 lines

migration/block-dirty-bitmap.c typedef struct DBMState { · line 282

Quoted 7 lines

+        g_hash_table_insert(alias_map, g_strdup(node_map_from), amin);

migration/block-dirty-bitmap.c +6 -0
+
+        for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+            const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+            const char *bmap_map_from, *bmap_map_to;
+
+            if (strlen(bmba->alias) > 255) {

and again

Quoted 6 lines

+                error_setg(errp,
+                           "The bitmap alias '%s' is longer than 255 bytes",
+                           bmba->alias);
+                goto fail;
+            }

Thanks for adding in the length checking since last revision!

Quoted 7 lines
migration/block-dirty-bitmap.c +1 -0
@@ -326,12 +538,29 @@ static int add_bitmaps_to_list(DBMSaveState *s,
 BlockDriverState *bs,
               return -1;
           }
   +        if (bitmap_aliases) {
+            bitmap_alias = g_hash_table_lookup(bitmap_aliases,
 bitmap_name);

migration/block-dirty-bitmap.c static int add_bitmaps_to_list(DBMSaveState *s, · line 553

Quoted 10 lines
migration/block-dirty-bitmap.c +10 -0
+            if (!bitmap_alias) {
+                /* Skip bitmaps with no alias */
+                continue;
+            }
+        } else {
+            if (strlen(bitmap_name) > 255) {
+                error_report("Cannot migrate bitmap '%s' on node '%s': "
+                             "Name is longer than 255 bytes",
+                             bitmap_name, bs_name);
+                return -1;

Another one.

Quoted 4 lines

I'm happy to make those touchups, and put this on my bitmaps queue for a pull request as soon as Paolo's meson stuff stabilizes.

Sounds good to me, thanks!

Max

Reviewed-by on v4 1/4
#4 lore raw
In reply to Max Reitz
Quoted 25 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

While touching this code, fix a bug where bitmap names longer than 255 bytes would fail an assertion in qemu_put_counted_string().

---
  qapi/migration.json            | 101 +++++++-
  migration/migration.h          |   3 +
  migration/block-dirty-bitmap.c | 409 ++++++++++++++++++++++++++++-----
  migration/migration.c          |  30 +++
  monitor/hmp-cmds.c             |  30 +++
  5 files changed, 517 insertions(+), 56 deletions(-)
qapi/migration.json +0 -0
index ea53b23dca..0c4ae102b1 100644

[..]

qapi/migration.json line 691

Quoted 10 lines

#

qapi/migration.json +9 -0
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, but not necessarily
+#          complete: On the source, unmapped bitmaps and all bitmaps
+#          on unmapped nodes will be ignored.  On the destination,
+#          all unmapped aliases in the incoming migration stream will
+#          be reported, but they will not result in failure.

Actually, on unknown alias we cancel incoming bitmap migration, which means that destination vm continues to run, other (non-bitmap) migration states continue to migrate but all further chunks of bitmap migration will be ignored. (I'm not sure it worth be mentioned)

Quoted 21 lines
qapi/migration.json +11 -1
+#          Note that the destination does not know about bitmaps it
+#          does not receive, so there is no limitation or requirement
+#          regarding the number of bitmaps received, or how they are
+#          named, or on which nodes they are placed.
+#          By default (when this parameter has never been set), bitmap
+#          names are mapped to themselves.  Nodes are mapped to their
+#          block device name if there is one, and to their node name
+#          otherwise. (Since 5.2)
+#
  # Since: 2.4
  ##
  { 'enum': 'MigrationParameter',
@@ -656,7 +712,8 @@
             'multifd-channels',
             'xbzrle-cache-size', 'max-postcopy-bandwidth',
             'max-cpu-throttle', 'multifd-compression',
-           'multifd-zlib-level' ,'multifd-zstd-level' ] }
+           'multifd-zlib-level' ,'multifd-zstd-level',
+           'block-bitmap-mapping' ] }
 
  ##

[..]

Quoted 26 lines
qapi/migration.json +15 -2
@@ -303,21 +497,39 @@ static int add_bitmaps_to_list(DBMSaveState *s, BlockDriverState *bs,
          return 0;
      }
 
+    bitmap_name = bdrv_dirty_bitmap_name(bitmap);
+
      if (!bs_name || strcmp(bs_name, "") == 0) {
          error_report("Bitmap '%s' in unnamed node can't be migrated",
-                     bdrv_dirty_bitmap_name(bitmap));
+                     bitmap_name);
          return -1;
      }
 
-    if (bs_name[0] == '#') {
+    if (alias_map) {
+        const AliasMapInnerNode *amin = g_hash_table_lookup(alias_map, bs_name);
+
+        if (!amin) {
+            /* Skip bitmaps on nodes with no alias */
+            return 0;
+        }
+
+        node_alias = amin->string;
+        bitmap_aliases = amin->subtree;
+    } else {
+        node_alias = bs_name;

qapi/migration.json static int add_bitmaps_to_list(DBMSaveState *s, BlockDriverState *bs, · line 528

Quoted 10 lines
qapi/migration.json +5 -1
+        bitmap_aliases = NULL;
+    }
+
+    if (node_alias[0] == '#') {
          error_report("Bitmap '%s' in a node with auto-generated "
                       "name '%s' can't be migrated",
-                     bdrv_dirty_bitmap_name(bitmap), bs_name);
+                     bitmap_name, node_alias);
          return -1;
      }

This check is related only to pre-alias_map behavior, so it's probably better to keep it inside else{} branch above. Still, aliases already checked to be wellformed, so this check will be always false anyway for aliases and will not hurt.

Quoted 8 lines
qapi/migration.json +2 -1
      FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
-        if (!bdrv_dirty_bitmap_name(bitmap)) {
+        bitmap_name = bdrv_dirty_bitmap_name(bitmap);
+        if (!bitmap_name) {
              continue;
          }
 

[..]

Quoted 60 lines
qapi/migration.json +33 -5
@@ -770,8 +1014,10 @@ static int dirty_bitmap_load_bits(QEMUFile *f, DBMLoadState *s)
      return 0;
  }
 
-static int dirty_bitmap_load_header(QEMUFile *f, DBMLoadState *s)
+static int dirty_bitmap_load_header(QEMUFile *f, DBMLoadState *s,
+                                    GHashTable *alias_map)
  {
+    GHashTable *bitmap_alias_map = NULL;
      Error *local_err = NULL;
      bool nothing;
      s->flags = qemu_get_bitmap_flags(f);
@@ -780,28 +1026,73 @@ static int dirty_bitmap_load_header(QEMUFile *f, DBMLoadState *s)
      nothing = s->flags == (s->flags & DIRTY_BITMAP_MIG_FLAG_EOS);
 
      if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
-        if (!qemu_get_counted_string(f, s->node_name)) {
-            error_report("Unable to read node name string");
+        if (!qemu_get_counted_string(f, s->node_alias)) {
+            error_report("Unable to read node alias string");
              return -EINVAL;
          }
+
          if (!s->cancelled) {
-            s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
+            if (alias_map) {
+                const AliasMapInnerNode *amin;
+
+                amin = g_hash_table_lookup(alias_map, s->node_alias);
+                if (!amin) {
+                    error_setg(&local_err, "Error: Unknown node alias '%s'",
+                               s->node_alias);
+                    s->bs = NULL;
+                } else {
+                    bitmap_alias_map = amin->subtree;
+                    s->bs = bdrv_lookup_bs(NULL, amin->string, &local_err);
+                }
+            } else {
+                s->bs = bdrv_lookup_bs(s->node_alias, s->node_alias,
+                                       &local_err);
+            }
              if (!s->bs) {
                  error_report_err(local_err);
                  cancel_incoming_locked(s);
              }
          }
-    } else if (!s->bs && !nothing && !s->cancelled) {
+    } else if (s->bs) {
+        if (alias_map) {
+            const AliasMapInnerNode *amin;
+
+            /* Must be present in the map, or s->bs would not be set */
+            amin = g_hash_table_lookup(alias_map, s->node_alias);
+            assert(amin != NULL);
+
+            bitmap_alias_map = amin->subtree;
+        }
+    } else if (!nothing && !s->cancelled) {
          error_report("Error: block device name is not set");
          cancel_incoming_locked(s);

qapi/migration.json static int dirty_bitmap_load_header(QEMUFile *f, DBMLoadState *s) · line 1077

Quoted 10 lines
qapi/migration.json +5 -1
      }
 
+    assert(nothing || s->cancelled || !!alias_map == !!bitmap_alias_map);
+
      if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
-        if (!qemu_get_counted_string(f, s->bitmap_name)) {
+        const char *bitmap_name;
+
+        if (!qemu_get_counted_string(f, s->bitmap_alias)) {
              error_report("Unable to read bitmap name string");

Probably s/name/alias/ like for node error message.

Quoted 21 lines
qapi/migration.json +15 -0
              return -EINVAL;
          }
+
          if (!s->cancelled) {
+            if (bitmap_alias_map) {
+                bitmap_name = g_hash_table_lookup(bitmap_alias_map,
+                                                  s->bitmap_alias);
+                if (!bitmap_name) {
+                    error_report("Error: Unknown bitmap alias '%s' on node "
+                                 "'%s' (alias '%s')", s->bitmap_alias,
+                                 s->bs->node_name, s->node_alias);
+                    cancel_incoming_locked(s);
+                }
+            } else {
+                bitmap_name = s->bitmap_alias;
+            }
+
+            g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
              s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
 
              /*

[..]

Quoted 12 lines
monitor/hmp-cmds.c +6 -0
@@ -469,6 +469,32 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
          monitor_printf(mon, "%s: '%s'\n",
              MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
              params->tls_authz);
+
+        if (params->has_block_bitmap_mapping) {
+            const BitmapMigrationNodeAliasList *bmnal;
+
+            monitor_printf(mon, "%s:\n",
+                           MigrationParameter_str(

monitor/hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 487

Quoted 10 lines
monitor/hmp-cmds.c +10 -0
+                               MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING));
+
+            for (bmnal = params->block_bitmap_mapping;
+                 bmnal;
+                 bmnal = bmnal->next)
+            {
+                const BitmapMigrationNodeAlias *bmna = bmnal->value;
+                const BitmapMigrationBitmapAliasList *bmbal;
+
+                monitor_printf(mon, "  '%s' -> '%s'\n",

'->' would look strange for incoming. Maybe, change to '--' or '~'.

Quoted 15 lines
monitor/hmp-cmds.c +10 -0
+                               bmna->node_name, bmna->alias);
+
+                for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+                    const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+
+                    monitor_printf(mon, "    '%s' -> '%s'\n",
+                                   bmba->name, bmba->alias);
+                }
+            }
+        }
      }
 
      qapi_free_MigrationParameters(params);

monitor/hmp-cmds.c void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) · line 1419

Quoted 10 lines
monitor/hmp-cmds.c +4 -0
@@ -1384,6 +1410,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
          p->has_announce_step = true;
          visit_type_size(v, param, &p->announce_step, &err);
          break;
+    case MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING:
+        error_setg(&err, "The block-bitmap-mapping parameter can only be set "
+                   "through QMP");
+        break;
      default:
          assert(0);
      }
Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw
Quoted 29 lines · Vladimir Sementsov-Ogievskiy
Quoted 25 lines · Max Reitz

This migration parameter allows mapping block node names and bitmap names to aliases for the purpose of block dirty bitmap migration.

This way, management tools can use different node and bitmap names on the source and destination and pass the mapping of how bitmaps are to be transferred to qemu (on the source, the destination, or even both with arbitrary aliases in the migration stream).

While touching this code, fix a bug where bitmap names longer than 255 bytes would fail an assertion in qemu_put_counted_string().

---
  qapi/migration.json            | 101 +++++++- migration/migration.h          |   3 + migration/block-dirty-bitmap.c | 409 ++++++++++++++++++++++++++++----- migration/migration.c          |  30 +++ monitor/hmp-cmds.c             |  30 +++

  5 files changed, 517 insertions(+), 56 deletions(-)
qapi/migration.json +0 -0
index ea53b23dca..0c4ae102b1 100644

[..]

Quoted 14 lines

qapi/migration.json line 691

Quoted 10 lines

  #

qapi/migration.json +9 -0
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+#          aliases for the purpose of dirty bitmap migration.  Such
+#          aliases may for example be the corresponding names on the
+#          opposite site.
+#          The mapping must be one-to-one, but not necessarily
+#          complete: On the source, unmapped bitmaps and all bitmaps
+#          on unmapped nodes will be ignored.  On the destination,
+#          all unmapped aliases in the incoming migration stream will
+#          be reported, but they will not result in failure.

Actually, on unknown alias we cancel incoming bitmap migration, which means that destination vm continues to run, other (non-bitmap) migration states continue to migrate but all further chunks of bitmap migration will be ignored. (I'm not sure it worth be mentioned)

Ah, yeah.

[...]

Quoted 33 lines
Quoted 33 lines
qapi/migration.json +18 -1
@@ -303,21 +497,39 @@ static int add_bitmaps_to_list(DBMSaveState *s,
 BlockDriverState *bs,
           return 0;
       }
   +    bitmap_name = bdrv_dirty_bitmap_name(bitmap);
+
       if (!bs_name || strcmp(bs_name, "") == 0) {
           error_report("Bitmap '%s' in unnamed node can't be migrated",
-                     bdrv_dirty_bitmap_name(bitmap));
+                     bitmap_name);
           return -1;
       }
   -    if (bs_name[0] == '#') {
+    if (alias_map) {
+        const AliasMapInnerNode *amin =
 g_hash_table_lookup(alias_map, bs_name);
+
+        if (!amin) {
+            /* Skip bitmaps on nodes with no alias */
+            return 0;
+        }
+
+        node_alias = amin->string;
+        bitmap_aliases = amin->subtree;
+    } else {
+        node_alias = bs_name;
+        bitmap_aliases = NULL;
+    }
+
+    if (node_alias[0] == '#') {

          error_report("Bitmap '%s' in a node with auto-generated " "name '%s' can't be migrated",
-                     bdrv_dirty_bitmap_name(bitmap), bs_name);

Quoted 8 lines
Quoted 3 lines

+                     bitmap_name, node_alias);
          return -1;
      }

This check is related only to pre-alias_map behavior, so it's probably better to keep it inside else{} branch above. Still, aliases already checked to be wellformed, so this check will be always false anyway for aliases and will not hurt.

Hm, it’s a trade off. It does look a bit weird, because how can aliases be auto-generated? But OTOH it makes it clearer that we’ll never allow non-wellformed aliases through.

[...]

Quoted 8 lines
Quoted 6 lines

      if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) { -        if (!qemu_get_counted_string(f, s->bitmap_name)) { +        const char *bitmap_name;
+
+        if (!qemu_get_counted_string(f, s->bitmap_alias)) { error_report("Unable to read bitmap name string");

Probably s/name/alias/ like for node error message.

Why not.

[...]

Quoted 18 lines
Quoted 18 lines
monitor/hmp-cmds.c +10 -0
@@ -469,6 +469,32 @@ void hmp_info_migrate_parameters(Monitor *mon,
 const QDict *qdict)
           monitor_printf(mon, "%s: '%s'\n",
               MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
               params->tls_authz);
+
+        if (params->has_block_bitmap_mapping) {
+            const BitmapMigrationNodeAliasList *bmnal;
+
+            monitor_printf(mon, "%s:\n",
+                           MigrationParameter_str(
+                              
 MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING));
+
+            for (bmnal = params->block_bitmap_mapping;
+                 bmnal;
Quoted 8 lines

monitor/hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 487

Quoted 6 lines
monitor/hmp-cmds.c +6 -0
+                 bmnal = bmnal->next)
+            {
+                const BitmapMigrationNodeAlias *bmna = bmnal->value;
+                const BitmapMigrationBitmapAliasList *bmbal;
+
+                monitor_printf(mon, "  '%s' -> '%s'\n",

'->' would look strange for incoming. Maybe, change to '--' or '~'.

Hmm, I prefer ->. The object’s name is the node/bitmap name, and that object gets an alias. So I find this to make sense even on the incoming side.

Max

author
#6 lore raw

PATCH v4 2/4 iotests.py: Add wait_for_runstate()

In reply to Max Reitz
---
 tests/qemu-iotests/iotests.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/qemu-iotests/iotests.py +4 -0
index 717b5b652c..ee93cf22db 100644
@@ -833,6 +833,10 @@ class VM(qtest.QEMUQtestMachine):
                    'Found node %s under %s (but expected %s)' % \
                    (node['name'], path, expected_node)
 
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] != runstate:
+            time.sleep(0.2)
+
 index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
 
 class QMPTestCase(unittest.TestCase):
Signature · 1 line
2.26.2
Reviewed-by on v4 2/4
#7 lore raw

Re: [PATCH v4 2/4] iotests.py: Add wait_for_runstate()

Quoted 4 lines · Max Reitz
---
  tests/qemu-iotests/iotests.py | 4 ++++
  1 file changed, 4 insertions(+)

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 838

Quoted 12 lines
tests/qemu-iotests/iotests.py +3 -0
index 717b5b652c..ee93cf22db 100644
@@ -833,6 +833,10 @@ class VM(qtest.QEMUQtestMachine):
                     'Found node %s under %s (but expected %s)' % \
                     (node['name'], path, expected_node)
 
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] != runstate:
+            time.sleep(0.2)

This looks like it could inf-loop if we have a bug where the status never changes as expected; but I guess CI bots have timeouts at higher layers that would detect if such a bug sneaks in.

Quoted 5 lines
tests/qemu-iotests/iotests.py +1 -0
+
  index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
 
  class QMPTestCase(unittest.TestCase):
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#8 lore raw

Re: [PATCH v4 2/4] iotests.py: Add wait_for_runstate()

Quoted 9 lines · * Eric Blake ([email protected]) wrote:
Quoted 4 lines · Max Reitz
---
  tests/qemu-iotests/iotests.py | 4 ++++
  1 file changed, 4 insertions(+)
Quoted 1 line
Quoted 14 lines

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 837

Quoted 10 lines
tests/qemu-iotests/iotests.py +3 -0
index 717b5b652c..ee93cf22db 100644
@@ -833,6 +833,10 @@ class VM(qtest.QEMUQtestMachine):
                     'Found node %s under %s (but expected %s)' % \
                     (node['name'], path, expected_node)
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] != runstate:
+            time.sleep(0.2)

This looks like it could inf-loop if we have a bug where the status never changes as expected; but I guess CI bots have timeouts at higher layers that would detect if such a bug sneaks in.

Although it might be useful to make sure when such a timeout lands, you know which state you thought you were waiting for.

Dave

Quoted 9 lines
Quoted 4 lines
tests/qemu-iotests/iotests.py +1 -0
+
  index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
  class QMPTestCase(unittest.TestCase):
 
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#9 lore raw

Re: [PATCH v4 2/4] iotests.py: Add wait_for_runstate()

Quoted 25 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Eric Blake ([email protected]) wrote:
Quoted 4 lines · Max Reitz
---
   tests/qemu-iotests/iotests.py | 4 ++++
   1 file changed, 4 insertions(+)
Quoted 1 line
Quoted 14 lines

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 837

Quoted 10 lines
tests/qemu-iotests/iotests.py +3 -0
index 717b5b652c..ee93cf22db 100644
@@ -833,6 +833,10 @@ class VM(qtest.QEMUQtestMachine):
                      'Found node %s under %s (but expected %s)' % \
                      (node['name'], path, expected_node)
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] != runstate:
+            time.sleep(0.2)

This looks like it could inf-loop if we have a bug where the status never changes as expected; but I guess CI bots have timeouts at higher layers that would detect if such a bug sneaks in.

Quoted 5 lines

Although it might be useful to make sure when such a timeout lands, you know which state you thought you were waiting for.

Dave

Timeout class is defined in iotests.py, so we can simply insert something like

... , timeout=60) -> None:
with Timeout(timeout, f"Timeout waiting for '{runstate}' runstate"):

     ...
Signature · 2 lines
Best regards,
Vladimir
author
#10 lore raw

Re: [PATCH v4 2/4] iotests.py: Add wait_for_runstate()

Quoted 36 lines · Vladimir Sementsov-Ogievskiy
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 19 lines · * Eric Blake ([email protected]) wrote:
Quoted 4 lines · Max Reitz

---
   tests/qemu-iotests/iotests.py | 4 ++++

   1 file changed, 4 insertions(+)
Quoted 11 lines
tests/qemu-iotests/iotests.py +2 -0
index 717b5b652c..ee93cf22db 100644
@@ -833,6 +833,10 @@ class VM(qtest.QEMUQtestMachine):
                       'Found node %s under %s (but expected %s)' % \
                       (node['name'], path, expected_node)
+    def wait_for_runstate(self, runstate: str) -> None:
+        while self.qmp('query-status')['return']['status'] !=
Quoted 8 lines
Quoted 2 lines

runstate:
+            time.sleep(0.2)

This looks like it could inf-loop if we have a bug where the status never
changes as expected; but I guess CI bots have timeouts at higher layers that
would detect if such a bug sneaks in.

Quoted 5 lines

Although it might be useful to make sure when such a timeout lands, you know which state you thought you were waiting for.

Dave

Quoted 6 lines

Timeout class is defined in iotests.py, so we can simply insert something like

 ... , timeout=60) -> None:
  with Timeout(timeout, f"Timeout waiting for '{runstate}' runstate"):
     ...

Actually, I’m not even using this function here in v4 anymore, so this patch might as well be dropped.

Max

author
#12 lore raw

PATCH v4 3/4 iotests.py: Let wait_migration() return on failure

In reply to Max Reitz

Let wait_migration() return on failure (with the return value indicating whether the migration was completed or has failed), so we can use it for migrations that are expected to fail, too.

---
 tests/qemu-iotests/iotests.py | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/qemu-iotests/iotests.py +12 -6
index ee93cf22db..f39fd580a6 100644
@@ -729,16 +729,22 @@ class VM(qtest.QEMUQtestMachine):
             }
         ]))
 
-    def wait_migration(self, expect_runstate):
+    def wait_migration(self, expect_runstate: Optional[str]) -> bool:
         while True:
             event = self.event_wait('MIGRATION')
             log(event, filters=[filter_qmp_event])
-            if event['data']['status'] == 'completed':
+            if event['data']['status'] in ('completed', 'failed'):
                 break
-        # The event may occur in finish-migrate, so wait for the expected
-        # post-migration runstate
-        while self.qmp('query-status')['return']['status'] != expect_runstate:
-            pass
+
+        if event['data']['status'] == 'completed':
+            # The event may occur in finish-migrate, so wait for the expected
+            # post-migration runstate
+            runstate = None
+            while runstate != expect_runstate:
+                runstate = self.qmp('query-status')['return']['status']
+            return True
+        else:
+            return False
 
     def node_info(self, node_name):
         nodes = self.qmp('query-named-block-nodes')
Signature · 1 line
2.26.2
Reviewed-by on v4 3/4
#13 lore raw

Re: [PATCH v4 3/4] iotests.py: Let wait_migration() return on failure

Quoted 9 lines · Max Reitz

Let wait_migration() return on failure (with the return value indicating whether the migration was completed or has failed), so we can use it for migrations that are expected to fail, too.

---
  tests/qemu-iotests/iotests.py | 18 ++++++++++++------
  1 file changed, 12 insertions(+), 6 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v4 3/4
#14 lore raw

Re: [PATCH v4 3/4] iotests.py: Let wait_migration() return on failure

In reply to Max Reitz
Quoted 26 lines · Max Reitz

Let wait_migration() return on failure (with the return value indicating whether the migration was completed or has failed), so we can use it for migrations that are expected to fail, too.

---
  tests/qemu-iotests/iotests.py | 18 ++++++++++++------
  1 file changed, 12 insertions(+), 6 deletions(-)
tests/qemu-iotests/iotests.py +2 -3
index ee93cf22db..f39fd580a6 100644
@@ -729,16 +729,22 @@ class VM(qtest.QEMUQtestMachine):
              }
          ]))
 
-    def wait_migration(self, expect_runstate):
+    def wait_migration(self, expect_runstate: Optional[str]) -> bool:
          while True:
              event = self.event_wait('MIGRATION')
              log(event, filters=[filter_qmp_event])
-            if event['data']['status'] == 'completed':
+            if event['data']['status'] in ('completed', 'failed'):
                  break
-        # The event may occur in finish-migrate, so wait for the expected

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 744

Quoted 10 lines
tests/qemu-iotests/iotests.py +7 -3
-        # post-migration runstate
-        while self.qmp('query-status')['return']['status'] != expect_runstate:
-            pass
+
+        if event['data']['status'] == 'completed':
+            # The event may occur in finish-migrate, so wait for the expected
+            # post-migration runstate
+            runstate = None
+            while runstate != expect_runstate:
+                runstate = self.qmp('query-status')['return']['status']

Would be good to use the helper from the previous patch. With it or not:

Quoted 7 lines
tests/qemu-iotests/iotests.py +3 -0
+            return True
+        else:
+            return False
 
      def node_info(self, node_name):
          nodes = self.qmp('query-named-block-nodes')
Signature · 2 lines
Best regards,
Vladimir
author
#15 lore raw

PATCH v4 4/4 iotests: Test node/bitmap aliases during migration

In reply to Max Reitz
---
 tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/300.out |   5 +
 tests/qemu-iotests/group   |   1 +
 3 files changed, 601 insertions(+)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
tests/qemu-iotests/300 new file +595 -0
new file mode 100755
index 0000000000..c6d86b1dbc
@@ -0,0 +1,595 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# Tests for dirty bitmaps migration with node aliases
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import random
+import re
+import time
+from typing import Dict, List, Optional, Union
+import iotests
+import qemu
+
+BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
+
+assert iotests.sock_dir is not None
+mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+    src_node_name: str = ''
+    dst_node_name: str = ''
+    src_bmap_name: str = ''
+    dst_bmap_name: str = ''
+
+    def setUp(self) -> None:
+        self.vm_a = iotests.VM(path_suffix='-a')
+        self.vm_a.add_blockdev(f'node-name={self.src_node_name},' \
+                               'driver=null-co')
+        self.vm_a.launch()
+
+        self.vm_b = iotests.VM(path_suffix='-b')
+        self.vm_b.add_blockdev(f'node-name={self.dst_node_name},' \
+                               'driver=null-co')
+        self.vm_b.add_incoming(f'unix:{mig_sock}')
+        self.vm_b.launch()
+
+        result = self.vm_a.qmp('block-dirty-bitmap-add',
+                               node=self.src_node_name,
+                               name=self.src_bmap_name)
+        self.assert_qmp(result, 'return', {})
+
+        # Dirty some random megabytes
+        for _ in range(9):
+            mb_ofs = random.randrange(1024)
+            self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')
+
+        result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
+                               node=self.src_node_name,
+                               name=self.src_bmap_name)
+        self.bitmap_hash_reference = result['return']['sha256']
+
+        caps = [{'capability': name, 'state': True} \
+                for name in ('dirty-bitmaps', 'events')]
+
+        for vm in (self.vm_a, self.vm_b):
+            result = vm.qmp('migrate-set-capabilities', capabilities=caps)
+            self.assert_qmp(result, 'return', {})
+
+    def tearDown(self) -> None:
+        self.vm_a.shutdown()
+        self.vm_b.shutdown()
+        try:
+            os.remove(mig_sock)
+        except OSError:
+            pass
+
+    def check_bitmap(self, bitmap_name_valid: bool) -> None:
+        result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
+                               node=self.dst_node_name,
+                               name=self.dst_bmap_name)
+        if bitmap_name_valid:
+            self.assert_qmp(result, 'return/sha256',
+                            self.bitmap_hash_reference)
+        else:
+            self.assert_qmp(result, 'error/desc',
+                            f"Dirty bitmap '{self.dst_bmap_name}' not found")
+
+    def migrate(self, bitmap_name_valid: bool = True,
+                migration_success: bool = True) -> None:
+        result = self.vm_a.qmp('migrate', uri=f'unix:{mig_sock}')
+        self.assert_qmp(result, 'return', {})
+
+        with iotests.Timeout(5, 'Timeout waiting for migration to complete'):
+            self.assertEqual(self.vm_a.wait_migration('postmigrate'),
+                             migration_success)
+            self.assertEqual(self.vm_b.wait_migration('running'),
+                             migration_success)
+
+        if migration_success:
+            self.check_bitmap(bitmap_name_valid)
+
+    def verify_dest_error(self, msg: Optional[str]) -> None:
+        """
+        Check whether the given error message is present in vm_b's log.
+        (vm_b is shut down to do so.)
+        If @msg is None, check that there has not been any error.
+        """
+        self.vm_b.shutdown()
+        if msg is None:
+            self.assertNotIn('qemu-system-', self.vm_b.get_log())
+        else:
+            self.assertIn(msg, self.vm_b.get_log())
+
+    @staticmethod
+    def mapping(node_name: str, node_alias: str,
+                bitmap_name: str, bitmap_alias: str) \
+        -> BlockBitmapMapping:
+        return [{
+            'node-name': node_name,
+            'alias': node_alias,
+            'bitmaps': [{
+                'name': bitmap_name,
+                'alias': bitmap_alias
+            }]
+        }]
+
+    def set_mapping(self, vm: iotests.VM, mapping: BlockBitmapMapping,
+                    error: Optional[str] = None) \
+        -> None:
+        """
+        Invoke migrate-set-parameters on @vm to set the given @mapping.
+        Check for success if @error is None, or verify the error message
+        if it is not.
+        On success, verify that "info migrate_parameters" on HMP returns
+        our mapping.  (Just to check its formatting code.)
+        """
+        result = vm.qmp('migrate-set-parameters',
+                        block_bitmap_mapping=mapping)
+
+        if error is None:
+            self.assert_qmp(result, 'return', {})
+
+            result = vm.qmp('human-monitor-command',
+                            command_line='info migrate_parameters')
+
+            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
+                                    result['return'], flags=re.MULTILINE)
+
+            self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
+                             self.to_hmp_mapping(mapping))
+        else:
+            self.assert_qmp(result, 'error/desc', error)
+
+    @staticmethod
+    def to_hmp_mapping(mapping: BlockBitmapMapping) -> str:
+        result = 'block-bitmap-mapping:\n'
+
+        for node in mapping:
+            result += f"  '{node['node-name']}' -> '{node['alias']}'\n"
+
+            assert isinstance(node['bitmaps'], list)
+            for bitmap in node['bitmaps']:
+                result += f"    '{bitmap['name']}' -> '{bitmap['alias']}'\n"
+
+        return result
+
+
+class TestAliasMigration(TestDirtyBitmapMigration):
+    src_node_name = 'node0'
+    dst_node_name = 'node0'
+    src_bmap_name = 'bmap0'
+    dst_bmap_name = 'bmap0'
+
+    def test_migration_without_alias(self) -> None:
+        self.migrate(self.src_node_name == self.dst_node_name and
+                     self.src_bmap_name == self.dst_bmap_name)
+
+        # Check for error message on the destination
+        if self.src_node_name != self.dst_node_name:
+            self.verify_dest_error(f"Cannot find " \
+                                   f"device={self.src_node_name} nor " \
+                                   f"node_name={self.src_node_name}")
+        else:
+            self.verify_dest_error(None)
+
+    def test_alias_on_src_migration(self) -> None:
+        mapping = self.mapping(self.src_node_name, self.dst_node_name,
+                               self.src_bmap_name, self.dst_bmap_name)
+
+        self.set_mapping(self.vm_a, mapping)
+        self.migrate()
+        self.verify_dest_error(None)
+
+    def test_alias_on_dst_migration(self) -> None:
+        mapping = self.mapping(self.dst_node_name, self.src_node_name,
+                               self.dst_bmap_name, self.src_bmap_name)
+
+        self.set_mapping(self.vm_b, mapping)
+        self.migrate()
+        self.verify_dest_error(None)
+
+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias',
+                               self.src_bmap_name, 'bmap-alias')
+
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',
+                               self.dst_bmap_name, 'bmap-alias')
+
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+        self.verify_dest_error(None)
+
+
+class TestNodeAliasMigration(TestAliasMigration):
+    src_node_name = 'node-src'
+    dst_node_name = 'node-dst'
+
+
+class TestBitmapAliasMigration(TestAliasMigration):
+    src_bmap_name = 'bmap-src'
+    dst_bmap_name = 'bmap-dst'
+
+
+class TestFullAliasMigration(TestAliasMigration):
+    src_node_name = 'node-src'
+    dst_node_name = 'node-dst'
+    src_bmap_name = 'bmap-src'
+    dst_bmap_name = 'bmap-dst'
+
+
+class TestLongBitmapNames(TestAliasMigration):
+    # Giving long bitmap names is OK, as long as there is a short alias for
+    # migration
+    src_bmap_name = 'a' * 512
+    dst_bmap_name = 'b' * 512
+
+    # Skip all tests that do not use the intermediate alias
+    def test_migration_without_alias(self) -> None:
+        pass
+
+    def test_alias_on_src_migration(self) -> None:
+        pass
+
+    def test_alias_on_dst_migration(self) -> None:
+        pass
+
+
+class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
+    src_node_name = 'node0'
+    dst_node_name = 'node0'
+    src_bmap_name = 'bmap0'
+    dst_bmap_name = 'bmap0'
+
+    """
+    Note that mapping nodes or bitmaps that do not exist is not an error.
+    """
+
+    def test_non_injective_node_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [
+            {
+                'node-name': 'node0',
+                'alias': 'common-alias',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            },
+            {
+                'node-name': 'node1',
+                'alias': 'common-alias',
+                'bitmaps': [{
+                    'name': 'bmap1',
+                    'alias': 'bmap-alias1'
+                }]
+            }
+        ]
+
+        self.set_mapping(self.vm_a, mapping,
+                         "Invalid mapping given for block-bitmap-mapping: " \
+                         "The node alias 'common-alias' is used twice")
+
+    def test_non_injective_bitmap_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': 'node0',
+            'alias': 'node-alias0',
+            'bitmaps': [
+                {
+                    'name': 'bmap0',
+                    'alias': 'common-alias'
+                },
+                {
+                    'name': 'bmap1',
+                    'alias': 'common-alias'
+                }
+            ]
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         "Invalid mapping given for block-bitmap-mapping: " \
+                         "The bitmap alias 'node-alias0'/'common-alias' is " \
+                         "used twice")
+
+    def test_ambiguous_node_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [
+            {
+                'node-name': 'node0',
+                'alias': 'node-alias0',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            },
+            {
+                'node-name': 'node0',
+                'alias': 'node-alias1',
+                'bitmaps': [{
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                }]
+            }
+        ]
+
+        self.set_mapping(self.vm_a, mapping,
+                         "Invalid mapping given for block-bitmap-mapping: " \
+                         "The node name 'node0' is mapped twice")
+
+    def test_ambiguous_bitmap_mapping(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': 'node0',
+            'alias': 'node-alias0',
+            'bitmaps': [
+                {
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias0'
+                },
+                {
+                    'name': 'bmap0',
+                    'alias': 'bmap-alias1'
+                }
+            ]
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         "Invalid mapping given for block-bitmap-mapping: " \
+                         "The bitmap 'node0'/'bmap0' is mapped twice")
+
+    def test_migratee_node_is_not_mapped_on_src(self) -> None:
+        self.set_mapping(self.vm_a, [])
+        # Should just ignore all bitmaps on unmapped nodes
+        self.migrate(False)
+        self.verify_dest_error(None)
+
+    def test_migratee_node_is_not_mapped_on_dst(self) -> None:
+        self.set_mapping(self.vm_b, [])
+        self.migrate(False)
+        self.verify_dest_error(f"Unknown node alias '{self.src_node_name}'")
+
+    def test_migratee_bitmap_is_not_mapped_on_src(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.src_node_name,
+            'alias': self.dst_node_name,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_a, mapping)
+        # Should just ignore all unmapped bitmaps
+        self.migrate(False)
+        self.verify_dest_error(None)
+
+    def test_migratee_bitmap_is_not_mapped_on_dst(self) -> None:
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.dst_node_name,
+            'alias': self.src_node_name,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_b, mapping)
+        self.migrate(False)
+        self.verify_dest_error(f"Unknown bitmap alias " \
+                               f"'{self.src_bmap_name}' " \
+                               f"on node '{self.dst_node_name}' " \
+                               f"(alias '{self.src_node_name}')")
+
+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps
+        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination
+        self.set_mapping(self.vm_b, [])
+
+        # The fact that there is a mapping on B without any bitmaps
+        # being received should be fine, not fatal
+        self.migrate(False)
+        self.verify_dest_error(None)
+
+    def test_non_wellformed_node_alias(self) -> None:
+        alias = '123-foo'
+
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.src_node_name,
+            'alias': alias,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         f"Invalid mapping given for block-bitmap-mapping: " \
+                         f"The node alias '{alias}' is not well-formed")
+
+    def test_node_alias_too_long(self) -> None:
+        alias = 'a' * 256
+
+        mapping: BlockBitmapMapping = [{
+            'node-name': self.src_node_name,
+            'alias': alias,
+            'bitmaps': []
+        }]
+
+        self.set_mapping(self.vm_a, mapping,
+                         f"Invalid mapping given for block-bitmap-mapping: " \
+                         f"The node alias '{alias}' is longer than 255 bytes")
+
+    def test_bitmap_alias_too_long(self) -> None:
+        alias = 'a' * 256
+
+        mapping = self.mapping(self.src_node_name, self.dst_node_name,
+                               self.src_bmap_name, alias)
+
+        self.set_mapping(self.vm_a, mapping,
+                         f"Invalid mapping given for block-bitmap-mapping: " \
+                         f"The bitmap alias '{alias}' is longer than 255 " \
+                         f"bytes")
+
+    def test_bitmap_name_too_long(self) -> None:
+        name = 'a' * 256
+
+        result = self.vm_a.qmp('block-dirty-bitmap-add',
+                               node=self.src_node_name,
+                               name=name)
+        self.assert_qmp(result, 'return', {})
+
+        self.migrate(False, False)
+
+        # Check for the error in the source's log
+        self.vm_a.shutdown()
+        self.assertIn(f"Cannot migrate bitmap '{name}' on node " \
+                      f"'{self.src_node_name}': Name is longer than 255 bytes",
+                      self.vm_a.get_log())
+
+        # Expect abnormal shutdown of the destination VM because of
+        # the failed migration
+        try:
+            self.vm_b.shutdown()
+        except qemu.machine.AbnormalShutdown:
+            pass
+
+    def test_aliased_bitmap_name_too_long(self) -> None:
+        # Longer than the maximum for bitmap names
+        self.dst_bmap_name = 'a' * 1024
+
+        mapping = self.mapping(self.dst_node_name, self.src_node_name,
+                               self.dst_bmap_name, self.src_bmap_name)
+
+        # We would have to create this bitmap during migration, and
+        # that would fail, because the name is too long.  Better to
+        # catch it early.
+        self.set_mapping(self.vm_b, mapping,
+                         f"Invalid mapping given for block-bitmap-mapping: " \
+                         f"The bitmap name '{self.dst_bmap_name}' is longer " \
+                         f"than 1023 bytes")
+
+    def test_node_name_too_long(self) -> None:
+        # Longer than the maximum for node names
+        self.dst_node_name = 'a' * 32
+
+        mapping = self.mapping(self.dst_node_name, self.src_node_name,
+                               self.dst_bmap_name, self.src_bmap_name)
+
+        # During migration, this would appear simply as a node that
+        # cannot be found.  Still better to catch impossible node
+        # names early (similar to test_non_wellformed_node_alias).
+        self.set_mapping(self.vm_b, mapping,
+                         f"Invalid mapping given for block-bitmap-mapping: " \
+                         f"The node name '{self.dst_node_name}' is longer " \
+                         f"than 31 bytes")
+
+
+class TestCrossAliasMigration(TestDirtyBitmapMigration):
+    """
+    Swap aliases, both to see that qemu does not get confused, and
+    that we can migrate multiple things at once.
+
+    So we migrate this:
+      node-a.bmap-a -> node-b.bmap-b
+      node-a.bmap-b -> node-b.bmap-a
+      node-b.bmap-a -> node-a.bmap-b
+      node-b.bmap-b -> node-a.bmap-a
+    """
+
+    src_node_name = 'node-a'
+    dst_node_name = 'node-b'
+    src_bmap_name = 'bmap-a'
+    dst_bmap_name = 'bmap-b'
+
+    def setUp(self) -> None:
+        TestDirtyBitmapMigration.setUp(self)
+
+        # Now create another block device and let both have two bitmaps each
+        result = self.vm_a.qmp('blockdev-add',
+                               node_name='node-b', driver='null-co')
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm_b.qmp('blockdev-add',
+                               node_name='node-a', driver='null-co')
+        self.assert_qmp(result, 'return', {})
+
+        bmaps_to_add = (('node-a', 'bmap-b'),
+                        ('node-b', 'bmap-a'),
+                        ('node-b', 'bmap-b'))
+
+        for (node, bmap) in bmaps_to_add:
+            result = self.vm_a.qmp('block-dirty-bitmap-add',
+                                   node=node, name=bmap)
+            self.assert_qmp(result, 'return', {})
+
+    @staticmethod
+    def cross_mapping() -> BlockBitmapMapping:
+        return [
+            {
+                'node-name': 'node-a',
+                'alias': 'node-b',
+                'bitmaps': [
+                    {
+                        'name': 'bmap-a',
+                        'alias': 'bmap-b'
+                    },
+                    {
+                        'name': 'bmap-b',
+                        'alias': 'bmap-a'
+                    }
+                ]
+            },
+            {
+                'node-name': 'node-b',
+                'alias': 'node-a',
+                'bitmaps': [
+                    {
+                        'name': 'bmap-b',
+                        'alias': 'bmap-a'
+                    },
+                    {
+                        'name': 'bmap-a',
+                        'alias': 'bmap-b'
+                    }
+                ]
+            }
+        ]
+
+    def verify_dest_has_all_bitmaps(self) -> None:
+        bitmaps = self.vm_b.query_bitmaps()
+
+        # Extract and sort bitmap names
+        for node in bitmaps:
+            bitmaps[node] = sorted((bmap['name'] for bmap in bitmaps[node]))
+
+        self.assertEqual(bitmaps,
+                         {'node-a': ['bmap-a', 'bmap-b'],
+                          'node-b': ['bmap-a', 'bmap-b']})
+
+    def test_alias_on_src(self) -> None:
+        self.set_mapping(self.vm_a, self.cross_mapping())
+
+        # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+        # that is enough
+        self.migrate()
+        self.verify_dest_has_all_bitmaps()
+        self.verify_dest_error(None)
+
+    def test_alias_on_dst(self) -> None:
+        self.set_mapping(self.vm_b, self.cross_mapping())
+
+        # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+        # that is enough
+        self.migrate()
+        self.verify_dest_has_all_bitmaps()
+        self.verify_dest_error(None)
+
+
+if __name__ == '__main__':
+    iotests.main(supported_protocols=['file'])
tests/qemu-iotests/300.out new file +5 -0
new file mode 100644
index 0000000000..cafb8161f7
@@ -0,0 +1,5 @@
+.....................................
+----------------------------------------------------------------------
+Ran 37 tests
+
+OK
tests/qemu-iotests/group +1 -0
index 1dc034d3af..17fdc2cdaa 100644
@@ -308,6 +308,7 @@
 296 rw
 297 meta
 299 auto quick
+300 migration
 301 backing quick
 302 quick
 304 rw quick
Signature · 1 line
2.26.2
#16 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

Quoted 4 lines · Max Reitz
---
  tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/300.out |   5 +

Rather sparse output (I hate debugging those sorts of outputs when the test is failing).

Review

Quoted 10 lines
  tests/qemu-iotests/group   |   1 +
  3 files changed, 601 insertions(+)

create mode 100755 tests/qemu-iotests/300
create mode 100644 tests/qemu-iotests/300.out

(quoted diff) +4 -0
+        # Dirty some random megabytes
+        for _ in range(9):
+            mb_ofs = random.randrange(1024)
+            self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')

It turns out that the discard operation likewise dirties the bitmap, but slightly faster (see edb90bbd). We could optimize it on top, but I'm not going to require a micro-optimizing to get it in. The test takes about 12 seconds to run for me, but you didn't mark it as such in 'group', so that's good; but it turns up a problem:

300 fail [20:55:54] [20:56:06] output mismatch (see 300.out.bad)

home/eblake/qemu-tmp2/tests/qemu-iotests/300.out.bad renamed from home/eblake/qemu-tmp2/tests/qemu-iotests/300.out +38 -2
@@ -1,5 +1,41 @@
-.....................................
+WARNING:qemu.machine:qemu received signal 11; command: "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64 -display none -vga none -chardev socket,id=mon,path=/tmp/tmp.qT831UThme/qemu-b-798452-monitor.sock -mon chardev=mon,mode=control -qtest unix:path=/tmp/tmp.qT831UThme/qemu-b-798452-qtest.sock -accel qtest -nodefaults -display none -accel qtest -blockdev node-name=node0,driver=null-co -incoming unix:/tmp/tmp.qT831UThme/mig_sock"
+.............FE.......................
+======================================================================
+ERROR: test_migratee_bitmap_is_not_mapped_on_dst (__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 435, in _do_shutdown
+    self._soft_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 415, in _soft_shutdown
+    self._qmp.cmd('quit')
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 266, in cmd
+    return self.cmd_obj(qmp_cmd)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 246, in cmd_obj
+    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
+BrokenPipeError: [Errno 32] Broken pipe
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "300", line 76, in tearDown
+    self.vm_b.shutdown()
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 465, in shutdown
+    self._do_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 438, in _do_shutdown
+    raise AbnormalShutdown("Could not perform graceful shutdown") \
+qemu.machine.AbnormalShutdown: Could not perform graceful shutdown
+
+======================================================================
+FAIL: test_migratee_bitmap_is_not_mapped_on_dst (__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "300", line 384, in test_migratee_bitmap_is_not_mapped_on_dst
+    self.migrate(False)
+  File "300", line 99, in migrate
+    self.assertEqual(self.vm_a.wait_migration('postmigrate'),
+AssertionError: False != True
+
 ----------------------------------------------------------------------
 Ran 37 tests
 
-OK
+FAILED (failures=1, errors=1)

I'm not sure why I'm seeing that, but it looks like you've got a bad deref somewhere in the alias code.

Review

Quoted 5 lines
(quoted diff) +5 -0
+class TestLongBitmapNames(TestAliasMigration):
+    # Giving long bitmap names is OK, as long as there is a short alias for
+    # migration
+    src_bmap_name = 'a' * 512
+    dst_bmap_name = 'b' * 512

This part's new compared to v3 ;) Looks like you've made several enhancements.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#17 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

Quoted 8 lines · Eric Blake
Quoted 4 lines · Max Reitz

---
  tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/300.out |   5 +

Rather sparse output (I hate debugging those sorts of outputs when the test is failing).

Hm. I don’t know, the stack trace usually gives a good idea and ./check -d gives QMP context.

The advantage of a sparse output is that we don’t need to adjust the reference output every time some optional field is added somewhere.

Quoted 90 lines
Quoted 11 lines

  tests/qemu-iotests/group   |   1 +

  3 files changed, 601 insertions(+)

  create mode 100755 tests/qemu-iotests/300
  create mode 100644 tests/qemu-iotests/300.out

+        # Dirty some random megabytes
+        for _ in range(9):
+            mb_ofs = random.randrange(1024)
+            self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')

It turns out that the discard operation likewise dirties the bitmap, but slightly faster (see edb90bbd).  We could optimize it on top, but I'm not going to require a micro-optimizing to get it in.  The test takes about 12 seconds to run for me, but you didn't mark it as such in 'group', so that's good; but it turns up a problem:

300      fail       [20:55:54] [20:56:06]                    output mismatch (see 300.out.bad)
--- /home/eblake/qemu-tmp2/tests/qemu-iotests/300.out    2020-08-19 20:53:11.087791988 -0500
+++ /home/eblake/qemu-tmp2/tests/qemu-iotests/300.out.bad    2020-08-19 20:56:06.092428756 -0500

(quoted diff) +1 -1
@@ -1,5 +1,41 @@
-.....................................
+WARNING:qemu.machine:qemu received signal 11; command:

"/home/eblake/qemu-tmp2/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64 -display none -vga none -chardev socket,id=mon,path=/tmp/tmp.qT831UThme/qemu-b-798452-monitor.sock -mon chardev=mon,mode=control -qtest unix:path=/tmp/tmp.qT831UThme/qemu-b-798452-qtest.sock -accel qtest -nodefaults -display none -accel qtest -blockdev node-name=node0,driver=null-co -incoming unix:/tmp/tmp.qT831UThme/mig_sock" +.............FE.......................
+======================================================================
+ERROR: test_migratee_bitmap_is_not_mapped_on_dst (__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 435, in _do_shutdown
+    self._soft_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 415, in _soft_shutdown
+    self._qmp.cmd('quit')
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 266, in cmd
+    return self.cmd_obj(qmp_cmd)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 246, in cmd_obj
+    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))

(quoted diff) +8 -0
+BrokenPipeError: [Errno 32] Broken pipe
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "300", line 76, in tearDown
+    self.vm_b.shutdown()
+  File

"/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 465, in shutdown
+    self._do_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 438, in _do_shutdown
+    raise AbnormalShutdown("Could not perform graceful shutdown") \

(quoted diff) +4 -0
+qemu.machine.AbnormalShutdown: Could not perform graceful shutdown
+
+======================================================================
+FAIL: test_migratee_bitmap_is_not_mapped_on_dst

(__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------

(quoted diff) +7 -0
+Traceback (most recent call last):
+  File "300", line 384, in test_migratee_bitmap_is_not_mapped_on_dst
+    self.migrate(False)
+  File "300", line 99, in migrate
+    self.assertEqual(self.vm_a.wait_migration('postmigrate'),
+AssertionError: False != True
+

 ----------------------------------------------------------------------
 Ran 37 tests

(quoted diff) +1 -1
-OK
+FAILED (failures=1, errors=1)
Quoted 2 lines

I'm not sure why I'm seeing that, but it looks like you've got a bad deref somewhere in the alias code.

Ah, crap.

This should fix it:

migration/block-dirty-bitmap.c +0 -0
index 89cb16b12c..d407dfefea 100644
@@ -1091,7 +1091,9 @@ static int dirty_bitmap_load_header(QEMUFile *f,

DBMLoadState *s,

             } else {
                 bitmap_name = s->bitmap_alias;
             }

+ }

+        if (!s->cancelled) {
             g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
             s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);

I had this originally, and then I decided to drop that hunk just before sending v4 because I couldn’t see the point. But we need it, because if the bitmap alias is unknown, the migration is cancelled, so we need to re-check s->cancalled after the alias lookup block.

Would you be OK with squashing that into patch 1?

Max

#18 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

Quoted 120 lines · Max Reitz
Quoted 8 lines · Eric Blake
Quoted 4 lines · Max Reitz
---
   tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++
   tests/qemu-iotests/300.out |   5 +

Rather sparse output (I hate debugging those sorts of outputs when the test is failing).

Hm. I don’t know, the stack trace usually gives a good idea and ./check -d gives QMP context.

The advantage of a sparse output is that we don’t need to adjust the reference output every time some optional field is added somewhere.

Quoted 90 lines
Quoted 11 lines
   tests/qemu-iotests/group   |   1 +
   3 files changed, 601 insertions(+)

  create mode 100755 tests/qemu-iotests/300
  create mode 100644 tests/qemu-iotests/300.out

+        # Dirty some random megabytes
+        for _ in range(9):
+            mb_ofs = random.randrange(1024)
+            self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')

It turns out that the discard operation likewise dirties the bitmap, but slightly faster (see edb90bbd).  We could optimize it on top, but I'm not going to require a micro-optimizing to get it in.  The test takes about 12 seconds to run for me, but you didn't mark it as such in 'group', so that's good; but it turns up a problem:

300      fail       [20:55:54] [20:56:06]                    output mismatch (see 300.out.bad)
--- /home/eblake/qemu-tmp2/tests/qemu-iotests/300.out    2020-08-19 20:53:11.087791988 -0500
+++ /home/eblake/qemu-tmp2/tests/qemu-iotests/300.out.bad    2020-08-19 20:56:06.092428756 -0500

(quoted diff) +1 -1
@@ -1,5 +1,41 @@
-.....................................
+WARNING:qemu.machine:qemu received signal 11; command:

"/home/eblake/qemu-tmp2/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64 -display none -vga none -chardev socket,id=mon,path=/tmp/tmp.qT831UThme/qemu-b-798452-monitor.sock -mon chardev=mon,mode=control -qtest unix:path=/tmp/tmp.qT831UThme/qemu-b-798452-qtest.sock -accel qtest -nodefaults -display none -accel qtest -blockdev node-name=node0,driver=null-co -incoming unix:/tmp/tmp.qT831UThme/mig_sock" +.............FE.......................
+======================================================================
+ERROR: test_migratee_bitmap_is_not_mapped_on_dst (__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 435, in _do_shutdown
+    self._soft_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 415, in _soft_shutdown
+    self._qmp.cmd('quit')
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 266, in cmd
+    return self.cmd_obj(qmp_cmd)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/qmp.py", line 246, in cmd_obj
+    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))

(quoted diff) +8 -0
+BrokenPipeError: [Errno 32] Broken pipe
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "300", line 76, in tearDown
+    self.vm_b.shutdown()
+  File

"/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 465, in shutdown
+    self._do_shutdown(timeout, has_quit)
+  File "/home/eblake/qemu-tmp2/tests/qemu-iotests/../../python/qemu/machine.py", line 438, in _do_shutdown
+    raise AbnormalShutdown("Could not perform graceful shutdown") \

(quoted diff) +4 -0
+qemu.machine.AbnormalShutdown: Could not perform graceful shutdown
+
+======================================================================
+FAIL: test_migratee_bitmap_is_not_mapped_on_dst

(__main__.TestBlockBitmapMappingErrors)
+----------------------------------------------------------------------

(quoted diff) +8 -1
+Traceback (most recent call last):
+  File "300", line 384, in test_migratee_bitmap_is_not_mapped_on_dst
+    self.migrate(False)
+  File "300", line 99, in migrate
+    self.assertEqual(self.vm_a.wait_migration('postmigrate'),
+AssertionError: False != True
+
  ----------------------------------------------------------------------
  Ran 37 tests
 
-OK
+FAILED (failures=1, errors=1)
Quoted 2 lines

I'm not sure why I'm seeing that, but it looks like you've got a bad deref somewhere in the alias code.

Ah, crap.

This should fix it:

migration/block-dirty-bitmap.c +0 -0
index 89cb16b12c..d407dfefea 100644

migration/block-dirty-bitmap.c static int dirty_bitmap_load_header(QEMUFile *f, · line 1099

Quoted 9 lines
migration/block-dirty-bitmap.c +2 -0
@@ -1091,7 +1091,9 @@ static int dirty_bitmap_load_header(QEMUFile *f,
 DBMLoadState *s,
              } else {
                  bitmap_name = s->bitmap_alias;
              }
+        }
 
+        if (!s->cancelled) {
              g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
              s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);

That's correct thing to do

Quoted 7 lines

I had this originally, and then I decided to drop that hunk just before sending v4 because I couldn’t see the point. But we need it, because if the bitmap alias is unknown, the migration is cancelled, so we need to re-check s->cancalled after the alias lookup block.

Would you be OK with squashing that into patch 1?

Ok for me.

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v4 4/4
#19 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

In reply to Max Reitz

# MYPYPATH=../../python/ mypy 300
300:154: error: Item "None" of "Optional[Match[Any]]" has no attribute "group" Found 1 error in 1 file (checked 1 source file)

- the only complain. Suggest a fix:

tests/qemu-iotests/300 +4 -4
index c6d86b1dbc..0241903743 100755
@@ -148,11 +148,11 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             result = vm.qmp('human-monitor-command',
                             command_line='info migrate_parameters')
 
-            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
-                                    result['return'], flags=re.MULTILINE)
+            m = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
+                          result['return'], flags=re.MULTILINE)
+            hmp_mapping = m.group(0).replace('\r', '') if m else None
 
-            self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
-                             self.to_hmp_mapping(mapping))
+            self.assertEqual(hmp_mapping, self.to_hmp_mapping(mapping))
         else:
             self.assert_qmp(result, 'error/desc', error)
 

===

# flake8 300
300:24:1: F401 'time' imported but unused
300:34:1: E302 expected 2 blank lines, found 1
300:42:67: E502 the backslash is redundant between brackets
300:47:67: E502 the backslash is redundant between brackets
300:67:53: E502 the backslash is redundant between brackets
300:122:9: E125 continuation line with same indent as next logical line
300:134:9: E125 continuation line with same indent as next logical line
300:185:52: E502 the backslash is redundant between brackets
300:186:72: E502 the backslash is redundant between brackets
300:285:77: E502 the backslash is redundant between brackets
300:305:77: E502 the backslash is redundant between brackets
300:306:78: E502 the backslash is redundant between brackets
300:330:77: E502 the backslash is redundant between brackets
300:350:77: E502 the backslash is redundant between brackets
300:385:57: E502 the backslash is redundant between brackets
300:386:59: E502 the backslash is redundant between brackets
300:387:67: E502 the backslash is redundant between brackets
300:412:78: E502 the backslash is redundant between brackets
300:425:78: E502 the backslash is redundant between brackets
300:435:78: E502 the backslash is redundant between brackets
300:436:76: E502 the backslash is redundant between brackets
300:451:66: E502 the backslash is redundant between brackets
300:473:78: E502 the backslash is redundant between brackets
300:474:79: E502 the backslash is redundant between brackets
300:488:78: E502 the backslash is redundant between brackets
300:489:77: E502 the backslash is redundant between brackets

- I post it just because ALE plugin in vim highlights all these things for me. Up to you, I don't ask you to fix it.

Quoted 9 lines · Max Reitz
---
  tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/300.out |   5 +
  tests/qemu-iotests/group   |   1 +
  3 files changed, 601 insertions(+)

create mode 100755 tests/qemu-iotests/300
create mode 100644 tests/qemu-iotests/300.out

[..]

Review

Quoted 11 lines
(quoted diff) +11 -0
+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias',
+                               self.src_bmap_name, 'bmap-alias')
+
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',
+                               self.dst_bmap_name, 'bmap-alias')
+
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+        self.verify_dest_error(None)

Hmm, probably verify_dest_error() should be called from migrate(), as you call it (almost) always after migrate()

Quoted 2 lines
(quoted diff) +2 -0
+
+

[..]

Review

Quoted 6 lines
(quoted diff) +6 -0
+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps
+        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination
+        self.set_mapping(self.vm_b, [])

Seems, you wanted to specify non-empty mapping for vm_b, yes? With any non-empty mapping here, just to better correspond to the comments:

(or keep this case with both mappings empty, and add one similar with empty mapping only on src)

Signature · 2 lines
Best regards,
Vladimir
#20 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

Quoted 16 lines · Vladimir Sementsov-Ogievskiy

# MYPYPATH=../../python/ mypy 300
300:154: error: Item "None" of "Optional[Match[Any]]" has no attribute "group" Found 1 error in 1 file (checked 1 source file)

- the only complain. Suggest a fix:

tests/qemu-iotests/300 +0 -1
index c6d86b1dbc..0241903743 100755
@@ -148,11 +148,11 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
              result = vm.qmp('human-monitor-command',
                              command_line='info migrate_parameters')
 
-            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n  
 .*)*\n',

tests/qemu-iotests/300 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 158

Quoted 10 lines
tests/qemu-iotests/300 +4 -3
-                                    result['return'], flags=re.MULTILINE)
+            m = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
+                          result['return'], flags=re.MULTILINE)
+            hmp_mapping = m.group(0).replace('\r', '') if m else None
 
-            self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
-                             self.to_hmp_mapping(mapping))
+            self.assertEqual(hmp_mapping, self.to_hmp_mapping(mapping))
          else:
              self.assert_qmp(result, 'error/desc', error)

Thanks; I'll fold that in to v5.

Quoted 31 lines

===

# flake8 300
300:24:1: F401 'time' imported but unused
300:34:1: E302 expected 2 blank lines, found 1
300:42:67: E502 the backslash is redundant between brackets
300:47:67: E502 the backslash is redundant between brackets
300:67:53: E502 the backslash is redundant between brackets
300:122:9: E125 continuation line with same indent as next logical line
300:134:9: E125 continuation line with same indent as next logical line
300:185:52: E502 the backslash is redundant between brackets
300:186:72: E502 the backslash is redundant between brackets
300:285:77: E502 the backslash is redundant between brackets
300:305:77: E502 the backslash is redundant between brackets
300:306:78: E502 the backslash is redundant between brackets
300:330:77: E502 the backslash is redundant between brackets
300:350:77: E502 the backslash is redundant between brackets
300:385:57: E502 the backslash is redundant between brackets
300:386:59: E502 the backslash is redundant between brackets
300:387:67: E502 the backslash is redundant between brackets
300:412:78: E502 the backslash is redundant between brackets
300:425:78: E502 the backslash is redundant between brackets
300:435:78: E502 the backslash is redundant between brackets
300:436:76: E502 the backslash is redundant between brackets
300:451:66: E502 the backslash is redundant between brackets
300:473:78: E502 the backslash is redundant between brackets
300:474:79: E502 the backslash is redundant between brackets
300:488:78: E502 the backslash is redundant between brackets
300:489:77: E502 the backslash is redundant between brackets
Quoted 2 lines

- I post it just because ALE plugin in vim highlights all these things for me. Up to you, I don't ask you to fix it.

Seems like an easy enough touchup to make.

Quoted 5 lines
Quoted 5 lines

+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias', +                               self.src_bmap_name, 'bmap-alias') +
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',

Quoted 8 lines
Quoted 6 lines

+                               self.dst_bmap_name, 'bmap-alias') +
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+        self.verify_dest_error(None)

Hmm, probably verify_dest_error() should be called from migrate(), as you call it (almost) always after migrate()

This one I did not touch; it can be a followup patch if desired.

Quoted 13 lines
Quoted 3 lines
(quoted diff) +2 -0
+
+

[..]

Quoted 6 lines

+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps +        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination +        self.set_mapping(self.vm_b, [])

Quoted 5 lines

Seems, you wanted to specify non-empty mapping for vm_b, yes? With any non-empty mapping here, just to better correspond to the comments:

(or keep this case with both mappings empty, and add one similar with empty mapping only on src)

Adding another case will now have to be a separate patch. At any rate, I've taken the liberty of including your R-b in my staging tree for the pending pull request, let me know if I should drop it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#21 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

Quoted 94 lines · Eric Blake
Quoted 23 lines · Vladimir Sementsov-Ogievskiy

# MYPYPATH=../../python/ mypy 300
300:154: error: Item "None" of "Optional[Match[Any]]" has no attribute "group" Found 1 error in 1 file (checked 1 source file)

- the only complain. Suggest a fix:

tests/qemu-iotests/300 +4 -4
index c6d86b1dbc..0241903743 100755
@@ -148,11 +148,11 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
               result = vm.qmp('human-monitor-command',
                               command_line='info migrate_parameters')
 
-            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n .*)*\n',
-                                    result['return'], flags=re.MULTILINE)
+            m = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n',
+                          result['return'], flags=re.MULTILINE)
+            hmp_mapping = m.group(0).replace('\r', '') if m else None
 
-            self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
-                             self.to_hmp_mapping(mapping))
+            self.assertEqual(hmp_mapping, self.to_hmp_mapping(mapping))
Quoted 2 lines

          else:
              self.assert_qmp(result, 'error/desc', error)

Thanks; I'll fold that in to v5.

Quoted 31 lines

===

# flake8 300
300:24:1: F401 'time' imported but unused
300:34:1: E302 expected 2 blank lines, found 1
300:42:67: E502 the backslash is redundant between brackets
300:47:67: E502 the backslash is redundant between brackets
300:67:53: E502 the backslash is redundant between brackets
300:122:9: E125 continuation line with same indent as next logical line
300:134:9: E125 continuation line with same indent as next logical line
300:185:52: E502 the backslash is redundant between brackets
300:186:72: E502 the backslash is redundant between brackets
300:285:77: E502 the backslash is redundant between brackets
300:305:77: E502 the backslash is redundant between brackets
300:306:78: E502 the backslash is redundant between brackets
300:330:77: E502 the backslash is redundant between brackets
300:350:77: E502 the backslash is redundant between brackets
300:385:57: E502 the backslash is redundant between brackets
300:386:59: E502 the backslash is redundant between brackets
300:387:67: E502 the backslash is redundant between brackets
300:412:78: E502 the backslash is redundant between brackets
300:425:78: E502 the backslash is redundant between brackets
300:435:78: E502 the backslash is redundant between brackets
300:436:76: E502 the backslash is redundant between brackets
300:451:66: E502 the backslash is redundant between brackets
300:473:78: E502 the backslash is redundant between brackets
300:474:79: E502 the backslash is redundant between brackets
300:488:78: E502 the backslash is redundant between brackets
300:489:77: E502 the backslash is redundant between brackets
Quoted 1 line

- I post it just because ALE plugin in vim highlights all these things for me. Up to you, I don't ask you to fix it.

Seems like an easy enough touchup to make.

Quoted 5 lines
Quoted 5 lines

+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias', +                               self.src_bmap_name, 'bmap-alias') +
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',

Quoted 8 lines
Quoted 6 lines

+                               self.dst_bmap_name, 'bmap-alias') +
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+        self.verify_dest_error(None)

Hmm, probably verify_dest_error() should be called from migrate(), as you call it (almost) always after migrate()

This one I did not touch; it can be a followup patch if desired.

Quoted 13 lines
Quoted 3 lines
(quoted diff) +2 -0
+
+

[..]

Quoted 6 lines

+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps +        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination +        self.set_mapping(self.vm_b, [])

Quoted 8 lines
Quoted 5 lines

Seems, you wanted to specify non-empty mapping for vm_b, yes? With any non-empty mapping here, just to better correspond to the comments:

(or keep this case with both mappings empty, and add one similar with empty mapping only on src)

Adding another case will now have to be a separate patch.  At any rate, I've taken the liberty of including your R-b in my staging tree for the pending pull request, let me know if I should drop it.

OK, thanks, no objections.

Signature · 2 lines
Best regards,
Vladimir
author
#22 lore raw

Re: [PATCH v4 4/4] iotests: Test node/bitmap aliases during migration

In reply to Vladimir Sementsov-Ogievskiy
Quoted 4 lines · Vladimir Sementsov-Ogievskiy

# MYPYPATH=../../python/ mypy 300
300:154: error: Item "None" of "Optional[Match[Any]]" has no attribute "group"
Found 1 error in 1 file (checked 1 source file)

:(

Quoted 12 lines

- the only complain. Suggest a fix:

tests/qemu-iotests/300 +0 -0
index c6d86b1dbc..0241903743 100755
@@ -148,11 +148,11 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
              result = vm.qmp('human-monitor-command',
                              command_line='info migrate_parameters')


-            hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n .*)*\n',

Quoted 8 lines

-                                    result['return'], flags=re.MULTILINE) +            m = re.search(r'^block-bitmap-mapping:\r?(\n  .*)*\n', +                          result['return'], flags=re.MULTILINE) +            hmp_mapping = m.group(0).replace('\r', '') if m else None
-            self.assertEqual(hmp_mapping.group(0).replace('\r', ''), -                             self.to_hmp_mapping(mapping))
+            self.assertEqual(hmp_mapping, self.to_hmp_mapping(mapping))

Looks good, thanks.

Quoted 6 lines

         else:
             self.assert_qmp(result, 'error/desc', error)
===

# flake8 300

[...]

Quoted 2 lines

- I post it just because ALE plugin in vim highlights all these things for me. Up to you, I don't ask you to fix it.

Thanks, I’ll run it from now on (unless I forget, like mypy above...).

Quoted 13 lines
Quoted 9 lines · Max Reitz

---
  tests/qemu-iotests/300     | 595 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/300.out |   5 + tests/qemu-iotests/group   |   1 +

  3 files changed, 601 insertions(+)

  create mode 100755 tests/qemu-iotests/300
  create mode 100644 tests/qemu-iotests/300.out

[..]

Quoted 14 lines

Review

Quoted 11 lines

+    def test_alias_on_both_migration(self) -> None:
+        src_map = self.mapping(self.src_node_name, 'node-alias', +                               self.src_bmap_name, 'bmap-alias')

(quoted diff) +8 -0
+
+        dst_map = self.mapping(self.dst_node_name, 'node-alias',
+                               self.dst_bmap_name, 'bmap-alias')
+
+        self.set_mapping(self.vm_a, src_map)
+        self.set_mapping(self.vm_b, dst_map)
+        self.migrate()
+        self.verify_dest_error(None)

Hmm, probably verify_dest_error() should be called from migrate(), as you call it (almost) always after migrate()

I don’t know, it shuts down the destination VM, so it would seem a bit strange to do that as part of migrate().

Quoted 8 lines
Quoted 6 lines

+    def test_unused_mapping_on_dst(self) -> None:
+        # Let the source not send any bitmaps +        self.set_mapping(self.vm_a, [])
+
+        # Establish some mapping on the destination +        self.set_mapping(self.vm_b, [])

Seems, you wanted to specify non-empty mapping for vm_b, yes?

Oops. I don’t know how it could have happened that I forgot that.

Quoted 5 lines

With any non-empty mapping here, just to better correspond to the comments:

(or keep this case with both mappings empty, and add one similar with empty mapping only on src)

I’ll see in what shape this patch lands in master, and then I’ll probably add that other case, yes.

Max