Skip to contentSkip to conversation

migration: introduce dirtylimit capability

PATCH In review [email protected] 2 participants 16 messages Jul 23, 2022 – Aug 18, 2022

lore
Share & tools
lore
#0 lore raw

cover migration: introduce dirtylimit capability

From: Hyman Huang(黄勇) <[email protected]>

Abstract
========

This series added a new migration capability called "dirtylimit". It can be enabled when dirty ring is enabled, and it'll improve the vCPU performance during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/[email protected]/

As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of migration can make the read-process not be penalized. This series wires up the vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup of dirtylimit during live migration.

To validate the implementation, i tested a 32 vCPU vm live migration with such model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}

The following are results:

host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s

  |---------------------+--------+------------+---------------|
  | UnixBench test item | Normal | Dirtylimit | Auto-converge |
  |---------------------+--------+------------+---------------|
  | dhry2reg            | 32800  | 32786      | 25292         |
  | whetstone-double    | 10326  | 10315      | 9847          |
  | pipe                | 15442  | 15271      | 14506         |
  | context1            | 7260   | 6235       | 4514          |
  | spawn               | 3663   | 3317       | 3249          |
  | syscall             | 4669   | 4667       | 3841          |
  |---------------------+--------+------------+---------------|

From the data above we can draw a conclusion that vcpus that do not dirty memory in vm are almost unaffected during the dirtylimit migration, but the auto converge way does.

I also tested the total time of dirtylimit migration with variable dirty memory size in vm.

senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s

  |-----------------------+----------------+-------------------|
  | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
  |-----------------------+----------------+-------------------|
  | 60                    | 2014           | 2131              |
  | 70                    | 5381           | 12590             |
  | 90                    | 6037           | 33545             |
  | 110                   | 7660           | [*]               |
  |-----------------------+----------------+-------------------|

[*]: This case means migration is not convergent.

senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s

  |-----------------------+----------------+-------------------|
  | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
  |-----------------------+----------------+-------------------|
  | 1600                  | 15842          | 27548             |
  | 2000                  | 19026          | 38447             |
  | 2400                  | 19897          | 46381             |
  | 2800                  | 22338          | 57149             |
  |-----------------------+----------------+-------------------|

Above data shows that dirtylimit way of migration can also reduce the total time of migration and it achieves convergence more easily in some case.

In addition to implement dirtylimit capability itself, this series add 3 tests for migration, aiming at playing around for developer simply:
1. qtest for dirty limit migration
2. support dirty ring way of migration for guestperf tool
3. support dirty limit migration for guestperf tool

Please review, thanks !

Hyman Huang (8):
qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters
migration: Introduce dirty-limit capability
migration: Implement dirty-limit convergence algo
migration: Export dirty-limit time info
tests: Add migration dirty-limit capability test tests/migration: Introduce dirty-ring-size option into guestperf tests/migration: Introduce dirty-limit into guestperf

 include/sysemu/dirtylimit.h             |  2 +
 migration/migration.c                   | 50 ++++++++++++++++++
 migration/migration.h                   |  1 +
 migration/ram.c                         | 53 ++++++++++++++-----
 migration/trace-events                  |  1 +
 monitor/hmp-cmds.c                      | 26 ++++++++++
 qapi/migration.json                     | 57 ++++++++++++++++----
 softmmu/dirtylimit.c                    | 33 +++++++++++-
 tests/migration/guestperf/comparison.py | 14 +++++
 tests/migration/guestperf/engine.py     | 33 +++++++++++-
 tests/migration/guestperf/hardware.py   |  8 ++-
 tests/migration/guestperf/progress.py   | 17 +++++-
 tests/migration/guestperf/scenario.py   | 11 +++-
 tests/migration/guestperf/shell.py      | 25 ++++++++-
 tests/qtest/migration-test.c            | 92 +++++++++++++++++++++++++++++++++
 15 files changed, 392 insertions(+), 31 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/8 qapi/migration: Introduce x-vcpu-dirty-limit-period parameter

From: Hyman Huang(黄勇) <[email protected]>

Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.

---
 migration/migration.c | 16 ++++++++++++++++
 monitor/hmp-cmds.c    |  8 ++++++++
 qapi/migration.json   | 31 ++++++++++++++++++++++++-------
 3 files changed, 48 insertions(+), 7 deletions(-)
migration/migration.c +16 -0
index e03f698..7b19f85 100644
@@ -116,6 +116,8 @@
 #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS    5
 #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */
+
 static NotifierList migration_state_notifiers =
     NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
 
@@ -962,6 +964,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
                        s->parameters.block_bitmap_mapping);
     }
 
+    params->has_x_vcpu_dirty_limit_period = true;
+    params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period;
+
     return params;
 }
 
@@ -1662,6 +1667,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
         dest->has_block_bitmap_mapping = true;
         dest->block_bitmap_mapping = params->block_bitmap_mapping;
     }
+
+    if (params->has_x_vcpu_dirty_limit_period) {
+        dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1784,6 +1793,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
             QAPI_CLONE(BitmapMigrationNodeAliasList,
                        params->block_bitmap_mapping);
     }
+    if (params->has_x_vcpu_dirty_limit_period) {
+        s->parameters.x_vcpu_dirty_limit_period =
+            params->x_vcpu_dirty_limit_period;
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -4384,6 +4397,9 @@ static Property migration_properties[] = {
     DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds),
     DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_hostname),
     DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz),
+    DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState,
+                       parameters.x_vcpu_dirty_limit_period,
+                       DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD),
 
     /* Migration capabilities */
     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
monitor/hmp-cmds.c +8 -0
index a6dc79e..64c996c 100644
@@ -532,6 +532,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
                 }
             }
         }
+
+        monitor_printf(mon, "%s: %" PRIu64 " ms\n",
+        MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD),
+        params->x_vcpu_dirty_limit_period);
     }
 
     qapi_free_MigrationParameters(params);
@@ -1351,6 +1355,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         error_setg(&err, "The block-bitmap-mapping parameter can only be set "
                    "through QMP");
         break;
+    case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD:
+        p->has_x_vcpu_dirty_limit_period = true;
+        visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err);
+        break;
     default:
         assert(0);
     }
qapi/migration.json +24 -7
index 81185d4..332c087 100644
@@ -776,8 +776,12 @@
 #                        block device name if there is one, and to their node name
 #                        otherwise. (Since 5.2)
 #
+# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
+#                             Defaults to 500ms. (Since 7.1)
+#
 # Features:
-# @unstable: Member @x-checkpoint-delay is experimental.
+# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
+#            are experimental.
 #
 # Since: 2.4
 ##
@@ -795,8 +799,9 @@
            'multifd-channels',
            'xbzrle-cache-size', 'max-postcopy-bandwidth',
            'max-cpu-throttle', 'multifd-compression',
-           'multifd-zlib-level' ,'multifd-zstd-level',
-           'block-bitmap-mapping' ] }
+           'multifd-zlib-level', 'multifd-zstd-level',
+           'block-bitmap-mapping',
+           { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] }
 
 ##
 # @MigrateSetParameters:
@@ -941,8 +946,12 @@
 #                        block device name if there is one, and to their node name
 #                        otherwise. (Since 5.2)
 #
+# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
+#                             Defaults to 500ms. (Since 7.1)
+#
 # Features:
-# @unstable: Member @x-checkpoint-delay is experimental.
+# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
+#            are experimental.
 #
 # Since: 2.4
 ##
@@ -976,7 +985,9 @@
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
             '*multifd-zstd-level': 'uint8',
-            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
+            '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
+                                            'features': [ 'unstable' ] } } }
 
 ##
 # @migrate-set-parameters:
@@ -1141,8 +1152,12 @@
 #                        block device name if there is one, and to their node name
 #                        otherwise. (Since 5.2)
 #
+# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
+#                             Defaults to 500ms. (Since 7.1)
+#
 # Features:
-# @unstable: Member @x-checkpoint-delay is experimental.
+# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
+#            are experimental.
 #
 # Since: 2.4
 ##
@@ -1174,7 +1189,9 @@
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
             '*multifd-zstd-level': 'uint8',
-            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
+            '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
+            '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
+                                            'features': [ 'unstable' ] } } }
 
 ##
 # @query-migrate-parameters:
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [PATCH 1/8] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter

Quoted 18 lines · On Sat, Jul 23, 2022 at 03:49:13PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.

---
 migration/migration.c | 16 ++++++++++++++++
 monitor/hmp-cmds.c    |  8 ++++++++
 qapi/migration.json   | 31 ++++++++++++++++++++++++-------
 3 files changed, 48 insertions(+), 7 deletions(-)
migration/migration.c +0 -0
index e03f698..7b19f85 100644

migration/migration.c line 119

Quoted 4 lines
migration/migration.c +1 -0
@@ -116,6 +116,8 @@
 #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS    5
 #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */

Why 500 but not DIRTYLIMIT_CALC_TIME_MS?

Is it intended to make this parameter experimental, but the other one not?

Thanks,

Signature · 1 line
Peter Xu
author
#3 lore raw

Re: [PATCH 1/8] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter

Quoted 22 lines · Peter Xu
Quoted 21 lines · On Sat, Jul 23, 2022 at 03:49:13PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.

---
  migration/migration.c | 16 ++++++++++++++++
  monitor/hmp-cmds.c    |  8 ++++++++
  qapi/migration.json   | 31 ++++++++++++++++++++++++-------
  3 files changed, 48 insertions(+), 7 deletions(-)
migration/migration.c +0 -0
index e03f698..7b19f85 100644
@@ -116,6 +116,8 @@
  #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS    5
  #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
Quoted 3 lines

migration/migration.c line 119

Quoted 1 line
migration/migration.c +1 -0
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */

Why 500 but not DIRTYLIMIT_CALC_TIME_MS?

This is a empirical value actually, the iteration time of migration is less than 1000ms normally. In my test it varies from 200ms to 500ms, we assume iteration time is 500ms and calculation period is 1000ms, so 2 iteration pass when 1 dirty page rate get calculated. We want calculation period as close to iteration time as possible so that 1 iteration pass, 1 new dirty page rate be calculated and get compared, hoping the dirtylimit working more precisely.

But as the "x-" prefix implies, i'm a little unsure that if the solution works。

Quoted 2 lines

Is it intended to make this parameter experimental, but the other one not?

Since i'm not very sure vcpu-dirty-limit-period have impact on migration(as described above), so it is made experimental. As to vcpu-dirty-limit, it indeed have impact on migration in theory, so it is not made experimental. But from another point of view, 2 parameter are introduced in the first time and none of them suffer lots of tests, it is also reasonable to make 2 parameter experimental, i'm not insist that.

Yong

Quoted 3 lines

Thanks,

#4 lore raw

PATCH 2/8 qapi/migration: Introduce vcpu-dirty-limit parameters

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.

"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.

This two parameters are used to help implement the dirty page rate limit algo of migration.

---
 migration/migration.c | 14 ++++++++++++++
 monitor/hmp-cmds.c    |  8 ++++++++
 qapi/migration.json   | 18 +++++++++++++++---
 3 files changed, 37 insertions(+), 3 deletions(-)
migration/migration.c +14 -0
index 7b19f85..ed1a47b 100644
@@ -117,6 +117,7 @@
 #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
 #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT            1       /* MB/s */
 
 static NotifierList migration_state_notifiers =
     NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
@@ -967,6 +968,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->has_x_vcpu_dirty_limit_period = true;
     params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period;
 
+    params->has_vcpu_dirty_limit = true;
+    params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit;
+
     return params;
 }
 
@@ -1671,6 +1675,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_x_vcpu_dirty_limit_period) {
         dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period;
     }
+
+    if (params->has_vcpu_dirty_limit) {
+        dest->vcpu_dirty_limit = params->vcpu_dirty_limit;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1797,6 +1805,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
         s->parameters.x_vcpu_dirty_limit_period =
             params->x_vcpu_dirty_limit_period;
     }
+    if (params->has_vcpu_dirty_limit) {
+        s->parameters.vcpu_dirty_limit = params->vcpu_dirty_limit;
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -4400,6 +4411,9 @@ static Property migration_properties[] = {
     DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState,
                        parameters.x_vcpu_dirty_limit_period,
                        DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD),
+    DEFINE_PROP_UINT64("vcpu-dirty-limit", MigrationState,
+                       parameters.vcpu_dirty_limit,
+                       DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT),
 
     /* Migration capabilities */
     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
monitor/hmp-cmds.c +8 -0
index 64c996c..acbc5e8 100644
@@ -536,6 +536,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: %" PRIu64 " ms\n",
         MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD),
         params->x_vcpu_dirty_limit_period);
+
+        monitor_printf(mon, "%s: %" PRIu64 " MB/s\n",
+            MigrationParameter_str(MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT),
+            params->vcpu_dirty_limit);
     }
 
     qapi_free_MigrationParameters(params);
@@ -1359,6 +1363,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_x_vcpu_dirty_limit_period = true;
         visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err);
         break;
+    case MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT:
+        p->has_vcpu_dirty_limit = true;
+        visit_type_size(v, param, &p->vcpu_dirty_limit, &err);
+        break;
     default:
         assert(0);
     }
qapi/migration.json +15 -3
index 332c087..0963bab 100644
@@ -779,6 +779,9 @@
 # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
 #                             Defaults to 500ms. (Since 7.1)
 #
+# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration.
+#                    Defaults to 1. (Since 7.1)
+#
 # Features:
 # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
 #            are experimental.
@@ -801,7 +804,8 @@
            'max-cpu-throttle', 'multifd-compression',
            'multifd-zlib-level', 'multifd-zstd-level',
            'block-bitmap-mapping',
-           { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] }
+           { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
+           'vcpu-dirty-limit'] }
 
 ##
 # @MigrateSetParameters:
@@ -949,6 +953,9 @@
 # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
 #                             Defaults to 500ms. (Since 7.1)
 #
+# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration.
+#                    Defaults to 1. (Since 7.1)
+#
 # Features:
 # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
 #            are experimental.
@@ -987,7 +994,8 @@
             '*multifd-zstd-level': 'uint8',
             '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
-                                            'features': [ 'unstable' ] } } }
+                                            'features': [ 'unstable' ] },
+            '*vcpu-dirty-limit': 'uint64'} }
 
 ##
 # @migrate-set-parameters:
@@ -1155,6 +1163,9 @@
 # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration.
 #                             Defaults to 500ms. (Since 7.1)
 #
+# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration.
+#                    Defaults to 1. (Since 7.1)
+#
 # Features:
 # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period
 #            are experimental.
@@ -1191,7 +1202,8 @@
             '*multifd-zstd-level': 'uint8',
             '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
-                                            'features': [ 'unstable' ] } } }
+                                            'features': [ 'unstable' ] },
+            '*vcpu-dirty-limit': 'uint64'} }
 
 ##
 # @query-migrate-parameters:
Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH 2/8] qapi/migration: Introduce vcpu-dirty-limit parameters

Quoted 25 lines · On Sat, Jul 23, 2022 at 03:49:14PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.

"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.

This two parameters are used to help implement the dirty page rate limit algo of migration.

---
 migration/migration.c | 14 ++++++++++++++
 monitor/hmp-cmds.c    |  8 ++++++++
 qapi/migration.json   | 18 +++++++++++++++---
 3 files changed, 37 insertions(+), 3 deletions(-)
migration/migration.c +0 -0
index 7b19f85..ed1a47b 100644

migration/migration.c line 120

Quoted 4 lines
migration/migration.c +1 -0
@@ -117,6 +117,7 @@
 #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
 #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT            1       /* MB/s */

This default value also looks a bit weird.. why 1MB/s? Thanks,

Signature · 1 line
Peter Xu
author
#6 lore raw

Re: [PATCH 2/8] qapi/migration: Introduce vcpu-dirty-limit parameters

Quoted 28 lines · Peter Xu
Quoted 27 lines · On Sat, Jul 23, 2022 at 03:49:14PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.

"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.

This two parameters are used to help implement the dirty page rate limit algo of migration.

---
  migration/migration.c | 14 ++++++++++++++
  monitor/hmp-cmds.c    |  8 ++++++++
  qapi/migration.json   | 18 +++++++++++++++---
  3 files changed, 37 insertions(+), 3 deletions(-)
migration/migration.c +0 -0
index 7b19f85..ed1a47b 100644
@@ -117,6 +117,7 @@
  #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
Quoted 4 lines

migration/migration.c line 120

Quoted 2 lines
migration/migration.c +1 -0
  #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD     500     /* ms */
+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT            1       /* MB/s */

This default value also looks a bit weird.. why 1MB/s? Thanks,

Indeed, it seems kind of weired, the reason to set default dirty limit to 1MB/s is that we want to keep the dirty limit working until vcpu dirty page rate drop to 1MB/s once dirtylimit capability enabled during migration. In this way, migration has the largest chance to get converged before vcpu dirty page rate drop to 1MB/s。 If we set default dirty limit greater than 1MB/s, the probability of success for migration may be reduced, and the default behavior of migration is try the best to become sucessful.

Quoted 1 line
#7 lore raw

PATCH 3/8 migration: Introduce dirty-limit capability

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Introduce migration dirty-limit capability, which can be turned on before live migration and limit dirty page rate durty live migration.

Introduce migrate_dirty_limit function to help check if dirty-limit capability enabled during live migration.

Meanwhile, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcoded.

dirty-limit capability is kind of like auto-converge but using dirty limit instead of traditional cpu-throttle to throttle guest down. To enable this feature, turn on the dirty-limit capability before live migration using migratioin-set-capabilities, and set the parameters "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably to speed up convergence.

---
 migration/migration.c | 10 ++++++++++
 migration/migration.h |  1 +
 qapi/migration.json   |  4 +++-
 softmmu/dirtylimit.c  | 11 ++++++++++-
 4 files changed, 24 insertions(+), 2 deletions(-)
migration/migration.c +10 -0
index ed1a47b..84b592e 100644
@@ -2508,6 +2508,15 @@ bool migrate_auto_converge(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE];
 }
 
+bool migrate_dirty_limit(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT];
+}
+
 bool migrate_zero_blocks(void)
 {
     MigrationState *s;
@@ -4436,6 +4445,7 @@ static Property migration_properties[] = {
     DEFINE_PROP_MIG_CAP("x-zero-copy-send",
             MIGRATION_CAPABILITY_ZERO_COPY_SEND),
 #endif
+    DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT),
 
     DEFINE_PROP_END_OF_LIST(),
 };
migration/migration.h +1 -0
index cdad8ac..7fbb9f8 100644
@@ -409,6 +409,7 @@ bool migrate_ignore_shared(void);
 bool migrate_validate_uuid(void);
 
 bool migrate_auto_converge(void);
+bool migrate_dirty_limit(void);
 bool migrate_use_multifd(void);
 bool migrate_pause_before_switchover(void);
 int migrate_multifd_channels(void);
qapi/migration.json +3 -1
index 0963bab..39e5f5e 100644
@@ -477,6 +477,8 @@
 #                    will be handled faster.  This is a performance feature and
 #                    should not affect the correctness of postcopy migration.
 #                    (since 7.1)
+# @dirty-limit: Use dirty-limit to throttle down guest if enabled.
+#               (since 7.1)
 #
 # Features:
 # @unstable: Members @x-colo and @x-ignore-shared are experimental.
@@ -492,7 +494,7 @@
            'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
            { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
            'validate-uuid', 'background-snapshot',
-           'zero-copy-send', 'postcopy-preempt'] }
+           'zero-copy-send', 'postcopy-preempt', 'dirty-limit'] }
 
 ##
 # @MigrationCapabilityStatus:
softmmu/dirtylimit.c +10 -1
index 8d98cb7..1fdd8c6 100644
@@ -23,6 +23,8 @@
 #include "exec/memory.h"
 #include "hw/boards.h"
 #include "sysemu/kvm.h"
+#include "migration/misc.h"
+#include "migration/migration.h"
 #include "trace.h"
 
 /*
@@ -75,11 +77,18 @@ static bool dirtylimit_quit;
 
 static void vcpu_dirty_rate_stat_collect(void)
 {
+    MigrationState *s = migrate_get_current();
     VcpuStat stat;
     int i = 0;
+    int64_t period = DIRTYLIMIT_CALC_TIME_MS;
+
+    if (migrate_dirty_limit() &&
+        migration_is_active(s)) {
+        period = s->parameters.x_vcpu_dirty_limit_period;
+    }
 
     /* calculate vcpu dirtyrate */
-    vcpu_calculate_dirtyrate(DIRTYLIMIT_CALC_TIME_MS,
+    vcpu_calculate_dirtyrate(period,
                              &stat,
                              GLOBAL_DIRTY_LIMIT,
                              false);
Signature · 1 line
1.8.3.1
#8 lore raw

PATCH 4/8 migration: Implement dirty-limit convergence algo

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.

---
 migration/ram.c        | 53 +++++++++++++++++++++++++++++++++++++-------------
 migration/trace-events |  1 +
 2 files changed, 41 insertions(+), 13 deletions(-)
migration/ram.c +40 -13
index b94669b..2a5cd23 100644
@@ -45,6 +45,7 @@
 #include "qapi/error.h"
 #include "qapi/qapi-types-migration.h"
 #include "qapi/qapi-events-migration.h"
+#include "qapi/qapi-commands-migration.h"
 #include "qapi/qmp/qerror.h"
 #include "trace.h"
 #include "exec/ram_addr.h"
@@ -57,6 +58,8 @@
 #include "qemu/iov.h"
 #include "multifd.h"
 #include "sysemu/runstate.h"
+#include "sysemu/dirtylimit.h"
+#include "sysemu/kvm.h"
 
 #include "hw/boards.h" /* for machine_dump_guest_core() */
 
@@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time)
     }
 }
 
+/*
+ * Enable dirty-limit to throttle down the guest
+ */
+static void migration_dirty_limit_guest(void)
+{
+    if (!dirtylimit_in_service()) {
+        MigrationState *s = migrate_get_current();
+        int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit;
+
+        /* Set quota dirtyrate if dirty limit not in service */
+        qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL);
+        trace_migration_dirty_limit_guest(quota_dirtyrate);
+    }
+}
+
 static void migration_trigger_throttle(RAMState *rs)
 {
     MigrationState *s = migrate_get_current();
@@ -1148,22 +1166,31 @@ static void migration_trigger_throttle(RAMState *rs)
     uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE;
     uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold / 100;
 
-    /* During block migration the auto-converge logic incorrectly detects
-     * that ram migration makes no progress. Avoid this by disabling the
-     * throttling logic during the bulk phase of block migration. */
-    if (migrate_auto_converge() && !blk_mig_bulk_active()) {
-        /* The following detection logic can be refined later. For now:
-           Check to see if the ratio between dirtied bytes and the approx.
-           amount of bytes that just got transferred since the last time
-           we were in this routine reaches the threshold. If that happens
-           twice, start or increase throttling. */
-
-        if ((bytes_dirty_period > bytes_dirty_threshold) &&
-            (++rs->dirty_rate_high_cnt >= 2)) {
+    /*
+     * The following detection logic can be refined later. For now:
+     * Check to see if the ratio between dirtied bytes and the approx.
+     * amount of bytes that just got transferred since the last time
+     * we were in this routine reaches the threshold. If that happens
+     * twice, start or increase throttling.
+     */
+
+    if ((bytes_dirty_period > bytes_dirty_threshold) &&
+        (++rs->dirty_rate_high_cnt >= 2)) {
+        rs->dirty_rate_high_cnt = 0;
+        /*
+         * During block migration the auto-converge logic incorrectly detects
+         * that ram migration makes no progress. Avoid this by disabling the
+         * throttling logic during the bulk phase of block migration
+         */
+
+        if (migrate_auto_converge() && !blk_mig_bulk_active()) {
             trace_migration_throttle();
-            rs->dirty_rate_high_cnt = 0;
             mig_throttle_guest_down(bytes_dirty_period,
                                     bytes_dirty_threshold);
+        } else if (migrate_dirty_limit() &&
+                   kvm_dirty_ring_enabled() &&
+                   migration_is_active(s)) {
+            migration_dirty_limit_guest();
         }
     }
 }
migration/trace-events +1 -0
index a34afe7..3eb4b0d 100644
@@ -89,6 +89,7 @@ migration_bitmap_sync_start(void) ""
 migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64
 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx"
 migration_throttle(void) ""
+migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s"
 ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx"
 ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p"
 ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=%d addr=0x%" PRIx64 " flags=0x%x"
Signature · 1 line
1.8.3.1
#9 lore raw

Re: [PATCH 4/8] migration: Implement dirty-limit convergence algo

Quoted 41 lines · On Sat, Jul 23, 2022 at 03:49:16PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.

---
 migration/ram.c        | 53 +++++++++++++++++++++++++++++++++++++-------------
 migration/trace-events |  1 +
 2 files changed, 41 insertions(+), 13 deletions(-)
migration/ram.c +7 -0
index b94669b..2a5cd23 100644
@@ -45,6 +45,7 @@
 #include "qapi/error.h"
 #include "qapi/qapi-types-migration.h"
 #include "qapi/qapi-events-migration.h"
+#include "qapi/qapi-commands-migration.h"
 #include "qapi/qmp/qerror.h"
 #include "trace.h"
 #include "exec/ram_addr.h"
@@ -57,6 +58,8 @@
 #include "qemu/iov.h"
 #include "multifd.h"
 #include "sysemu/runstate.h"
+#include "sysemu/dirtylimit.h"
+#include "sysemu/kvm.h"
 
 #include "hw/boards.h" /* for machine_dump_guest_core() */
 
@@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time)
     }
 }
 
+/*
+ * Enable dirty-limit to throttle down the guest
+ */
+static void migration_dirty_limit_guest(void)

migration/ram.c static void migration_update_rates(RAMState *rs, int64_t end_time) · line 1158

Quoted 10 lines
migration/ram.c +10 -0
+{
+    if (!dirtylimit_in_service()) {
+        MigrationState *s = migrate_get_current();
+        int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit;
+
+        /* Set quota dirtyrate if dirty limit not in service */
+        qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL);
+        trace_migration_dirty_limit_guest(quota_dirtyrate);
+    }
+}

What if migration is cancelled? Do we have logic to stop the dirty limit, or should we?

Signature · 1 line
Peter Xu
author
#10 lore raw

Re: [PATCH 4/8] migration: Implement dirty-limit convergence algo

Quoted 47 lines · Peter Xu
Quoted 46 lines · On Sat, Jul 23, 2022 at 03:49:16PM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.

---
  migration/ram.c        | 53 +++++++++++++++++++++++++++++++++++++-------------
  migration/trace-events |  1 +
  2 files changed, 41 insertions(+), 13 deletions(-)
migration/ram.c +12 -0
index b94669b..2a5cd23 100644
@@ -45,6 +45,7 @@
  #include "qapi/error.h"
  #include "qapi/qapi-types-migration.h"
  #include "qapi/qapi-events-migration.h"
+#include "qapi/qapi-commands-migration.h"
  #include "qapi/qmp/qerror.h"
  #include "trace.h"
  #include "exec/ram_addr.h"
@@ -57,6 +58,8 @@
  #include "qemu/iov.h"
  #include "multifd.h"
  #include "sysemu/runstate.h"
+#include "sysemu/dirtylimit.h"
+#include "sysemu/kvm.h"
 
  #include "hw/boards.h" /* for machine_dump_guest_core() */
 
@@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time)
      }
  }
 
+/*
+ * Enable dirty-limit to throttle down the guest
+ */
+static void migration_dirty_limit_guest(void)
+{
+    if (!dirtylimit_in_service()) {
+        MigrationState *s = migrate_get_current();
+        int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit;
+
Quoted 8 lines

migration/ram.c static void migration_update_rates(RAMState *rs, int64_t end_time) · line 1158

Quoted 5 lines
migration/ram.c +5 -0
+        /* Set quota dirtyrate if dirty limit not in service */
+        qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL);
+        trace_migration_dirty_limit_guest(quota_dirtyrate);
+    }
+}

What if migration is cancelled? Do we have logic to stop the dirty limit, or should we?

Yes, we should have logic to stop dirty limit, i'll add that. Thanks for your suggestion. :)

Yong

Quoted 1 line
#11 lore raw

PATCH 5/8 migration: Export dirty-limit time info

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Export dirty limit throttle time and estimated ring full time, through which we can observe the process of dirty limit during live migration.

---
 include/sysemu/dirtylimit.h |  2 ++
 migration/migration.c       | 10 ++++++++++
 monitor/hmp-cmds.c          | 10 ++++++++++
 qapi/migration.json         | 10 +++++++++-
 softmmu/dirtylimit.c        | 22 ++++++++++++++++++++++
 5 files changed, 53 insertions(+), 1 deletion(-)
include/sysemu/dirtylimit.h +2 -0
index 8d2c1f3..98cc4a6 100644
@@ -34,4 +34,6 @@ void dirtylimit_set_vcpu(int cpu_index,
 void dirtylimit_set_all(uint64_t quota,
                         bool enable);
 void dirtylimit_vcpu_execute(CPUState *cpu);
+int64_t dirtylimit_throttle_us_per_full(void);
+int64_t dirtylimit_us_ring_full(void);
 #endif
migration/migration.c +10 -0
index 84b592e..81a46e2 100644
@@ -61,6 +61,7 @@
 #include "sysemu/cpus.h"
 #include "yank_functions.h"
 #include "sysemu/qtest.h"
+#include "sysemu/dirtylimit.h"
 
 #define MAX_THROTTLE  (128 << 20)      /* Migration transfer speed throttling */
 
@@ -1109,6 +1110,15 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
         info->ram->remaining = ram_bytes_remaining();
         info->ram->dirty_pages_rate = ram_counters.dirty_pages_rate;
     }
+
+    if (migrate_dirty_limit() && dirtylimit_in_service()) {
+        info->has_dirty_limit_throttle_us_per_full = true;
+        info->dirty_limit_throttle_us_per_full =
+                            dirtylimit_throttle_us_per_full();
+
+        info->has_dirty_limit_us_ring_full = true;
+        info->dirty_limit_us_ring_full = dirtylimit_us_ring_full();
+    }
 }
 
 static void populate_disk_info(MigrationInfo *info)
monitor/hmp-cmds.c +10 -0
index acbc5e8..accc869 100644
@@ -358,6 +358,16 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
                        info->cpu_throttle_percentage);
     }
 
+    if (info->has_dirty_limit_throttle_us_per_full) {
+        monitor_printf(mon, "dirty-limit throttle time: %" PRIu64 " us\n",
+                       info->dirty_limit_throttle_us_per_full);
+    }
+
+    if (info->has_dirty_limit_us_ring_full) {
+        monitor_printf(mon, "dirty-limit ring full time: %" PRIu64 " us\n",
+                       info->dirty_limit_us_ring_full);
+    }
+
     if (info->has_postcopy_blocktime) {
         monitor_printf(mon, "postcopy blocktime: %u\n",
                        info->postcopy_blocktime);
qapi/migration.json +9 -1
index 39e5f5e..6b8283f 100644
@@ -242,6 +242,12 @@
 #                   Present and non-empty when migration is blocked.
 #                   (since 6.0)
 #
+# @dirty-limit-throttle-us-per-full: Throttle time (us) during the period of
+#                                    dirty ring full (since 7.0)
+#
+# @dirty-limit-us-ring-full: Estimated periodic time (us) of dirty ring full.
+#                            (since 7.0)
+#
 # Since: 0.14
 ##
 { 'struct': 'MigrationInfo',
@@ -259,7 +265,9 @@
            '*postcopy-blocktime' : 'uint32',
            '*postcopy-vcpu-blocktime': ['uint32'],
            '*compression': 'CompressionStats',
-           '*socket-address': ['SocketAddress'] } }
+           '*socket-address': ['SocketAddress'],
+           '*dirty-limit-throttle-us-per-full': 'int64',
+           '*dirty-limit-us-ring-full': 'int64'} }
 
 ##
 # @query-migrate:
softmmu/dirtylimit.c +22 -0
index 1fdd8c6..1251b27 100644
@@ -546,6 +546,28 @@ static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index)
     return info;
 }
 
+/* Pick up first vcpu throttle time by default */
+int64_t dirtylimit_throttle_us_per_full(void)
+{
+    CPUState *cpu = first_cpu;
+    return cpu->throttle_us_per_full;
+}
+
+/*
+ * Estimate dirty ring full time under current dirty page rate.
+ * Return -1 if guest doesn't dirty memory.
+ */
+int64_t dirtylimit_us_ring_full(void)
+{
+    uint64_t curr_rate = vcpu_dirty_rate_get(0);
+
+    if (!curr_rate) {
+        return -1;
+    }
+
+    return dirtylimit_dirty_ring_full_time(curr_rate);
+}
+
 static struct DirtyLimitInfoList *dirtylimit_query_all(void)
 {
     int i, index;
Signature · 1 line
1.8.3.1
#12 lore raw

PATCH 6/8 tests: Add migration dirty-limit capability test

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Add migration dirty-limit capability test if kernel support dirty ring.

Migration dirty-limit capability introduce dirty limit capability, two parameters: x-vcpu-dirty-limit-period and vcpu-dirty-limit are introduced to implement the live migration with dirty limit.

The test case enable the capability and set the corresponding parameters to test migration. When migration switch to pre-switchover phase, like the auto-converge, checking if migration satisfy the convergence condition.

---
 tests/qtest/migration-test.c | 92 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
tests/qtest/migration-test.c +92 -0
index 71595a7..88503a1 100644
@@ -2412,6 +2412,96 @@ static void test_vcpu_dirty_limit(void)
     dirtylimit_stop_vm(vm);
 }
 
+static void test_migrate_dirty_limit(void)
+{
+    g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
+    QTestState *from, *to;
+    int64_t remaining, throttle_us_per_full;
+    /*
+     * We want the test to be stable and as fast as possible.
+     * E.g., with 1Gb/s bandwith migration may pass without dirty limit,
+     * so we need to decrease a bandwidth.
+     */
+    const int64_t dirtylimit_period = 1000, dirtylimit_value = 50;
+    const int64_t max_bandwidth = 400000000; /* ~400Mb/s */
+    const int64_t downtime_limit = 250; /* 250ms */
+    /*
+     * We migrate through unix-socket (> 500Mb/s).
+     * Thus, expected migration speed ~= bandwidth limit (< 500Mb/s).
+     * So, we can predict expected_threshold
+     */
+    const int64_t expected_threshold = max_bandwidth * downtime_limit / 1000;
+    MigrateCommon args = {
+        .start = {
+            .use_dirty_ring = true,
+        },
+        .listen_uri = uri,
+        .connect_uri = uri,
+    };
+
+    if (test_migrate_start(&from, &to, args.listen_uri, &args.start)) {
+        return;
+    }
+
+    migrate_set_capability(from, "dirty-limit", true);
+    migrate_set_parameter_int(from, "x-vcpu-dirty-limit-period",
+                              dirtylimit_period);
+    migrate_set_parameter_int(from, "vcpu-dirty-limit", dirtylimit_value);
+
+    /*
+     * Set the initial parameters so that the migration could not converge
+     * without dirty limit.
+     */
+    migrate_set_parameter_int(from, "downtime-limit", 1);
+    migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */
+
+    /* To check limit rate after precopy */
+    migrate_set_capability(from, "pause-before-switchover", true);
+
+    /* Wait for the first serial output from the source */
+    wait_for_serial("src_serial");
+
+    migrate_qmp(from, uri, "{}");
+
+    /* Wait for dirty limit begins */
+    throttle_us_per_full = 0;
+    while (throttle_us_per_full == 0) {
+        throttle_us_per_full =
+            read_migrate_property_int(from, "dirty-limit-throttle-us-per-full");
+        usleep(100);
+        g_assert_false(got_stop);
+    }
+
+    /*
+     * The dirty limit rate should equals the return value of
+     * query-vcpu-dirty-limit if dirty limit cap set
+     */
+    g_assert_cmpint(dirtylimit_value, ==, get_limit_rate(from));
+
+    /* Now, when we tested if dirty limit works, let it converge */
+    migrate_set_parameter_int(from, "downtime-limit", downtime_limit);
+    migrate_set_parameter_int(from, "max-bandwidth", max_bandwidth);
+
+    /*
+     * Wait for pre-switchover status to check if migration
+     * satisfy the convergence condition
+     */
+    wait_for_migration_status(from, "pre-switchover", NULL);
+
+    remaining = read_ram_property_int(from, "remaining");
+    g_assert_cmpint(remaining, <,
+                    (expected_threshold + expected_threshold / 100));
+
+    migrate_continue(from, "pre-switchover");
+
+    qtest_qmp_eventwait(to, "RESUME");
+
+    wait_for_serial("dest_serial");
+    wait_for_migration_complete(from);
+
+    test_migrate_end(from, to, true);
+}
+
 static bool kvm_dirty_ring_supported(void)
 {
 #if defined(__linux__) && defined(HOST_X86_64)
@@ -2577,6 +2667,8 @@ int main(int argc, char **argv)
                        test_precopy_unix_dirty_ring);
         qtest_add_func("/migration/vcpu_dirty_limit",
                        test_vcpu_dirty_limit);
+        qtest_add_func("/migration/dirty_limit",
+                       test_migrate_dirty_limit);
     }
 
     ret = g_test_run();
Signature · 1 line
1.8.3.1
#13 lore raw

PATCH 7/8 tests/migration: Introduce dirty-ring-size option into guestperf

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Guestperf tool does not enable diry ring feature when test migration by default.

To support dirty ring migration performance test, introduce dirty-ring-size option into guestperf tools, which ranges in [1024, 65536].

To set dirty ring size with 4096 during migration test:

$ ./tests/migration/guestperf.py --dirty-ring-size 4096 xxx
---
 tests/migration/guestperf/engine.py   | 7 ++++++-
 tests/migration/guestperf/hardware.py | 8 ++++++--
 tests/migration/guestperf/shell.py    | 7 ++++++-
 3 files changed, 18 insertions(+), 4 deletions(-)
tests/migration/guestperf/engine.py +6 -1
index 87a6ab2..2b98f00 100644
@@ -304,7 +304,6 @@ def _get_common_args(self, hardware, tunnelled=False):
             cmdline = "'" + cmdline + "'"
 
         argv = [
-            "-accel", "kvm",
             "-cpu", "host",
             "-kernel", self._kernel,
             "-initrd", self._initrd,
@@ -315,6 +314,12 @@ def _get_common_args(self, hardware, tunnelled=False):
             "-smp", str(hardware._cpus),
         ]
 
+        if hardware._dirty_ring_size:
+            argv.extend(["-accel", "kvm,dirty-ring-size=%s" %
+                         hardware._dirty_ring_size])
+        else:
+            argv.extend(["-accel", "kvm"])
+
         if self._debug:
             argv.extend(["-device", "sga"])
 
tests/migration/guestperf/hardware.py +6 -2
index 3145785..f779cc0 100644
@@ -23,7 +23,8 @@ def __init__(self, cpus=1, mem=1,
                  src_cpu_bind=None, src_mem_bind=None,
                  dst_cpu_bind=None, dst_mem_bind=None,
                  prealloc_pages = False,
-                 huge_pages=False, locked_pages=False):
+                 huge_pages=False, locked_pages=False,
+                 dirty_ring_size=0):
         self._cpus = cpus
         self._mem = mem # GiB
         self._src_mem_bind = src_mem_bind # List of NUMA nodes
@@ -33,6 +34,7 @@ def __init__(self, cpus=1, mem=1,
         self._prealloc_pages = prealloc_pages
         self._huge_pages = huge_pages
         self._locked_pages = locked_pages
+        self._dirty_ring_size = dirty_ring_size
 
 
     def serialize(self):
@@ -46,6 +48,7 @@ def serialize(self):
             "prealloc_pages": self._prealloc_pages,
             "huge_pages": self._huge_pages,
             "locked_pages": self._locked_pages,
+            "dirty_ring_size": self._dirty_ring_size,
         }
 
     @classmethod
@@ -59,4 +62,5 @@ def deserialize(cls, data):
             data["dst_mem_bind"],
             data["prealloc_pages"],
             data["huge_pages"],
-            data["locked_pages"])
+            data["locked_pages"],
+            data["dirty_ring_size"])
tests/migration/guestperf/shell.py +6 -1
index 8a809e3..559616f 100644
@@ -60,6 +60,8 @@ def __init__(self):
         parser.add_argument("--prealloc-pages", dest="prealloc_pages", default=False)
         parser.add_argument("--huge-pages", dest="huge_pages", default=False)
         parser.add_argument("--locked-pages", dest="locked_pages", default=False)
+        parser.add_argument("--dirty-ring-size", dest="dirty_ring_size",
+                            default=0, type=int)
 
         self._parser = parser
 
@@ -89,7 +91,10 @@ def split_map(value):
 
                         locked_pages=args.locked_pages,
                         huge_pages=args.huge_pages,
-                        prealloc_pages=args.prealloc_pages)
+                        prealloc_pages=args.prealloc_pages,
+
+                        dirty_ring_size=args.dirty_ring_size)
+
 
 
 class Shell(BaseShell):
Signature · 1 line
1.8.3.1
#14 lore raw

PATCH 8/8 tests/migration: Introduce dirty-limit into guestperf

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Guestperf tool does not cover the dirty-limit migration currently, support this feature.

To enable dirty-limit, setting x-vcpu-dirty-limit-period as 500ms and vcpu-dirty-limit as 10MB/s:

$ ./tests/migration/guestperf.py \
    --dirty-limit --x-vcpu-dirty-limit-period 500 \
    --dirty-limit --vcpu-dirty-limit 10 \
    --output output.json

To run the entire standardized set of dirty-limit-enabled comparisons, with unix migration:

$ ./tests/migration/guestperf-batch.py \
    --dst-host localhost --transport unix \
    --filter compr-dirty-limit-period* --output outputdir
---
 tests/migration/guestperf/comparison.py | 14 ++++++++++++++
 tests/migration/guestperf/engine.py     | 26 ++++++++++++++++++++++++++
 tests/migration/guestperf/progress.py   | 17 +++++++++++++++--
 tests/migration/guestperf/scenario.py   | 11 ++++++++++-
 tests/migration/guestperf/shell.py      | 18 +++++++++++++++++-
 5 files changed, 82 insertions(+), 4 deletions(-)
tests/migration/guestperf/comparison.py +14 -0
index c03b3f6..ccc0db9 100644
@@ -135,4 +135,18 @@ def __init__(self, name, scenarios):
         Scenario("compr-multifd-channels-64",
                  multifd=True, multifd_channels=64),
     ]),
+
+
+    # Looking at effect of dirty-limit with
+    # varying x_vcpu_dirty_limit_period
+    Comparison("compr-dirty-limit", scenarios = [
+        Scenario("compr-dirty-limit-period-100",
+                 dirty_limit=True, x_vcpu_dirty_limit_period=100),
+        Scenario("compr-dirty-limit-period-500",
+                 dirty_limit=True, x_vcpu_dirty_limit_period=500),
+        Scenario("compr-dirty-limit-period-1000",
+                 dirty_limit=True, x_vcpu_dirty_limit_period=1000),
+    ]),
+
+
 ]
tests/migration/guestperf/engine.py +26 -0
index 2b98f00..2f29471 100644
@@ -103,6 +103,8 @@ def _migrate_progress(self, vm):
             info.get("expected-downtime", 0),
             info.get("setup-time", 0),
             info.get("cpu-throttle-percentage", 0),
+            info.get("dirty-limit-throttle-us-per-full", 0),
+            info.get("dirty-limit-us-ring-full", 0),
         )
 
     def _migrate(self, hardware, scenario, src, dst, connect_uri):
@@ -204,6 +206,30 @@ def _migrate(self, hardware, scenario, src, dst, connect_uri):
             resp = dst.command("migrate-set-parameters",
                                multifd_channels=scenario._multifd_channels)
 
+        if scenario._dirty_limit:
+            if not hardware._dirty_ring_size:
+                raise Exception("dirty ring size must be configured when "
+                                "testing dirty limit migration")
+
+            resp = src.command("migrate-set-capabilities",
+                               capabilities = [
+                                   { "capability": "dirty-limit",
+                                     "state": True }
+                               ])
+            resp = src.command("migrate-set-parameters",
+                x_vcpu_dirty_limit_period=scenario._x_vcpu_dirty_limit_period)
+            resp = src.command("migrate-set-parameters",
+                               vcpu_dirty_limit=scenario._vcpu_dirty_limit)
+            resp = dst.command("migrate-set-capabilities",
+                               capabilities = [
+                                   { "capability": "dirty-limit",
+                                     "state": True }
+                               ])
+            resp = dst.command("migrate-set-parameters",
+                x_vcpu_dirty_limit_period=scenario._x_vcpu_dirty_limit_period)
+            resp = dst.command("migrate-set-parameters",
+                               vcpu_dirty_limit=scenario._vcpu_dirty_limit)
+
         resp = src.command("migrate", uri=connect_uri)
 
         post_copy = False
tests/migration/guestperf/progress.py +15 -2
index ab1ee57..dd5d86b 100644
@@ -81,7 +81,9 @@ def __init__(self,
                  downtime,
                  downtime_expected,
                  setup_time,
-                 throttle_pcent):
+                 throttle_pcent,
+                 dirty_limit_throttle_us_per_full,
+                 dirty_limit_us_ring_full):
 
         self._status = status
         self._ram = ram
@@ -91,6 +93,11 @@ def __init__(self,
         self._downtime_expected = downtime_expected
         self._setup_time = setup_time
         self._throttle_pcent = throttle_pcent
+        self._dirty_limit_throttle_us_per_full =
+            dirty_limit_throttle_us_per_full
+        self._dirty_limit_us_ring_full =
+            dirty_limit_us_ring_full
+
 
     def serialize(self):
         return {
@@ -102,6 +109,10 @@ def serialize(self):
             "downtime_expected": self._downtime_expected,
             "setup_time": self._setup_time,
             "throttle_pcent": self._throttle_pcent,
+            "dirty_limit_throttle_time_per_full":
+                self._dirty_limit_throttle_us_per_full,
+            "dirty_limit_ring_full_time":
+                self._dirty_limit_us_ring_full,
         }
 
     @classmethod
@@ -114,4 +125,6 @@ def deserialize(cls, data):
             data["downtime"],
             data["downtime_expected"],
             data["setup_time"],
-            data["throttle_pcent"])
+            data["throttle_pcent"],
+            data["dirty_limit_throttle_time_per_full"],
+            data["dirty_limit_ring_full_time"])
tests/migration/guestperf/scenario.py +10 -1
index de70d9b..154c4f5 100644
@@ -30,7 +30,9 @@ def __init__(self, name,
                  auto_converge=False, auto_converge_step=10,
                  compression_mt=False, compression_mt_threads=1,
                  compression_xbzrle=False, compression_xbzrle_cache=10,
-                 multifd=False, multifd_channels=2):
+                 multifd=False, multifd_channels=2,
+                 dirty_limit=False, x_vcpu_dirty_limit_period=500,
+                 vcpu_dirty_limit=1):
 
         self._name = name
 
@@ -60,6 +62,10 @@ def __init__(self, name,
         self._multifd = multifd
         self._multifd_channels = multifd_channels
 
+        self._dirty_limit = dirty_limit
+        self._x_vcpu_dirty_limit_period = x_vcpu_dirty_limit_period
+        self._vcpu_dirty_limit = vcpu_dirty_limit
+
     def serialize(self):
         return {
             "name": self._name,
@@ -79,6 +85,9 @@ def serialize(self):
             "compression_xbzrle_cache": self._compression_xbzrle_cache,
             "multifd": self._multifd,
             "multifd_channels": self._multifd_channels,
+            "dirty_limit": self._dirty_limit,
+            "x_vcpu_dirty_limit_period": self._x_vcpu_dirty_limit_period,
+            "vcpu_dirty_limit": self._vcpu_dirty_limit,
         }
 
     @classmethod
tests/migration/guestperf/shell.py +17 -1
index 559616f..23fe895 100644
@@ -132,6 +132,17 @@ def __init__(self):
         parser.add_argument("--multifd-channels", dest="multifd_channels",
                             default=2, type=int)
 
+        parser.add_argument("--dirty-limit", dest="dirty_limit", default=False,
+                            action="store_true")
+
+        parser.add_argument("--x-vcpu-dirty-limit-period",
+                            dest="x_vcpu_dirty_limit_period",
+                            default=500, type=int)
+
+        parser.add_argument("--vcpu-dirty-limit",
+                            dest="vcpu_dirty_limit",
+                            default=1, type=int)
+
     def get_scenario(self, args):
         return Scenario(name="perfreport",
                         downtime=args.downtime,
@@ -155,7 +166,12 @@ def get_scenario(self, args):
                         compression_xbzrle_cache=args.compression_xbzrle_cache,
 
                         multifd=args.multifd,
-                        multifd_channels=args.multifd_channels)
+                        multifd_channels=args.multifd_channels,
+
+                        dirty_limit=args.dirty_limit,
+                        x_vcpu_dirty_limit_period=
+                            args.x_vcpu_dirty_limit_period,
+                        vcpu_dirty_limit=args.vcpu_dirty_limit)
 
     def run(self, argv):
         args = self._parser.parse_args(argv)
Signature · 1 line
1.8.3.1
author
#15 lore raw
In reply to [email protected]

Ping.
How about this series? hoping to get comments if anyone has played
with it.

Thanks !

Hyman

Quoted 106 lines · 在 2022/7/23 15:49, [email protected] 写道:

From: Hyman Huang(黄勇) <[email protected]>

Abstract
========

This series added a new migration capability called "dirtylimit". It can be enabled when dirty ring is enabled, and it'll improve the vCPU performance during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/[email protected]/

As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of migration can make the read-process not be penalized. This series wires up the vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup of dirtylimit during live migration.

To validate the implementation, i tested a 32 vCPU vm live migration with such model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}

The following are results:

host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s

   |---------------------+--------+------------+---------------|
   | UnixBench test item | Normal | Dirtylimit | Auto-converge |
   |---------------------+--------+------------+---------------|
   | dhry2reg            | 32800  | 32786      | 25292         |
   | whetstone-double    | 10326  | 10315      | 9847          |
   | pipe                | 15442  | 15271      | 14506         |
   | context1            | 7260   | 6235       | 4514          |
   | spawn               | 3663   | 3317       | 3249          |
   | syscall             | 4669   | 4667       | 3841          |
   |---------------------+--------+------------+---------------|

From the data above we can draw a conclusion that vcpus that do not dirty memory
in vm are almost unaffected during the dirtylimit migration, but the auto converge way does.

I also tested the total time of dirtylimit migration with variable dirty memory size in vm.

senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s

   |-----------------------+----------------+-------------------|
   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
   |-----------------------+----------------+-------------------|
   | 60                    | 2014           | 2131              |
   | 70                    | 5381           | 12590             |
   | 90                    | 6037           | 33545             |
   | 110                   | 7660           | [*]               |
   |-----------------------+----------------+-------------------|

[*]: This case means migration is not convergent.

senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s

   |-----------------------+----------------+-------------------|
   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
   |-----------------------+----------------+-------------------|
   | 1600                  | 15842          | 27548             |
   | 2000                  | 19026          | 38447             |
   | 2400                  | 19897          | 46381             |
   | 2800                  | 22338          | 57149             |
   |-----------------------+----------------+-------------------|

Above data shows that dirtylimit way of migration can also reduce the total time of migration and it achieves convergence more easily in some case.

In addition to implement dirtylimit capability itself, this series add 3 tests for migration, aiming at playing around for developer simply:
1. qtest for dirty limit migration
2. support dirty ring way of migration for guestperf tool
3. support dirty limit migration for guestperf tool

Please review, thanks !

Hyman Huang (8):
qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters
migration: Introduce dirty-limit capability
migration: Implement dirty-limit convergence algo
migration: Export dirty-limit time info
tests: Add migration dirty-limit capability test tests/migration: Introduce dirty-ring-size option into guestperf tests/migration: Introduce dirty-limit into guestperf

  include/sysemu/dirtylimit.h             |  2 +
  migration/migration.c                   | 50 ++++++++++++++++++
  migration/migration.h                   |  1 +
  migration/ram.c                         | 53 ++++++++++++++-----
  migration/trace-events                  |  1 +
  monitor/hmp-cmds.c                      | 26 ++++++++++
  qapi/migration.json                     | 57 ++++++++++++++++----
  softmmu/dirtylimit.c                    | 33 +++++++++++-
  tests/migration/guestperf/comparison.py | 14 +++++
  tests/migration/guestperf/engine.py     | 33 +++++++++++-
  tests/migration/guestperf/hardware.py   |  8 ++-
  tests/migration/guestperf/progress.py   | 17 +++++-
  tests/migration/guestperf/scenario.py   | 11 +++-
  tests/migration/guestperf/shell.py      | 25 ++++++++-
  tests/qtest/migration-test.c            | 92 +++++++++++++++++++++++++++++++++
  15 files changed, 392 insertions(+), 31 deletions(-)