migration: introduce dirtylimit capability
cover migration: introduce dirtylimit capability
From: Hyman Huang(黄勇) <[email protected]>
Abstract
========
This series added a new migration capability called "dirtylimit". It can be enabled when dirty ring is enabled, and it'll improve the vCPU performance during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/[email protected]/
As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of migration can make the read-process not be penalized. This series wires up the vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup of dirtylimit during live migration.
To validate the implementation, i tested a 32 vCPU vm live migration with such model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
The following are results:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
|---------------------+--------+------------+---------------| | UnixBench test item | Normal | Dirtylimit | Auto-converge | |---------------------+--------+------------+---------------| | dhry2reg | 32800 | 32786 | 25292 | | whetstone-double | 10326 | 10315 | 9847 | | pipe | 15442 | 15271 | 14506 | | context1 | 7260 | 6235 | 4514 | | spawn | 3663 | 3317 | 3249 | | syscall | 4669 | 4667 | 3841 | |---------------------+--------+------------+---------------|
From the data above we can draw a conclusion that vcpus that do not dirty memory in vm are almost unaffected during the dirtylimit migration, but the auto converge way does.
I also tested the total time of dirtylimit migration with variable dirty memory size in vm.
senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 60 | 2014 | 2131 | | 70 | 5381 | 12590 | | 90 | 6037 | 33545 | | 110 | 7660 | [*] | |-----------------------+----------------+-------------------|
[*]: This case means migration is not convergent.
senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s
|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 1600 | 15842 | 27548 | | 2000 | 19026 | 38447 | | 2400 | 19897 | 46381 | | 2800 | 22338 | 57149 | |-----------------------+----------------+-------------------|
Above data shows that dirtylimit way of migration can also reduce the total time of migration and it achieves convergence more easily in some case.
In addition to implement dirtylimit capability itself, this series add 3 tests for migration, aiming at playing around for developer simply:
1. qtest for dirty limit migration
2. support dirty ring way of migration for guestperf tool
3. support dirty limit migration for guestperf tool
Please review, thanks !
Hyman Huang (8):
qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters
migration: Introduce dirty-limit capability
migration: Implement dirty-limit convergence algo
migration: Export dirty-limit time info
tests: Add migration dirty-limit capability test tests/migration: Introduce dirty-ring-size option into guestperf tests/migration: Introduce dirty-limit into guestperf
include/sysemu/dirtylimit.h | 2 + migration/migration.c | 50 ++++++++++++++++++ migration/migration.h | 1 + migration/ram.c | 53 ++++++++++++++----- migration/trace-events | 1 + monitor/hmp-cmds.c | 26 ++++++++++ qapi/migration.json | 57 ++++++++++++++++---- softmmu/dirtylimit.c | 33 +++++++++++- tests/migration/guestperf/comparison.py | 14 +++++ tests/migration/guestperf/engine.py | 33 +++++++++++- tests/migration/guestperf/hardware.py | 8 ++- tests/migration/guestperf/progress.py | 17 +++++- tests/migration/guestperf/scenario.py | 11 +++- tests/migration/guestperf/shell.py | 25 ++++++++- tests/qtest/migration-test.c | 92 +++++++++++++++++++++++++++++++++ 15 files changed, 392 insertions(+), 31 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/8 qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
From: Hyman Huang(黄勇) <[email protected]>
Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.
--- migration/migration.c | 16 ++++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 31 ++++++++++++++++++++++++------- 3 files changed, 48 insertions(+), 7 deletions(-)
migration/migration.c +16 -0
@@ -116,6 +116,8 @@#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */ +static NotifierList migration_state_notifiers =NOTIFIER_LIST_INITIALIZER (migration_state_notifiers);
@@ -962,6 +964,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) s->parameters.block_bitmap_mapping); } + params->has_x_vcpu_dirty_limit_period = true; + params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; +return params; }
@@ -1662,6 +1667,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->has_block_bitmap_mapping = true; dest->block_bitmap_mapping = params->block_bitmap_mapping; } + +if (params->has_x_vcpu_dirty_limit_period) { + dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1784,6 +1793,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)QAPI_CLONE (BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } +if (params->has_x_vcpu_dirty_limit_period) { + s->parameters.x_vcpu_dirty_limit_period = + params->x_vcpu_dirty_limit_period; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
@@ -4384,6 +4397,9 @@ static Property migration_properties[] = {DEFINE_PROP_STRING ("tls-creds" , MigrationState, parameters.tls_creds),DEFINE_PROP_STRING ("tls-hostname" , MigrationState, parameters.tls_hostname),DEFINE_PROP_STRING ("tls-authz" , MigrationState, parameters.tls_authz), +DEFINE_PROP_UINT64 ("x-vcpu-dirty-limit-period" , MigrationState, + parameters.x_vcpu_dirty_limit_period, +DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD ),/* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
monitor/hmp-cmds.c +8 -0
@@ -532,6 +532,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) } } } + +monitor_printf (mon,"%s: %" PRIu64" ms\n" , +MigrationParameter_str (MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD ), + params->x_vcpu_dirty_limit_period); }qapi_free_MigrationParameters (params);
@@ -1351,6 +1355,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)error_setg (&err,"The block-bitmap-mapping parameter can only be set " "through QMP" );break ; +case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD : + p->has_x_vcpu_dirty_limit_period = true; +visit_type_size (v, param, &p->x_vcpu_dirty_limit_period, &err); +break ;default :assert (0 ); }
qapi/migration.json +24 -7
@@ -776,8 +776,12 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. +# Defaults to 500ms. (Since 7.1) +# # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ##
@@ -795,8 +799,9 @@ 'multifd-channels', 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', - 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'multifd-zlib-level', 'multifd-zstd-level', + 'block-bitmap-mapping', + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } ## # @MigrateSetParameters:
@@ -941,8 +946,12 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. +# Defaults to 500ms. (Since 7.1) +# # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ##
@@ -976,7 +985,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @migrate-set-parameters:
@@ -1141,8 +1152,12 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. +# Defaults to 500ms. (Since 7.1) +# # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ##
@@ -1174,7 +1189,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @query-migrate-parameters:
Signature · 1 line
1.8.3.1
Re: [PATCH 1/8] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
Quoted 18 lines · On Sat, Jul 23, 2022 at 03:49:13PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/migration.c | 16 ++++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 31 ++++++++++++++++++++++++------- 3 files changed, 48 insertions(+), 7 deletions(-)migration/migration.c +0
-0
migration/migration.c line 119
Quoted 4 lines
migration/migration.c +1
-0@@ -116,6 +116,8 @@#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */
Why 500 but not DIRTYLIMIT_CALC_TIME_MS?
Is it intended to make this parameter experimental, but the other one not?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 1/8] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
Quoted 22 lines · Peter Xu
Quoted 21 lines · On Sat, Jul 23, 2022 at 03:49:13PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is used to make dirtyrate calculation period configurable.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/migration.c | 16 ++++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 31 ++++++++++++++++++++++++------- 3 files changed, 48 insertions(+), 7 deletions(-)migration/migration.c +0
-0@@ -116,6 +116,8 @@#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100
Quoted 3 lines
migration/migration.c line 119
Quoted 1 line
migration/migration.c +1
-0+#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */ Why 500 but not
DIRTYLIMIT_CALC_TIME_MS?
This is a empirical value actually, the iteration time of migration is less than 1000ms normally. In my test it varies from 200ms to 500ms, we assume iteration time is 500ms and calculation period is 1000ms, so 2 iteration pass when 1 dirty page rate get calculated. We want calculation period as close to iteration time as possible so that 1 iteration pass, 1 new dirty page rate be calculated and get compared, hoping the dirtylimit working more precisely.
But as the "x-" prefix implies, i'm a little unsure that if the solution works。
Quoted 2 lines
Is it intended to make this parameter experimental, but the other one not?
Since i'm not very sure vcpu-dirty-limit-period have impact on migration(as described above), so it is made experimental. As to vcpu-dirty-limit, it indeed have impact on migration in theory, so it is not made experimental. But from another point of view, 2 parameter are introduced in the first time and none of them suffer lots of tests, it is also reasonable to make 2 parameter experimental, i'm not insist that.
Yong
Quoted 3 lines
Thanks,
PATCH 2/8 qapi/migration: Introduce vcpu-dirty-limit parameters
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.
"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.
This two parameters are used to help implement the dirty page rate limit algo of migration.
--- migration/migration.c | 14 ++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 37 insertions(+), 3 deletions(-)
migration/migration.c +14 -0
@@ -117,6 +117,7 @@#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1/* MB/s */ static NotifierList migration_state_notifiers =NOTIFIER_LIST_INITIALIZER (migration_state_notifiers);
@@ -967,6 +968,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_x_vcpu_dirty_limit_period = true; params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; + params->has_vcpu_dirty_limit = true; + params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit; +return params; }
@@ -1671,6 +1675,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_x_vcpu_dirty_limit_period) { dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } + +if (params->has_vcpu_dirty_limit) { + dest->vcpu_dirty_limit = params->vcpu_dirty_limit; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1797,6 +1805,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) s->parameters.x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } +if (params->has_vcpu_dirty_limit) { + s->parameters.vcpu_dirty_limit = params->vcpu_dirty_limit; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
@@ -4400,6 +4411,9 @@ static Property migration_properties[] = {DEFINE_PROP_UINT64 ("x-vcpu-dirty-limit-period" , MigrationState, parameters.x_vcpu_dirty_limit_period,DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD ), +DEFINE_PROP_UINT64 ("vcpu-dirty-limit" , MigrationState, + parameters.vcpu_dirty_limit, +DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT ),/* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
monitor/hmp-cmds.c +8 -0
@@ -536,6 +536,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %" PRIu64" ms\n" ,MigrationParameter_str (MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD ), params->x_vcpu_dirty_limit_period); + +monitor_printf (mon,"%s: %" PRIu64" MB/s\n" , +MigrationParameter_str (MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT ), + params->vcpu_dirty_limit); }qapi_free_MigrationParameters (params);
@@ -1359,6 +1363,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_x_vcpu_dirty_limit_period = true;visit_type_size (v, param, &p->x_vcpu_dirty_limit_period, &err);break ; +case MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT : + p->has_vcpu_dirty_limit = true; +visit_type_size (v, param, &p->vcpu_dirty_limit, &err); +break ;default :assert (0 ); }
qapi/migration.json +15 -3
@@ -779,6 +779,9 @@ # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. # Defaults to 500ms. (Since 7.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.1) +# # Features: # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental.
@@ -801,7 +804,8 @@ 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', 'block-bitmap-mapping', - { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, + 'vcpu-dirty-limit'] } ## # @MigrateSetParameters:
@@ -949,6 +953,9 @@ # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. # Defaults to 500ms. (Since 7.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.1) +# # Features: # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental.
@@ -987,7 +994,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @migrate-set-parameters:
@@ -1155,6 +1163,9 @@ # @x-vcpu-dirty-limit-period: Periodic time (ms) of dirty limit during live migration. # Defaults to 500ms. (Since 7.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.1) +# # Features: # @unstable: Member @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental.
@@ -1191,7 +1202,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @query-migrate-parameters:
Signature · 1 line
1.8.3.1
Re: [PATCH 2/8] qapi/migration: Introduce vcpu-dirty-limit parameters
Quoted 25 lines · On Sat, Jul 23, 2022 at 03:49:14PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.
"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.
This two parameters are used to help implement the dirty page rate limit algo of migration.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/migration.c | 14 ++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 37 insertions(+), 3 deletions(-)migration/migration.c +0
-0
migration/migration.c line 120
Quoted 4 lines
migration/migration.c +1
-0@@ -117,6 +117,7 @@#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1/* MB/s */
This default value also looks a bit weird.. why 1MB/s? Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/8] qapi/migration: Introduce vcpu-dirty-limit parameters
Quoted 28 lines · Peter Xu
Quoted 27 lines · On Sat, Jul 23, 2022 at 03:49:14PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration.
"vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.
This two parameters are used to help implement the dirty page rate limit algo of migration.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/migration.c | 14 ++++++++++++++ monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 37 insertions(+), 3 deletions(-)migration/migration.c +0
-0@@ -117,6 +117,7 @@#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100
Quoted 4 lines
migration/migration.c line 120
Quoted 2 lines
migration/migration.c +1
-0#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 500/* ms */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1/* MB/s */ This default value also looks a bit weird.. why 1MB/s? Thanks,
Indeed, it seems kind of weired, the reason to set default dirty limit to 1MB/s is that we want to keep the dirty limit working until vcpu dirty page rate drop to 1MB/s once dirtylimit capability enabled during migration. In this way, migration has the largest chance to get converged before vcpu dirty page rate drop to 1MB/s。 If we set default dirty limit greater than 1MB/s, the probability of success for migration may be reduced, and the default behavior of migration is try the best to become sucessful.
Quoted 1 line
PATCH 3/8 migration: Introduce dirty-limit capability
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Introduce migration dirty-limit capability, which can be turned on before live migration and limit dirty page rate durty live migration.
Introduce migrate_dirty_limit function to help check if dirty-limit capability enabled during live migration.
Meanwhile, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcoded.
dirty-limit capability is kind of like auto-converge but using dirty limit instead of traditional cpu-throttle to throttle guest down. To enable this feature, turn on the dirty-limit capability before live migration using migratioin-set-capabilities, and set the parameters "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably to speed up convergence.
--- migration/migration.c | 10 ++++++++++ migration/migration.h | 1 + qapi/migration.json | 4 +++- softmmu/dirtylimit.c | 11 ++++++++++- 4 files changed, 24 insertions(+), 2 deletions(-)
migration/migration.c +10 -0
@@ -2508,6 +2508,15 @@ bool migrate_auto_converge(void)return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE ]; } +bool migrate_dirty_limit (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT ]; +} +bool migrate_zero_blocks (void ) {MigrationState *s;
@@ -4436,6 +4445,7 @@ static Property migration_properties[] = {DEFINE_PROP_MIG_CAP ("x-zero-copy-send" ,MIGRATION_CAPABILITY_ZERO_COPY_SEND ), #endif +DEFINE_PROP_MIG_CAP ("x-dirty-limit" ,MIGRATION_CAPABILITY_DIRTY_LIMIT ), DEFINE_PROP_END_OF_LIST(), };
migration/migration.h +1 -0
@@ -409,6 +409,7 @@ bool migrate_ignore_shared(void);bool migrate_validate_uuid (void );bool migrate_auto_converge (void ); +bool migrate_dirty_limit (void );bool migrate_use_multifd (void );bool migrate_pause_before_switchover (void );int migrate_multifd_channels (void );
qapi/migration.json +3 -1
@@ -477,6 +477,8 @@ # will be handled faster. This is a performance feature and # should not affect the correctness of postcopy migration. # (since 7.1) +# @dirty-limit: Use dirty-limit to throttle down guest if enabled. +# (since 7.1) # # Features: # @unstable: Members @x-colo and @x-ignore-shared are experimental.
@@ -492,7 +494,7 @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt'] } + 'zero-copy-send', 'postcopy-preempt', 'dirty-limit'] } ## # @MigrationCapabilityStatus:
softmmu/dirtylimit.c +10 -1
@@ -23,6 +23,8 @@#include "exec/memory.h" #include "hw/boards.h" #include "sysemu/kvm.h" +#include "migration/misc.h" +#include "migration/migration.h" #include "trace.h" /*
@@ -75,11 +77,18 @@ static bool dirtylimit_quit;static void vcpu_dirty_rate_stat_collect (void ) { +MigrationState *s =migrate_get_current ();VcpuStat stat;int i =0 ; +int64_t period =DIRTYLIMIT_CALC_TIME_MS ; + +if (migrate_dirty_limit () && +migration_is_active (s)) { + period = s->parameters.x_vcpu_dirty_limit_period; + }/* calculate vcpu dirtyrate */ -vcpu_calculate_dirtyrate (DIRTYLIMIT_CALC_TIME_MS , +vcpu_calculate_dirtyrate (period, &stat,GLOBAL_DIRTY_LIMIT , false);
Signature · 1 line
1.8.3.1
PATCH 4/8 migration: Implement dirty-limit convergence algo
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.
--- migration/ram.c | 53 +++++++++++++++++++++++++++++++++++++------------- migration/trace-events | 1 + 2 files changed, 41 insertions(+), 13 deletions(-)
migration/ram.c +40 -13
@@ -45,6 +45,7 @@#include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h"
@@ -57,6 +58,8 @@#include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */
@@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* +* Enable dirty-limit to throttle down the guest +*/ +static void migration_dirty_limit_guest (void ) +{ +if (!dirtylimit_in_service ()) { +MigrationState *s =migrate_get_current (); +int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit; + +/* Set quota dirtyrate if dirty limit not in service */ +qmp_set_vcpu_dirty_limit (false,-1 , quota_dirtyrate,NULL ); +trace_migration_dirty_limit_guest (quota_dirtyrate); + } +} +static void migration_trigger_throttle (RAMState *rs) {MigrationState *s =migrate_get_current ();
@@ -1148,22 +1166,31 @@ static void migration_trigger_throttle(RAMState *rs)uint64_t bytes_dirty_period = rs->num_dirty_pages_period *TARGET_PAGE_SIZE ;uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold /100 ; -/* During block migration the auto-converge logic incorrectly detects -* that ram migration makes no progress. Avoid this by disabling the -* throttling logic during the bulk phase of block migration. */ -if (migrate_auto_converge () && !blk_mig_bulk_active ()) { -/* The following detection logic can be refined later. For now: -Check to see if the ratio between dirtied bytes and the approx. -amount of bytes that just got transferred since the last time -we were in this routine reaches the threshold. If that happens -twice, start or increase throttling. */ - -if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >=2 )) { +/* +* The following detection logic can be refined later. For now: +* Check to see if the ratio between dirtied bytes and the approx. +* amount of bytes that just got transferred since the last time +* we were in this routine reaches the threshold. If that happens +* twice, start or increase throttling. +*/ + +if ((bytes_dirty_period > bytes_dirty_threshold) && + (++rs->dirty_rate_high_cnt >=2 )) { + rs->dirty_rate_high_cnt =0 ; +/* +* During block migration the auto-converge logic incorrectly detects +* that ram migration makes no progress. Avoid this by disabling the +* throttling logic during the bulk phase of block migration +*/ + +if (migrate_auto_converge () && !blk_mig_bulk_active ()) {trace_migration_throttle (); - rs->dirty_rate_high_cnt =0 ;mig_throttle_guest_down (bytes_dirty_period, bytes_dirty_threshold); + }else if (migrate_dirty_limit () && +kvm_dirty_ring_enabled () && +migration_is_active (s)) { +migration_dirty_limit_guest (); } } }
migration/trace-events +1 -0
@@ -89,6 +89,7 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" migration_throttle(void) "" +migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=%d addr=0x%" PRIx64 " flags=0x%x"
Signature · 1 line
1.8.3.1
Re: [PATCH 4/8] migration: Implement dirty-limit convergence algo
Quoted 41 lines · On Sat, Jul 23, 2022 at 03:49:16PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/ram.c | 53 +++++++++++++++++++++++++++++++++++++------------- migration/trace-events | 1 + 2 files changed, 41 insertions(+), 13 deletions(-)migration/ram.c +7
-0@@ -45,6 +45,7 @@#include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h" @@ -57,6 +58,8 @@#include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */ @@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* +* Enable dirty-limit to throttle down the guest +*/ +static void migration_dirty_limit_guest (void )
migration/ram.c static void migration_update_rates(RAMState *rs, int64_t end_time) · line 1158
Quoted 10 lines
migration/ram.c +10
-0+{ +if (!dirtylimit_in_service ()) { +MigrationState *s =migrate_get_current (); +int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit; + +/* Set quota dirtyrate if dirty limit not in service */ +qmp_set_vcpu_dirty_limit (false,-1 , quota_dirtyrate,NULL ); +trace_migration_dirty_limit_guest (quota_dirtyrate); + } +}
What if migration is cancelled? Do we have logic to stop the dirty limit, or should we?
Signature · 1 line
Peter Xu
Re: [PATCH 4/8] migration: Implement dirty-limit convergence algo
Quoted 47 lines · Peter Xu
Quoted 46 lines · On Sat, Jul 23, 2022 at 03:49:16PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent.
Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/ram.c | 53 +++++++++++++++++++++++++++++++++++++------------- migration/trace-events | 1 + 2 files changed, 41 insertions(+), 13 deletions(-)migration/ram.c +12
-0@@ -45,6 +45,7 @@#include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h" @@ -57,6 +58,8 @@#include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */ @@ -1139,6 +1142,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* +* Enable dirty-limit to throttle down the guest +*/ +static void migration_dirty_limit_guest (void ) +{ +if (!dirtylimit_in_service ()) { +MigrationState *s =migrate_get_current (); +int64_t quota_dirtyrate = s->parameters.vcpu_dirty_limit; +
Quoted 8 lines
migration/ram.c static void migration_update_rates(RAMState *rs, int64_t end_time) · line 1158
Quoted 5 lines
migration/ram.c +5
-0+/* Set quota dirtyrate if dirty limit not in service */ +qmp_set_vcpu_dirty_limit (false,-1 , quota_dirtyrate,NULL ); +trace_migration_dirty_limit_guest (quota_dirtyrate); + } +}What if migration is cancelled? Do we have logic to stop the dirty limit, or should we?
Yes, we should have logic to stop dirty limit, i'll add that. Thanks for your suggestion. :)
Yong
Quoted 1 line
PATCH 5/8 migration: Export dirty-limit time info
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Export dirty limit throttle time and estimated ring full time, through which we can observe the process of dirty limit during live migration.
--- include/sysemu/dirtylimit.h | 2 ++ migration/migration.c | 10 ++++++++++ monitor/hmp-cmds.c | 10 ++++++++++ qapi/migration.json | 10 +++++++++- softmmu/dirtylimit.c | 22 ++++++++++++++++++++++ 5 files changed, 53 insertions(+), 1 deletion(-)
include/sysemu/dirtylimit.h +2 -0
@@ -34,4 +34,6 @@ void dirtylimit_set_vcpu(int cpu_index,void dirtylimit_set_all (uint64_t quota,bool enable);void dirtylimit_vcpu_execute (CPUState *cpu); +int64_t dirtylimit_throttle_us_per_full (void ); +int64_t dirtylimit_us_ring_full (void );#endif
migration/migration.c +10 -0
@@ -61,6 +61,7 @@#include "sysemu/cpus.h" #include "yank_functions.h" #include "sysemu/qtest.h" +#include "sysemu/dirtylimit.h" #define MAX_THROTTLE (128 << 20)/* Migration transfer speed throttling */
@@ -1109,6 +1110,15 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->remaining =ram_bytes_remaining (); info->ram->dirty_pages_rate = ram_counters.dirty_pages_rate; } + +if (migrate_dirty_limit () &&dirtylimit_in_service ()) { + info->has_dirty_limit_throttle_us_per_full = true; + info->dirty_limit_throttle_us_per_full = +dirtylimit_throttle_us_per_full (); + + info->has_dirty_limit_us_ring_full = true; + info->dirty_limit_us_ring_full =dirtylimit_us_ring_full (); + } }static void populate_disk_info (MigrationInfo *info)
monitor/hmp-cmds.c +10 -0
@@ -358,6 +358,16 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->cpu_throttle_percentage); } +if (info->has_dirty_limit_throttle_us_per_full) { +monitor_printf (mon,"dirty-limit throttle time: %" PRIu64" us\n" , + info->dirty_limit_throttle_us_per_full); + } + +if (info->has_dirty_limit_us_ring_full) { +monitor_printf (mon,"dirty-limit ring full time: %" PRIu64" us\n" , + info->dirty_limit_us_ring_full); + } +if (info->has_postcopy_blocktime) {monitor_printf (mon,"postcopy blocktime: %u\n" , info->postcopy_blocktime);
qapi/migration.json +9 -1
@@ -242,6 +242,12 @@ # Present and non-empty when migration is blocked. # (since 6.0) # +# @dirty-limit-throttle-us-per-full: Throttle time (us) during the period of +# dirty ring full (since 7.0) +# +# @dirty-limit-us-ring-full: Estimated periodic time (us) of dirty ring full. +# (since 7.0) +# # Since: 0.14 ## { 'struct': 'MigrationInfo',
@@ -259,7 +265,9 @@ '*postcopy-blocktime' : 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], '*compression': 'CompressionStats', - '*socket-address': ['SocketAddress'] } } + '*socket-address': ['SocketAddress'], + '*dirty-limit-throttle-us-per-full': 'int64', + '*dirty-limit-us-ring-full': 'int64'} } ## # @query-migrate:
softmmu/dirtylimit.c +22 -0
@@ -546,6 +546,28 @@ static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index)return info; } +/* Pick up first vcpu throttle time by default */ +int64_t dirtylimit_throttle_us_per_full (void ) +{ +CPUState *cpu = first_cpu; +return cpu->throttle_us_per_full; +} + +/* +* Estimate dirty ring full time under current dirty page rate. +* Return -1 if guest doesn't dirty memory. +*/ +int64_t dirtylimit_us_ring_full (void ) +{ +uint64_t curr_rate =vcpu_dirty_rate_get (0 ); + +if (!curr_rate) { +return -1 ; + } + +return dirtylimit_dirty_ring_full_time (curr_rate); +} +static struct DirtyLimitInfoList *dirtylimit_query_all (void ) {int i, index;
Signature · 1 line
1.8.3.1
PATCH 6/8 tests: Add migration dirty-limit capability test
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Add migration dirty-limit capability test if kernel support dirty ring.
Migration dirty-limit capability introduce dirty limit capability, two parameters: x-vcpu-dirty-limit-period and vcpu-dirty-limit are introduced to implement the live migration with dirty limit.
The test case enable the capability and set the corresponding parameters to test migration. When migration switch to pre-switchover phase, like the auto-converge, checking if migration satisfy the convergence condition.
--- tests/qtest/migration-test.c | 92 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+)
tests/qtest/migration-test.c +92 -0
@@ -2412,6 +2412,96 @@ static void test_vcpu_dirty_limit(void)dirtylimit_stop_vm (vm); } +static void test_migrate_dirty_limit (void ) +{ + g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); +QTestState *from, *to; +int64_t remaining, throttle_us_per_full; +/* +* We want the test to be stable and as fast as possible. +* E.g., with 1Gb/s bandwith migration may pass without dirty limit, +* so we need to decrease a bandwidth. +*/ +const int64_t dirtylimit_period =1000 , dirtylimit_value =50 ; +const int64_t max_bandwidth =400000000 ;/* ~400Mb/s */ +const int64_t downtime_limit =250 ;/* 250ms */ +/* +* We migrate through unix-socket (> 500Mb/s). +* Thus, expected migration speed ~= bandwidth limit (< 500Mb/s). +* So, we can predict expected_threshold +*/ +const int64_t expected_threshold = max_bandwidth * downtime_limit /1000 ; +MigrateCommon args = { + .start = { + .use_dirty_ring = true, + }, + .listen_uri = uri, + .connect_uri = uri, + }; + +if (test_migrate_start (&from, &to, args.listen_uri, &args.start)) { +return ; + } + +migrate_set_capability (from,"dirty-limit" , true); +migrate_set_parameter_int (from,"x-vcpu-dirty-limit-period" , + dirtylimit_period); +migrate_set_parameter_int (from,"vcpu-dirty-limit" , dirtylimit_value); + +/* +* Set the initial parameters so that the migration could not converge +* without dirty limit. +*/ +migrate_set_parameter_int (from,"downtime-limit" ,1 ); +migrate_set_parameter_int (from,"max-bandwidth" ,100000000 );/* ~100Mb/s */ + +/* To check limit rate after precopy */ +migrate_set_capability (from,"pause-before-switchover" , true); + +/* Wait for the first serial output from the source */ +wait_for_serial ("src_serial" ); + +migrate_qmp (from, uri,"{}" ); + +/* Wait for dirty limit begins */ + throttle_us_per_full =0 ; +while (throttle_us_per_full ==0 ) { + throttle_us_per_full = +read_migrate_property_int (from,"dirty-limit-throttle-us-per-full" ); +usleep (100 ); +g_assert_false (got_stop); + } + +/* +* The dirty limit rate should equals the return value of +* query-vcpu-dirty-limit if dirty limit cap set +*/ + g_assert_cmpint(dirtylimit_value, ==, get_limit_rate(from)); + +/* Now, when we tested if dirty limit works, let it converge */ +migrate_set_parameter_int (from,"downtime-limit" , downtime_limit); +migrate_set_parameter_int (from,"max-bandwidth" , max_bandwidth); + +/* +* Wait for pre-switchover status to check if migration +* satisfy the convergence condition +*/ +wait_for_migration_status (from,"pre-switchover" ,NULL ); + + remaining =read_ram_property_int (from,"remaining" ); + g_assert_cmpint(remaining, <, + (expected_threshold + expected_threshold /100 )); + +migrate_continue (from,"pre-switchover" ); + +qtest_qmp_eventwait (to,"RESUME" ); + +wait_for_serial ("dest_serial" ); +wait_for_migration_complete (from); + +test_migrate_end (from, to, true); +} +static bool kvm_dirty_ring_supported (void ) {#if defined(__linux__) && defined(HOST_X86_64 )
@@ -2577,6 +2667,8 @@ int main(int argc, char **argv) test_precopy_unix_dirty_ring);qtest_add_func ("/migration/vcpu_dirty_limit" , test_vcpu_dirty_limit); +qtest_add_func ("/migration/dirty_limit" , + test_migrate_dirty_limit); } ret =g_test_run ();
Signature · 1 line
1.8.3.1
PATCH 7/8 tests/migration: Introduce dirty-ring-size option into guestperf
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Guestperf tool does not enable diry ring feature when test migration by default.
To support dirty ring migration performance test, introduce dirty-ring-size option into guestperf tools, which ranges in [1024, 65536].
To set dirty ring size with 4096 during migration test:
$ ./tests/migration/guestperf.py --dirty-ring-size 4096 xxx
--- tests/migration/guestperf/engine.py | 7 ++++++- tests/migration/guestperf/hardware.py | 8 ++++++-- tests/migration/guestperf/shell.py | 7 ++++++- 3 files changed, 18 insertions(+), 4 deletions(-)
tests/migration/guestperf/engine.py +6 -1
@@ -304,7 +304,6 @@ def _get_common_args(self, hardware, tunnelled=False): cmdline = "'" + cmdline + "'" argv = [ - "-accel", "kvm", "-cpu", "host", "-kernel", self._kernel, "-initrd", self._initrd,
@@ -315,6 +314,12 @@ def _get_common_args(self, hardware, tunnelled=False): "-smp", str(hardware._cpus), ] + if hardware._dirty_ring_size: + argv.extend(["-accel", "kvm,dirty-ring-size=%s" % + hardware._dirty_ring_size]) + else: + argv.extend(["-accel", "kvm"]) + if self._debug: argv.extend(["-device", "sga"])
tests/migration/guestperf/hardware.py +6 -2
@@ -23,7 +23,8 @@ def __init__(self, cpus=1, mem=1, src_cpu_bind=None, src_mem_bind=None, dst_cpu_bind=None, dst_mem_bind=None, prealloc_pages = False, - huge_pages=False, locked_pages=False): + huge_pages=False, locked_pages=False, + dirty_ring_size=0): self._cpus = cpus self._mem = mem # GiB self._src_mem_bind = src_mem_bind # List of NUMA nodes
@@ -33,6 +34,7 @@ def __init__(self, cpus=1, mem=1, self._prealloc_pages = prealloc_pages self._huge_pages = huge_pages self._locked_pages = locked_pages + self._dirty_ring_size = dirty_ring_size def serialize(self):
@@ -46,6 +48,7 @@ def serialize(self): "prealloc_pages": self._prealloc_pages, "huge_pages": self._huge_pages, "locked_pages": self._locked_pages, + "dirty_ring_size": self._dirty_ring_size, } @classmethod
@@ -59,4 +62,5 @@ def deserialize(cls, data): data["dst_mem_bind"], data["prealloc_pages"], data["huge_pages"], - data["locked_pages"]) + data["locked_pages"], + data["dirty_ring_size"])
tests/migration/guestperf/shell.py +6 -1
@@ -60,6 +60,8 @@ def __init__(self): parser.add_argument("--prealloc-pages", dest="prealloc_pages", default=False) parser.add_argument("--huge-pages", dest="huge_pages", default=False) parser.add_argument("--locked-pages", dest="locked_pages", default=False) + parser.add_argument("--dirty-ring-size", dest="dirty_ring_size", + default=0, type=int) self._parser = parser
@@ -89,7 +91,10 @@ def split_map(value): locked_pages=args.locked_pages, huge_pages=args.huge_pages, - prealloc_pages=args.prealloc_pages) + prealloc_pages=args.prealloc_pages, + + dirty_ring_size=args.dirty_ring_size) + class Shell(BaseShell):
Signature · 1 line
1.8.3.1
PATCH 8/8 tests/migration: Introduce dirty-limit into guestperf
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Guestperf tool does not cover the dirty-limit migration currently, support this feature.
To enable dirty-limit, setting x-vcpu-dirty-limit-period as 500ms and vcpu-dirty-limit as 10MB/s:
$ ./tests/migration/guestperf.py \
--dirty-limit --x-vcpu-dirty-limit-period 500 \
--dirty-limit --vcpu-dirty-limit 10 \
--output output.jsonTo run the entire standardized set of dirty-limit-enabled comparisons, with unix migration:
$ ./tests/migration/guestperf-batch.py \
--dst-host localhost --transport unix \
--filter compr-dirty-limit-period* --output outputdir--- tests/migration/guestperf/comparison.py | 14 ++++++++++++++ tests/migration/guestperf/engine.py | 26 ++++++++++++++++++++++++++ tests/migration/guestperf/progress.py | 17 +++++++++++++++-- tests/migration/guestperf/scenario.py | 11 ++++++++++- tests/migration/guestperf/shell.py | 18 +++++++++++++++++- 5 files changed, 82 insertions(+), 4 deletions(-)
tests/migration/guestperf/comparison.py +14 -0
@@ -135,4 +135,18 @@ def __init__(self, name, scenarios): Scenario("compr-multifd-channels-64", multifd=True, multifd_channels=64), ]), + + + # Looking at effect of dirty-limit with + # varying x_vcpu_dirty_limit_period + Comparison("compr-dirty-limit", scenarios = [ + Scenario("compr-dirty-limit-period-100", + dirty_limit=True, x_vcpu_dirty_limit_period=100), + Scenario("compr-dirty-limit-period-500", + dirty_limit=True, x_vcpu_dirty_limit_period=500), + Scenario("compr-dirty-limit-period-1000", + dirty_limit=True, x_vcpu_dirty_limit_period=1000), + ]), + + ]
tests/migration/guestperf/engine.py +26 -0
@@ -103,6 +103,8 @@ def _migrate_progress(self, vm): info.get("expected-downtime", 0), info.get("setup-time", 0), info.get("cpu-throttle-percentage", 0), + info.get("dirty-limit-throttle-us-per-full", 0), + info.get("dirty-limit-us-ring-full", 0), ) def _migrate(self, hardware, scenario, src, dst, connect_uri):
@@ -204,6 +206,30 @@ def _migrate(self, hardware, scenario, src, dst, connect_uri): resp = dst.command("migrate-set-parameters", multifd_channels=scenario._multifd_channels) + if scenario._dirty_limit: + if not hardware._dirty_ring_size: + raise Exception("dirty ring size must be configured when " + "testing dirty limit migration") + + resp = src.command("migrate-set-capabilities", + capabilities = [ + { "capability": "dirty-limit", + "state": True } + ]) + resp = src.command("migrate-set-parameters", + x_vcpu_dirty_limit_period=scenario._x_vcpu_dirty_limit_period) + resp = src.command("migrate-set-parameters", + vcpu_dirty_limit=scenario._vcpu_dirty_limit) + resp = dst.command("migrate-set-capabilities", + capabilities = [ + { "capability": "dirty-limit", + "state": True } + ]) + resp = dst.command("migrate-set-parameters", + x_vcpu_dirty_limit_period=scenario._x_vcpu_dirty_limit_period) + resp = dst.command("migrate-set-parameters", + vcpu_dirty_limit=scenario._vcpu_dirty_limit) + resp = src.command("migrate", uri=connect_uri) post_copy = False
tests/migration/guestperf/progress.py +15 -2
@@ -81,7 +81,9 @@ def __init__(self, downtime, downtime_expected, setup_time, - throttle_pcent): + throttle_pcent, + dirty_limit_throttle_us_per_full, + dirty_limit_us_ring_full): self._status = status self._ram = ram
@@ -91,6 +93,11 @@ def __init__(self, self._downtime_expected = downtime_expected self._setup_time = setup_time self._throttle_pcent = throttle_pcent + self._dirty_limit_throttle_us_per_full = + dirty_limit_throttle_us_per_full + self._dirty_limit_us_ring_full = + dirty_limit_us_ring_full + def serialize(self): return {
@@ -102,6 +109,10 @@ def serialize(self): "downtime_expected": self._downtime_expected, "setup_time": self._setup_time, "throttle_pcent": self._throttle_pcent, + "dirty_limit_throttle_time_per_full": + self._dirty_limit_throttle_us_per_full, + "dirty_limit_ring_full_time": + self._dirty_limit_us_ring_full, } @classmethod
@@ -114,4 +125,6 @@ def deserialize(cls, data): data["downtime"], data["downtime_expected"], data["setup_time"], - data["throttle_pcent"]) + data["throttle_pcent"], + data["dirty_limit_throttle_time_per_full"], + data["dirty_limit_ring_full_time"])
tests/migration/guestperf/scenario.py +10 -1
@@ -30,7 +30,9 @@ def __init__(self, name, auto_converge=False, auto_converge_step=10, compression_mt=False, compression_mt_threads=1, compression_xbzrle=False, compression_xbzrle_cache=10, - multifd=False, multifd_channels=2): + multifd=False, multifd_channels=2, + dirty_limit=False, x_vcpu_dirty_limit_period=500, + vcpu_dirty_limit=1): self._name = name
@@ -60,6 +62,10 @@ def __init__(self, name, self._multifd = multifd self._multifd_channels = multifd_channels + self._dirty_limit = dirty_limit + self._x_vcpu_dirty_limit_period = x_vcpu_dirty_limit_period + self._vcpu_dirty_limit = vcpu_dirty_limit + def serialize(self): return { "name": self._name,
@@ -79,6 +85,9 @@ def serialize(self): "compression_xbzrle_cache": self._compression_xbzrle_cache, "multifd": self._multifd, "multifd_channels": self._multifd_channels, + "dirty_limit": self._dirty_limit, + "x_vcpu_dirty_limit_period": self._x_vcpu_dirty_limit_period, + "vcpu_dirty_limit": self._vcpu_dirty_limit, } @classmethod
tests/migration/guestperf/shell.py +17 -1
@@ -132,6 +132,17 @@ def __init__(self): parser.add_argument("--multifd-channels", dest="multifd_channels", default=2, type=int) + parser.add_argument("--dirty-limit", dest="dirty_limit", default=False, + action="store_true") + + parser.add_argument("--x-vcpu-dirty-limit-period", + dest="x_vcpu_dirty_limit_period", + default=500, type=int) + + parser.add_argument("--vcpu-dirty-limit", + dest="vcpu_dirty_limit", + default=1, type=int) + def get_scenario(self, args): return Scenario(name="perfreport", downtime=args.downtime,
@@ -155,7 +166,12 @@ def get_scenario(self, args): compression_xbzrle_cache=args.compression_xbzrle_cache, multifd=args.multifd, - multifd_channels=args.multifd_channels) + multifd_channels=args.multifd_channels, + + dirty_limit=args.dirty_limit, + x_vcpu_dirty_limit_period= + args.x_vcpu_dirty_limit_period, + vcpu_dirty_limit=args.vcpu_dirty_limit) def run(self, argv): args = self._parser.parse_args(argv)
Signature · 1 line
1.8.3.1
Ping.
How about this series? hoping to get comments if anyone has played
with it.
Thanks !
Hyman
Quoted 106 lines · 在 2022/7/23 15:49, [email protected] 写道:
From: Hyman Huang(黄勇) <[email protected]>
Abstract
========This series added a new migration capability called "dirtylimit". It can be enabled when dirty ring is enabled, and it'll improve the vCPU performance during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/[email protected]/As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of migration can make the read-process not be penalized. This series wires up the vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup of dirtylimit during live migration.
To validate the implementation, i tested a 32 vCPU vm live migration with such model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}The following are results:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s|---------------------+--------+------------+---------------| | UnixBench test item | Normal | Dirtylimit | Auto-converge | |---------------------+--------+------------+---------------| | dhry2reg | 32800 | 32786 | 25292 | | whetstone-double | 10326 | 10315 | 9847 | | pipe | 15442 | 15271 | 14506 | | context1 | 7260 | 6235 | 4514 | | spawn | 3663 | 3317 | 3249 | | syscall | 4669 | 4667 | 3841 | |---------------------+--------+------------+---------------|From the data above we can draw a conclusion that vcpus that do not dirty memory
in vm are almost unaffected during the dirtylimit migration, but the auto converge way does.I also tested the total time of dirtylimit migration with variable dirty memory size in vm.
senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 60 | 2014 | 2131 | | 70 | 5381 | 12590 | | 90 | 6037 | 33545 | | 110 | 7660 | [*] | |-----------------------+----------------+-------------------|[*]: This case means migration is not convergent.
senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 1600 | 15842 | 27548 | | 2000 | 19026 | 38447 | | 2400 | 19897 | 46381 | | 2800 | 22338 | 57149 | |-----------------------+----------------+-------------------|Above data shows that dirtylimit way of migration can also reduce the total time of migration and it achieves convergence more easily in some case.
In addition to implement dirtylimit capability itself, this series add 3 tests for migration, aiming at playing around for developer simply:
1. qtest for dirty limit migration
2. support dirty ring way of migration for guestperf tool
3. support dirty limit migration for guestperf toolPlease review, thanks !
Hyman Huang (8):
qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters
migration: Introduce dirty-limit capability
migration: Implement dirty-limit convergence algo
migration: Export dirty-limit time info
tests: Add migration dirty-limit capability test tests/migration: Introduce dirty-ring-size option into guestperf tests/migration: Introduce dirty-limit into guestperfinclude/sysemu/dirtylimit.h | 2 + migration/migration.c | 50 ++++++++++++++++++ migration/migration.h | 1 + migration/ram.c | 53 ++++++++++++++----- migration/trace-events | 1 + monitor/hmp-cmds.c | 26 ++++++++++ qapi/migration.json | 57 ++++++++++++++++---- softmmu/dirtylimit.c | 33 +++++++++++- tests/migration/guestperf/comparison.py | 14 +++++ tests/migration/guestperf/engine.py | 33 +++++++++++- tests/migration/guestperf/hardware.py | 8 ++- tests/migration/guestperf/progress.py | 17 +++++- tests/migration/guestperf/scenario.py | 11 +++- tests/migration/guestperf/shell.py | 25 ++++++++- tests/qtest/migration-test.c | 92 +++++++++++++++++++++++++++++++++ 15 files changed, 392 insertions(+), 31 deletions(-)