migration: introduce dirtylimit capability
cover migration: introduce dirtylimit capability
From: Hyman Huang(黄勇) <[email protected]>
Abstract
========
This series added a new migration capability called "dirtylimit". It can be enabled when dirty ring is enabled, and it'll improve the vCPU performance during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/[email protected]/
As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of migration can make the read-process not be penalized. This series wires up the vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup of dirtylimit during live migration.
To validate the implementation, i tested a 32 vCPU vm live migration with such model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
The following are results:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
|---------------------+--------+------------+---------------| | UnixBench test item | Normal | Dirtylimit | Auto-converge | |---------------------+--------+------------+---------------| | dhry2reg | 32800 | 32786 | 25292 | | whetstone-double | 10326 | 10315 | 9847 | | pipe | 15442 | 15271 | 14506 | | context1 | 7260 | 6235 | 4514 | | spawn | 3663 | 3317 | 3249 | | syscall | 4669 | 4667 | 3841 | |---------------------+--------+------------+---------------|
From the data above we can draw a conclusion that vcpus that do not dirty memory in vm are almost unaffected during the dirtylimit migration, but the auto converge way does.
I also tested the total time of dirtylimit migration with variable dirty memory size in vm.
senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 60 | 2014 | 2131 | | 70 | 5381 | 12590 | | 90 | 6037 | 33545 | | 110 | 7660 | [*] | |-----------------------+----------------+-------------------|
[*]: This case means migration is not convergent.
senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s
|-----------------------+----------------+-------------------| | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) | |-----------------------+----------------+-------------------| | 1600 | 15842 | 27548 | | 2000 | 19026 | 38447 | | 2400 | 19897 | 46381 | | 2800 | 22338 | 57149 | |-----------------------+----------------+-------------------|
Above data shows that dirtylimit way of migration can also reduce the total time of migration and it achieves convergence more easily in some case.
Since the dependent patchset has not been merged yet, i post the series in the the following link if anyone want to have a try, once the depenent patchset is ready and merged, the dirtylimit can be availiable by patching series:
https://github.com/newfriday/qemu/tree/dirtylimit_cap_v1
This series is for RFC, comments and reviews about the API, code logic, algo implementation or anything else are welcomed and appreciated
Please review, thanks !
Yong
Hyman Huang (6):
qapi/migration: Introduce vcpu-dirtylimit-period parameters qapi/migration: Introduce vcpu-dirtylimit parameters
migration: Implement dirtylimit convergence algo
migration: Introduce dirtylimit capability
migration: Add dirtylimit data into migration info
tests: Add migration dirtylimit capability test
include/sysemu/dirtylimit.h | 2 + migration/migration.c | 48 ++++++++++++++++++++++++ migration/migration.h | 1 + migration/ram.c | 53 +++++++++++++++++++------- migration/trace-events | 1 + monitor/hmp-cmds.c | 15 ++++++++ qapi/migration.json | 47 ++++++++++++++++++++--- softmmu/dirtylimit.c | 33 +++++++++++++++- tests/qtest/migration-test.c | 89 ++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 269 insertions(+), 20 deletions(-)
Signature · 1 line
1.8.3.1
RFC 1/6 qapi/migration: Introduce vcpu-dirtylimit-period parameters
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirtylimit-period" migration parameters, which is used to makes dirtyrate calculation period configurable.
To implement that, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcode.
Meanwhile, introduce migrate_dirtylimit function to help check if dirtylimit enabled during live migration, set it false by default.
--- migration/migration.c | 20 ++++++++++++++++++++ migration/migration.h | 1 + monitor/hmp-cmds.c | 8 ++++++++ qapi/migration.json | 19 +++++++++++++++---- softmmu/dirtylimit.c | 11 ++++++++++- 5 files changed, 54 insertions(+), 5 deletions(-)
migration/migration.c +20 -0
@@ -116,6 +116,8 @@#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 +#define DEFAULT_MIGRATE_VCPU_DIRTYLIMIT_PERIOD 500/* ms */ +static NotifierList migration_state_notifiers =NOTIFIER_LIST_INITIALIZER (migration_state_notifiers);
@@ -921,6 +923,9 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) s->parameters.block_bitmap_mapping); } + params->has_vcpu_dirtylimit_period = true; + params->vcpu_dirtylimit_period = s->parameters.vcpu_dirtylimit_period; +return params; }
@@ -1581,6 +1586,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->has_block_bitmap_mapping = true; dest->block_bitmap_mapping = params->block_bitmap_mapping; } + +if (params->has_vcpu_dirtylimit_period) { + dest->vcpu_dirtylimit_period = params->vcpu_dirtylimit_period; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1703,6 +1712,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)QAPI_CLONE (BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } +if (params->has_vcpu_dirtylimit_period) { + s->parameters.vcpu_dirtylimit_period = params->vcpu_dirtylimit_period; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
@@ -2403,6 +2415,11 @@ bool migrate_auto_converge(void)return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE ]; } +bool migrate_dirtylimit (void ) +{ +return false; +} +bool migrate_zero_blocks (void ) {MigrationState *s;
@@ -4221,6 +4238,9 @@ static Property migration_properties[] = {DEFINE_PROP_SIZE ("announce-step" , MigrationState, parameters.announce_step,DEFAULT_MIGRATE_ANNOUNCE_STEP ), +DEFINE_PROP_UINT64 ("vcpu-dirtylimit-period" , MigrationState, + parameters.vcpu_dirtylimit_period, +DEFAULT_MIGRATE_VCPU_DIRTYLIMIT_PERIOD ),/* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
migration/migration.h +1 -0
@@ -369,6 +369,7 @@ bool migrate_ignore_shared(void);bool migrate_validate_uuid (void );bool migrate_auto_converge (void ); +bool migrate_dirtylimit (void );bool migrate_use_multifd (void );bool migrate_pause_before_switchover (void );int migrate_multifd_channels (void );
monitor/hmp-cmds.c +8 -0
@@ -525,6 +525,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) } } } + +monitor_printf (mon,"%s: %" PRIu64" MB/s\n" , +MigrationParameter_str (MIGRATION_PARAMETER_VCPU_DIRTYLIMIT_PERIOD ), + params->vcpu_dirtylimit_period); }qapi_free_MigrationParameters (params);
@@ -1344,6 +1348,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)error_setg (&err,"The block-bitmap-mapping parameter can only be set " "through QMP" );break ; +case MIGRATION_PARAMETER_VCPU_DIRTYLIMIT_PERIOD : + p->has_vcpu_dirtylimit_period = true; +visit_type_size (v, param, &p->vcpu_dirtylimit_period, &err); +break ;default :assert (0 ); }
qapi/migration.json +15 -4
@@ -760,6 +760,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -779,8 +782,8 @@ 'multifd-channels', 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', - 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'multifd-zlib-level', 'multifd-zstd-level', + 'block-bitmap-mapping', 'vcpu-dirtylimit-period'] } ## # @MigrateSetParameters:
@@ -925,6 +928,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -960,7 +966,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*vcpu-dirtylimit-period': 'uint64'} } ## # @migrate-set-parameters:
@@ -1125,6 +1132,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -1158,7 +1168,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*vcpu-dirtylimit-period': 'uint64'} } ## # @query-migrate-parameters:
softmmu/dirtylimit.c +10 -1
@@ -23,6 +23,8 @@#include "exec/memory.h" #include "hw/boards.h" #include "sysemu/kvm.h" +#include "migration/misc.h" +#include "migration/migration.h" #include "trace.h" /*
@@ -75,11 +77,18 @@ static bool dirtylimit_quit;static void vcpu_dirty_rate_stat_collect (void ) { +MigrationState *s =migrate_get_current ();VcpuStat stat;int i =0 ; +int64_t period =DIRTYLIMIT_CALC_TIME_MS ; + +if (migrate_dirtylimit () && +migration_is_active (s)) { + period = s->parameters.vcpu_dirtylimit_period; + }/* calculate vcpu dirtyrate */ -vcpu_calculate_dirtyrate (DIRTYLIMIT_CALC_TIME_MS , +vcpu_calculate_dirtyrate (period, &stat,GLOBAL_DIRTY_LIMIT , false);
Signature · 1 line
1.8.3.1
Re: [RFC 1/6] qapi/migration: Introduce vcpu-dirtylimit-period parameters
Quoted 4 lines · On Tue, May 17, 2022 at 02:35:01PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirtylimit-period" migration parameters, which is used to makes dirtyrate calculation period
make
Quoted 4 lines
configurable.
To implement that, refactor
vcpu_dirty_rate_stat_collectso that period can be configured instead of hardcode.
hardcoded
Quoted 7 lines
Meanwhile, introduce
migrate_dirtylimitfunction to help check if dirtylimit enabled during live migration, set it false by default.Signed-off-by Hyman Huang(黄勇) <[email protected]>---
Focusing just on UI...
qapi/migration.json line 764
Quoted 7 lines
+++
b/qapi/migration.jsonqapi/migration.json +2
-0@@ -760,6 +760,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0)
The next release is 7.1. You'll need to fix this and all other references.
Do we want 'dirty-limit' instead of 'dirtylimit'? There was a recent thread on how to translate QAPI to other languages that are a bit more insistent on MixedCase, where properly separating English words makes it easier to translate to the expected case.
qapi/migration.json line 1139
Quoted 11 lines
qapi/migration.json +3
-0## # @migrate-set-parameters:@@ -1125,6 +1132,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental.
Is this feature ready for prime time, or do we want to initially name it x-vcpu-dirty[-]limit-period to mark it experimental?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [RFC 1/6] qapi/migration: Introduce vcpu-dirtylimit-period parameters
Quoted 25 lines · Eric Blake
Quoted 4 lines · On Tue, May 17, 2022 at 02:35:01PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirtylimit-period" migration parameters, which is used to makes dirtyrate calculation period
make
Quoted 4 lines
configurable.
To implement that, refactor
vcpu_dirty_rate_stat_collectso that period can be configured instead of hardcode.hardcoded
Quoted 7 lines
Meanwhile, introduce
migrate_dirtylimitfunction to help check if dirtylimit enabled during live migration, set it false by default.Signed-off-by Hyman Huang(黄勇) <[email protected]>---
Focusing just on UI...
Quoted 9 lines
qapi/migration.json line 764
Quoted 7 lines
+++
b/qapi/migration.jsonqapi/migration.json +2
-0@@ -760,6 +760,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0)The next release is 7.1. You'll need to fix this and all other references.
Ok, i'll fix that in the v1.
Quoted 6 lines
Do we want 'dirty-limit' instead of 'dirtylimit'? There was a recent thread on how to translate QAPI to other languages that are a bit more insistent on MixedCase, where properly separating English words makes it easier to translate to the expected case.
Changing the parameter name sounds ok to me, i'm not insistent that。
Quoted 6 lines
Quoted 6 lines
qapi/migration.json +0
-0## # @migrate-set-parameters:@@ -1125,6 +1132,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) #
Quoted 8 lines
qapi/migration.json line 767
Quoted 5 lines
qapi/migration.json +3
-0+# @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. +# Defaults to 500ms. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental.Is this feature ready for prime time, or do we want to initially name it x-vcpu-dirty[-]limit-period to mark it experimental?
Indeed, for this fresh new feature, finding factors affecting migration need more practice, marking it experimental could be much safer, i'll do that in v1.
Quoted 1 line
Signature · 3 lines
Best regard Hyman Huang(黄勇)
RFC 2/6 qapi/migration: Introduce vcpu-dirtylimit parameters
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirtylimit" migration parameter used to limit dirty page rate during live migration.
"vcpu-dirtylimit" and "vcpu-dirtylimit-period" are two dirtylimit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.
This two parameters are used to help implement the dirty page rate limit algo of migration.
--- migration/migration.c | 13 +++++++++++++ monitor/hmp-cmds.c | 7 +++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 35 insertions(+), 3 deletions(-)
migration/migration.c +13 -0
@@ -117,6 +117,7 @@#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100#define DEFAULT_MIGRATE_VCPU_DIRTYLIMIT_PERIOD 500/* ms */ +#define DEFAULT_MIGRATE_VCPU_DIRTYLIMIT 1/* MB/s */ static NotifierList migration_state_notifiers =NOTIFIER_LIST_INITIALIZER (migration_state_notifiers);
@@ -925,6 +926,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_vcpu_dirtylimit_period = true; params->vcpu_dirtylimit_period = s->parameters.vcpu_dirtylimit_period; + params->has_vcpu_dirtylimit = true; + params->vcpu_dirtylimit = s->parameters.vcpu_dirtylimit;return params; }
@@ -1590,6 +1593,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_vcpu_dirtylimit_period) { dest->vcpu_dirtylimit_period = params->vcpu_dirtylimit_period; } + +if (params->has_vcpu_dirtylimit) { + dest->vcpu_dirtylimit = params->vcpu_dirtylimit; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1715,6 +1722,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_vcpu_dirtylimit_period) { s->parameters.vcpu_dirtylimit_period = params->vcpu_dirtylimit_period; } +if (params->has_vcpu_dirtylimit) { + s->parameters.vcpu_dirtylimit = params->vcpu_dirtylimit; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
@@ -4241,6 +4251,9 @@ static Property migration_properties[] = {DEFINE_PROP_UINT64 ("vcpu-dirtylimit-period" , MigrationState, parameters.vcpu_dirtylimit_period,DEFAULT_MIGRATE_VCPU_DIRTYLIMIT_PERIOD ), +DEFINE_PROP_UINT64 ("vcpu-dirtylimit" , MigrationState, + parameters.vcpu_dirtylimit, +DEFAULT_MIGRATE_VCPU_DIRTYLIMIT ),/* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
monitor/hmp-cmds.c +7 -0
@@ -529,6 +529,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %" PRIu64" MB/s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_VCPU_DIRTYLIMIT_PERIOD ), params->vcpu_dirtylimit_period); +monitor_printf (mon,"%s: %" PRIu64" MB/s\n" , +MigrationParameter_str (MIGRATION_PARAMETER_VCPU_DIRTYLIMIT ), + params->vcpu_dirtylimit); }qapi_free_MigrationParameters (params);
@@ -1352,6 +1355,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_vcpu_dirtylimit_period = true;visit_type_size (v, param, &p->vcpu_dirtylimit_period, &err);break ; +case MIGRATION_PARAMETER_VCPU_DIRTYLIMIT : + p->has_vcpu_dirtylimit = true; +visit_type_size (v, param, &p->vcpu_dirtylimit, &err); +break ;default :assert (0 ); }
qapi/migration.json +15 -3
@@ -763,6 +763,9 @@ # @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. # Defaults to 500ms. (Since 7.0) # +# @vcpu-dirtylimit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -783,7 +786,8 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', - 'block-bitmap-mapping', 'vcpu-dirtylimit-period'] } + 'block-bitmap-mapping', 'vcpu-dirtylimit-period', + 'vcpu-dirtylimit'] } ## # @MigrateSetParameters:
@@ -931,6 +935,9 @@ # @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. # Defaults to 500ms. (Since 7.0) # +# @vcpu-dirtylimit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -967,7 +974,8 @@ '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*vcpu-dirtylimit-period': 'uint64'} } + '*vcpu-dirtylimit-period': 'uint64', + '*vcpu-dirtylimit': 'uint64'} } ## # @migrate-set-parameters:
@@ -1135,6 +1143,9 @@ # @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. # Defaults to 500ms. (Since 7.0) # +# @vcpu-dirtylimit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.0) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. #
@@ -1169,7 +1180,8 @@ '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*vcpu-dirtylimit-period': 'uint64'} } + '*vcpu-dirtylimit-period': 'uint64', + '*vcpu-dirtylimit': 'uint64'} } ## # @query-migrate-parameters:
Signature · 1 line
1.8.3.1
Re: [RFC 2/6] qapi/migration: Introduce vcpu-dirtylimit parameters
Quoted 17 lines · On Tue, May 17, 2022 at 02:35:02PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce "vcpu-dirtylimit" migration parameter used to limit dirty page rate during live migration.
"vcpu-dirtylimit" and "vcpu-dirtylimit-period" are two dirtylimit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters.
This two parameters are used to help implement the dirty page rate limit algo of migration.
Signed-off-by Hyman Huang(黄勇) <[email protected]>---
+++b/qapi/migration.json
qapi/migration.json line 768
Quoted 6 lines
qapi/migration.json +3
-0@@ -763,6 +763,9 @@ # @vcpu-dirtylimit-period: Periodic time (ms) of dirtylimit during live migration. # Defaults to 500ms. (Since 7.0) # +# @vcpu-dirtylimit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 7.0) +#
Same comments as in patch 1 (since 7.1, question on whether dirty-limit makes more sense).
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
RFC 3/6 migration: Implement dirtylimit convergence algo
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Implement dirtylimit convergence algo for live migration, which is kind of like auto-converge algo but using dirtylimit instead of cpu throttle to make migration convergent.
--- migration/ram.c | 53 +++++++++++++++++++++++++++++++++++++------------- migration/trace-events | 1 + 2 files changed, 41 insertions(+), 13 deletions(-)
migration/ram.c +40 -13
@@ -44,6 +44,7 @@#include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h"
@@ -56,6 +57,8 @@#include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */
@@ -1082,6 +1085,21 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* +* Enable dirtylimit to throttle down the guest +*/ +static void migration_dirtylimit_guest (void ) +{ +if (!dirtylimit_in_service ()) { +MigrationState *s =migrate_get_current (); +int64_t quota_dirtyrate = s->parameters.vcpu_dirtylimit; + +/* Set quota dirtyrate if dirty limit not in service */ +qmp_set_vcpu_dirty_limit (false,-1 , quota_dirtyrate,NULL ); +trace_migration_dirtylimit_guest (quota_dirtyrate); + } +} +static void migration_trigger_throttle (RAMState *rs) {MigrationState *s =migrate_get_current ();
@@ -1091,22 +1109,31 @@ static void migration_trigger_throttle(RAMState *rs)uint64_t bytes_dirty_period = rs->num_dirty_pages_period *TARGET_PAGE_SIZE ;uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold /100 ; -/* During block migration the auto-converge logic incorrectly detects -* that ram migration makes no progress. Avoid this by disabling the -* throttling logic during the bulk phase of block migration. */ -if (migrate_auto_converge () && !blk_mig_bulk_active ()) { -/* The following detection logic can be refined later. For now: -Check to see if the ratio between dirtied bytes and the approx. -amount of bytes that just got transferred since the last time -we were in this routine reaches the threshold. If that happens -twice, start or increase throttling. */ - -if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >=2 )) { +/* +* The following detection logic can be refined later. For now: +* Check to see if the ratio between dirtied bytes and the approx. +* amount of bytes that just got transferred since the last time +* we were in this routine reaches the threshold. If that happens +* twice, start or increase throttling. +*/ + +if ((bytes_dirty_period > bytes_dirty_threshold) && + (++rs->dirty_rate_high_cnt >=2 )) { + rs->dirty_rate_high_cnt =0 ; +/* +* During block migration the auto-converge logic incorrectly detects +* that ram migration makes no progress. Avoid this by disabling the +* throttling logic during the bulk phase of block migration. +*/ + +if (migrate_auto_converge () && !blk_mig_bulk_active ()) {trace_migration_throttle (); - rs->dirty_rate_high_cnt =0 ;mig_throttle_guest_down (bytes_dirty_period, bytes_dirty_threshold); + }else if (migrate_dirtylimit () && +kvm_dirty_ring_enabled () && +migration_is_active (s)) { +migration_dirtylimit_guest (); } } }
migration/trace-events +1 -0
@@ -89,6 +89,7 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" migration_throttle(void) "" +migration_dirtylimit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
Signature · 1 line
1.8.3.1
RFC 4/6 migration: Introduce dirtylimit capability
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Introduce migration dirtylimit capability, which can be turned on before live migration and limit dirty page rate durty live migration.
Dirtylimit dirtylimit capability is kind of like auto-converge but using dirtylimit instead of traditional cpu-throttle to throttle guest down.
To enable this feature, turn on the dirtylimit capability before live migration using migratioin-set-capabilities, and set dirtylimit-related parameters "vcpu-dirtylimit", "vcpu-dirtylimit-period" suitably to speed up convergence.
--- migration/migration.c | 7 ++++++- qapi/migration.json | 5 ++++- 2 files changed, 10 insertions(+), 2 deletions(-)
migration/migration.c +6 -1
@@ -2427,7 +2427,11 @@ bool migrate_auto_converge(void)bool migrate_dirtylimit (void ) { -return false; +MigrationState *s; + + s =migrate_get_current (); + +return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTYLIMIT ]; }bool migrate_zero_blocks (void )
@@ -4270,6 +4274,7 @@ static Property migration_properties[] = {DEFINE_PROP_MIG_CAP ("x-multifd" ,MIGRATION_CAPABILITY_MULTIFD ),DEFINE_PROP_MIG_CAP ("x-background-snapshot" ,MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT ), +DEFINE_PROP_MIG_CAP ("x-dirtylimit" ,MIGRATION_CAPABILITY_DIRTYLIMIT ), DEFINE_PROP_END_OF_LIST(), };
qapi/migration.json +4 -1
@@ -463,6 +463,9 @@ # procedure starts. The VM RAM is saved with running VM. # (since 6.0) # +# @dirtylimit: Use dirtylimit to throttle down guest if enabled. +# (since 7.0) +# # Features: # @unstable: Members @x-colo and @x-ignore-shared are experimental. #
@@ -476,7 +479,7 @@ 'block', 'return-path', 'pause-before-switchover', 'multifd', 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, - 'validate-uuid', 'background-snapshot'] } + 'validate-uuid', 'background-snapshot', 'dirtylimit'] } ## # @MigrationCapabilityStatus:
Signature · 1 line
1.8.3.1
Re: [RFC 4/6] migration: Introduce dirtylimit capability
Quoted 5 lines · On Tue, May 17, 2022 at 02:35:04PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
Introduce migration dirtylimit capability, which can be turned on before live migration and limit dirty page rate durty live migration.
s/durty/during/ ?
Quoted 2 lines
Dirtylimit dirtylimit capability is kind of like
Doubled word
Quoted 13 lines
auto-converge but using dirtylimit instead of traditional cpu-throttle to throttle guest down.
To enable this feature, turn on the dirtylimit capability before live migration using migratioin-set-capabilities, and set dirtylimit-related parameters "vcpu-dirtylimit", "vcpu-dirtylimit-period" suitably to speed up convergence.
Signed-off-by Hyman Huang(黄勇) <[email protected]>---
+++
b/qapi/migration.json
qapi/migration.json line 467
Quoted 5 lines
qapi/migration.json +2
-0@@ -463,6 +463,9 @@ # procedure starts. The VM RAM is saved with running VM. # (since 6.0) # +# @dirtylimit: Use dirtylimit to throttle down guest if enabled. +# (since 7.0)
7.1
same question about naming it 'dirty-limit'
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
RFC 5/6 migration: Add dirtylimit data into migration info
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Add dirtylimit throttle data into migration info, through which we can observe the process of dirtylimit during live migration.
--- include/sysemu/dirtylimit.h | 2 ++ migration/migration.c | 10 ++++++++++ qapi/migration.json | 11 ++++++++++- softmmu/dirtylimit.c | 22 ++++++++++++++++++++++ 4 files changed, 44 insertions(+), 1 deletion(-)
include/sysemu/dirtylimit.h +2 -0
@@ -34,4 +34,6 @@ void dirtylimit_set_vcpu(int cpu_index,void dirtylimit_set_all (uint64_t quota,bool enable);void dirtylimit_vcpu_execute (CPUState *cpu); +int64_t dirtylimit_throttle_us_per_full (void ); +int64_t dirtylimit_us_per_full (void );#endif
migration/migration.c +10 -0
@@ -61,6 +61,7 @@#include "sysemu/cpus.h" #include "yank_functions.h" #include "sysemu/qtest.h" +#include "sysemu/dirtylimit.h" #define MAX_THROTTLE (128 << 20)/* Migration transfer speed throttling */
@@ -1065,6 +1066,15 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->remaining =ram_bytes_remaining (); info->ram->dirty_pages_rate = ram_counters.dirty_pages_rate; } + +if (migrate_dirtylimit () &&dirtylimit_in_service ()) { + info->has_dirtylimit_throttle_us_per_full = true; + info->dirtylimit_throttle_us_per_full = +dirtylimit_throttle_us_per_full (); + + info->has_dirtylimit_us_per_full = true; + info->dirtylimit_us_per_full =dirtylimit_us_per_full (); + } }static void populate_disk_info (MigrationInfo *info)
qapi/migration.json +10 -1
@@ -239,6 +239,13 @@ # Present and non-empty when migration is blocked. # (since 6.0) # +# @dirtylimit-throttle-us-per-full: Throttle time (us) during the period of +# dirty ring full. +# (since 7.0) +# +# @dirtylimit-us-per-full: Estimated the periodic time (us) of dirty ring full. +# (since 7.0) +# # Since: 0.14 ## { 'struct': 'MigrationInfo',
@@ -256,7 +263,9 @@ '*postcopy-blocktime' : 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], '*compression': 'CompressionStats', - '*socket-address': ['SocketAddress'] } } + '*socket-address': ['SocketAddress'], + '*dirtylimit-throttle-us-per-full': 'int64', + '*dirtylimit-us-per-full': 'int64'} } ## # @query-migrate:
softmmu/dirtylimit.c +22 -0
@@ -546,6 +546,28 @@ static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index)return info; } +/* Pick up first vcpu throttle time by default */ +int64_t dirtylimit_throttle_us_per_full (void ) +{ +CPUState *cpu = first_cpu; +return cpu->throttle_us_per_full; +} + +/* +* Estimate dirty ring full time under current dirty page rate. +* Return -1 if guest doesn't dirty memory. +*/ +int64_t dirtylimit_us_per_full (void ) +{ +uint64_t curr_rate =vcpu_dirty_rate_get (0 ); + +if (!curr_rate) { +return -1 ; + } + +return dirtylimit_dirty_ring_full_time (curr_rate); +} +static struct DirtyLimitInfoList *dirtylimit_query_all (void ) {int i, index;
Signature · 1 line
1.8.3.1
RFC 6/6 tests: Add migration dirtylimit capability test
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
Add migration dirtylimit capability test if kernel support dirty ring.
Migration dirtylimit capability introduce dirtylimit capability , two parameters: vcpu-dirtylimit-period and vcpu-dirtylimit to implement the live migration with dirtylimit.
The test case enable the capability and set the corresponding parameters to test migration. When migration switch to pre-switchover phase, like the auto-converge, checking if migration satisfy the convergence condition.
--- tests/qtest/migration-test.c | 89 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
tests/qtest/migration-test.c +89 -0
@@ -1650,6 +1650,93 @@ static void test_vcpu_dirty_limit(void)dirtylimit_stop_vm (vm); } +static void test_migrate_dirty_limit (void ) +{ + g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); +MigrateStart *args =migrate_start_new (); +QTestState *from, *to; +int64_t remaining, throttle_us_per_full; +/* +* We want the test to be stable and as fast as possible. +* E.g., with 1Gb/s bandwith migration may pass without dirtylimit, +* so we need to decrease a bandwidth. +*/ +const int64_t dirtylimit_period =1000 , dirtylimit_value =50 ; +const int64_t max_bandwidth =400000000 ;/* ~400Mb/s */ +const int64_t downtime_limit =250 ;/* 250ms */ +/* +* We migrate through unix-socket (> 500Mb/s). +* Thus, expected migration speed ~= bandwidth limit (< 500Mb/s). +* So, we can predict expected_threshold +*/ +const int64_t expected_threshold = max_bandwidth * downtime_limit /1000 ; + +/* Enable dirty ring logging */ + args->use_dirty_ring = true; + +if (test_migrate_start (&from, &to, uri, &args)) { +return ; + } + +migrate_set_capability (from,"dirtylimit" , true); +migrate_set_parameter_int (from,"vcpu-dirtylimit-period" , + dirtylimit_period); +migrate_set_parameter_int (from,"vcpu-dirtylimit" , dirtylimit_value); + +/* +* Set the initial parameters so that the migration could not converge +* without dirtylimit. +*/ +migrate_set_parameter_int (from,"downtime-limit" ,1 ); +migrate_set_parameter_int (from,"max-bandwidth" ,100000000 );/* ~100Mb/s */ + +/* To check limit rate after precopy */ +migrate_set_capability (from,"pause-before-switchover" , true); + +/* Wait for the first serial output from the source */ +wait_for_serial ("src_serial" ); + +migrate_qmp (from, uri,"{}" ); + +/* Wait for dirtylimit begins */ + throttle_us_per_full =0 ; +while (throttle_us_per_full ==0 ) { + throttle_us_per_full = +read_migrate_property_int (from,"dirtylimit-throttle-us-per-full" ); +usleep (100 ); +g_assert_false (got_stop); + } + +/* +* The dirtylimit rate should equals the return value of +* query-vcpu-dirty-limit if dirtylimit cap set +*/ + g_assert_cmpint(dirtylimit_value, ==, get_limit_rate(from)); + +/* Now, when we tested if dirtylimit works, let it converge */ +migrate_set_parameter_int (from,"downtime-limit" , downtime_limit); +migrate_set_parameter_int (from,"max-bandwidth" , max_bandwidth); + +/* +* Wait for pre-switchover status to check if migration +* satisfy the convergence condition +*/ +wait_for_migration_status (from,"pre-switchover" ,NULL ); + + remaining =read_ram_property_int (from,"remaining" ); + g_assert_cmpint(remaining, <, + (expected_threshold + expected_threshold /100 )); + +migrate_continue (from,"pre-switchover" ); + +qtest_qmp_eventwait (to,"RESUME" ); + +wait_for_serial ("dest_serial" ); +wait_for_migration_complete (from); + +test_migrate_end (from, to, true); +} +static bool kvm_dirty_ring_supported (void ) {#if defined(__linux__) && defined(HOST_X86_64 )
@@ -1741,6 +1828,8 @@ int main(int argc, char **argv) test_precopy_unix_dirty_ring);qtest_add_func ("/migration/vcpu_dirty_limit" , test_vcpu_dirty_limit); +qtest_add_func ("/migration/dirty_limit" , + test_migrate_dirty_limit); } ret =g_test_run ();
Signature · 1 line
1.8.3.1