migration: Unify capabilities and parameters
cover migration: Unify capabilities and parameters
Hi everyone, I did a cleanup (if it can be called that) of the user input validation for capabilities and parameters and turned the two concepts into a single 'options' to be stored in a MigrationConfig object.
RFC mostly because this idea exposes (pre-existing) issues around how to validate capabilities that are mutually excludent and options that need to be enabled together.
I'd also like some feedback on what approach to take regarding compatibility.
Let me know what you think. I know it's a lot to look at, any comments are welcomed and don't worry on trying to cover everything. This is long-term work. Thank you.
The reasons for this work are:
------------------------------
Capabilities are just boolean parameters that can only be set before migration. For the majority of the code there's no distinction between the two.
Having a single structure allows the qmp_migrate command to receive the migration configuration all at once:
{ 'command': 'migrate',
'data': {'*uri': 'str',
'*channels': [ 'MigrationChannel' ],
+ '*config': 'MigrationConfig',
'*detach': 'bool', '*resume': 'bool' } }
+{ 'struct': 'MigrationConfig',
+ 'data': { '*announce-initial': 'size',
+ '*announce-max': 'size',
+ ... <-- all parameters and capabilities as optional
+} }
(optionally fold 'detach' and 'resume' into MigrationConfig)
Other benefits of a single configuration struct:
- allows the removal of two commands:
migrate-set-capabilities/query-capabilities which simplifies the user interface (migrate-set-parameters, or similar, is still required for options that need to be adjusted during migration);
- removes (some of) the triplication in migration.json;
- simplifies the (future) work of implementing a handshake feature for
migration: only one structure to negotiate over and less reliance on commands other than 'migrate'.
The major changes in this series are:
-------------------------------------
- Add a (QAPI) type hierarchy:
MigrationConfigBase
(most config options)
|
+-------------------------|-------------------------+
| | |
MigrationConfig MigrationParameters MigrateSetParameters
(internal use, s->config, (compat with (compat with new query/set-config) query-migrate-parameters) migrate-set-parameters)
- Remove migrate_params_test_apply. This function duplicates a lot of code;
- Add compatibility routines to convert from the existing QMP user input into the new MigrationConfig for internal use;
- Merge capabilities and parameters validation into one function;
- Convert qmp_migrate and qmp_migrate_incoming to use the new structure.
Open questions:
---------------
- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)
I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if migration_is_running()', without adding more duplication all around.
- What about savevm?
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
- HMP?
Can we convert the strings passed via hmp_set_parameters without having an enum of parameters? Duplication problem again.
- incoming defer?
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from qmp_migrate on source and send it to the destination for negotiation.
- last but definitely not least:
Changing caps from a list of bools into struct members complicates the validation because some caps must be checked against every other cap already set. And the user could be playing games with switching caps on and off, so there's a lot of redundant checking the must be made. The current code already has a bunch of gaps in that regard.
For this series I opted to not check has_* fields for capabilities, i.e. to validate them all every time migrate_config_check() is called.
Fabiano Rosas (12):
migration: Normalize tls arguments
migration: Run a post update routine after setting parameters
migration: Fix parameter validation
migration: Reduce a bit of duplication in migration.json
migration: Remove the parameters copy during validation
migration: Introduce new MigrationConfig structure
migration: Replace s->parameters with s->config
migration: Do away with usage of QERR_INVALID_PARAMETER_VALUE
migration: Replace s->capabilities with s->config
migration: Merge parameters and capability checks [PoC] migration: Add query/set commands for MigrationConfig [PoC] migration: Allow migrate commands to provide the migration
config
Markus Armbruster (1):
migration: Fix latent bug in migrate_params_test_apply()
migration/migration-hmp-cmds.c | 5 +- migration/migration.c | 52 +- migration/migration.h | 5 +- migration/options.c | 1386 ++++++++++++++++---------------- migration/options.h | 25 +- migration/page_cache.c | 6 +- migration/ram.c | 9 +- migration/savevm.c | 8 +- migration/tls.c | 2 +- qapi/migration.json | 571 +++++++------ system/vl.c | 3 +- 11 files changed, 1079 insertions(+), 993 deletions(-)
Signature · 1 line
2.35.3
RFC PATCH 1/13 migration: Fix latent bug in migrate_params_test_apply()
From: Markus Armbruster <[email protected]>
migrate_params_test_apply() neglects to apply tls_authz. Currently harmless, because migrate_params_check() doesn't care. Fix it anyway.
Message-ID: <[email protected]>
--- migration/options.c | 5 +++++ 1 file changed, 5 insertions(+)
migration/options.c +5 -0
@@ -1193,6 +1193,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->tls_hostname = params->tls_hostname->u.s; } +if (params->tls_authz) { +assert (params->tls_authz->type ==QTYPE_QSTRING ); + dest->tls_authz = params->tls_authz->u.s; + } +if (params->has_max_bandwidth) { dest->max_bandwidth = params->max_bandwidth; }
Signature · 1 line
2.35.3
RFC PATCH 2/13 migration: Normalize tls arguments
In reply to Fabiano RosasThe tls_creds, tls_authz and tls_hostname arguments are strings that can be set by the user. They are allowed to be either a valid string, an empty string or NULL. The values "" and NULL are effectively treated the same by the code, but this is not entirely clear because the handling is not uniform.
Make the 3 variables be handled the same and at the same place in options.c. Note that this affects only the internal usage of the variables.
(migrate_tls() had to be moved to be able to use migrate_tls_creds())
--- migration/options.c | 81 ++++++++++++++++++++++++--------------------- migration/tls.c | 2 +- 2 files changed, 44 insertions(+), 39 deletions(-)
migration/options.c +43 -38
@@ -379,13 +379,6 @@ bool migrate_rdma(void)return s->rdma_migration; } -bool migrate_tls (void ) -{ -MigrationState *s =migrate_get_current (); - -return s->parameters.tls_creds && *s->parameters.tls_creds; -} - typedef enum WriteTrackingSupport { WT_SUPPORT_UNKNOWN = 0, WT_SUPPORT_ABSENT,
@@ -814,21 +807,41 @@ const char *migrate_tls_authz(void) {MigrationState *s =migrate_get_current (); -return s->parameters.tls_authz; +if (s->parameters.tls_authz && + *s->parameters.tls_authz) { +return s->parameters.tls_authz; + } + +return NULL ; }const char *migrate_tls_creds (void ) {MigrationState *s =migrate_get_current (); -return s->parameters.tls_creds; +if (s->parameters.tls_creds && + *s->parameters.tls_creds) { +return s->parameters.tls_creds; + } + +return NULL ; }const char *migrate_tls_hostname (void ) {MigrationState *s =migrate_get_current (); -return s->parameters.tls_hostname; +if (s->parameters.tls_hostname && + *s->parameters.tls_hostname) { +return s->parameters.tls_hostname; + } + +return NULL ; +} + +bool migrate_tls (void ) +{ +return !!migrate_tls_creds (); }uint64_t migrate_vcpu_dirty_limit_period (void )
@@ -883,8 +896,10 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->cpu_throttle_increment = s->parameters.cpu_throttle_increment; params->has_cpu_throttle_tailslow = true; params->cpu_throttle_tailslow = s->parameters.cpu_throttle_tailslow; - params->tls_creds =g_strdup (s->parameters.tls_creds); - params->tls_hostname =g_strdup (s->parameters.tls_hostname); + params->tls_creds =g_strdup (s->parameters.tls_creds ? + s->parameters.tls_creds :"" ); + params->tls_hostname =g_strdup (s->parameters.tls_hostname ? + s->parameters.tls_hostname :"" ); params->tls_authz =g_strdup (s->parameters.tls_authz ? s->parameters.tls_authz :"" ); params->has_max_bandwidth = true;
@@ -945,6 +960,7 @@ void migrate_params_init(MigrationParameters *params) { params->tls_hostname =g_strdup ("" ); params->tls_creds =g_strdup ("" ); + params->tls_authz =g_strdup ("" );/* Set has_* up only for parameter checks */ params->has_throttle_trigger_threshold = true;
@@ -1184,18 +1200,27 @@ static void migrate_params_test_apply(MigrateSetParameters *params, }if (params->tls_creds) { -assert (params->tls_creds->type ==QTYPE_QSTRING ); - dest->tls_creds = params->tls_creds->u.s; +if (params->tls_creds->type ==QTYPE_QNULL ) { + dest->tls_creds =NULL ; + }else { + dest->tls_creds = params->tls_creds->u.s; + } }if (params->tls_hostname) { -assert (params->tls_hostname->type ==QTYPE_QSTRING ); - dest->tls_hostname = params->tls_hostname->u.s; +if (params->tls_hostname->type ==QTYPE_QNULL ) { + dest->tls_hostname =NULL ; + }else { + dest->tls_hostname = params->tls_hostname->u.s; + } }if (params->tls_authz) { -assert (params->tls_authz->type ==QTYPE_QSTRING ); - dest->tls_authz = params->tls_authz->u.s; +if (params->tls_authz->type ==QTYPE_QNULL ) { + dest->tls_authz =NULL ; + }else { + dest->tls_authz = params->tls_authz->u.s; + } }if (params->has_max_bandwidth) {
@@ -1413,26 +1438,6 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) {MigrationParameters tmp; -/* TODO Rewrite "" to null instead for all three tls_* parameters */ -if (params->tls_creds - && params->tls_creds->type ==QTYPE_QNULL ) { -qobject_unref (params->tls_creds->u.n); - params->tls_creds->type =QTYPE_QSTRING ; - params->tls_creds->u.s =strdup ("" ); - } -if (params->tls_hostname - && params->tls_hostname->type ==QTYPE_QNULL ) { -qobject_unref (params->tls_hostname->u.n); - params->tls_hostname->type =QTYPE_QSTRING ; - params->tls_hostname->u.s =strdup ("" ); - } -if (params->tls_authz - && params->tls_authz->type ==QTYPE_QNULL ) { -qobject_unref (params->tls_authz->u.n); - params->tls_authz->type =QTYPE_QSTRING ; - params->tls_authz->u.s =strdup ("" ); - } -migrate_params_test_apply (params, &tmp);if (!migrate_params_check (&tmp, errp)) {
migration/tls.c +1 -1
@@ -126,7 +126,7 @@ QIOChannelTLS *migration_tls_client_create(QIOChannel *ioc, }const char *tls_hostname =migrate_tls_hostname (); -if (tls_hostname && *tls_hostname) { +if (tls_hostname) { hostname = tls_hostname; }
Signature · 1 line
2.35.3
Re: [RFC PATCH 02/13] migration: Normalize tls arguments
Quoted 25 lines · Fabiano Rosas
The
tls_creds,tls_authzandtls_hostnamearguments are strings that can be set by the user. They are allowed to be either a valid string, an empty string or NULL. The values "" and NULL are effectively treated the same by the code, but this is not entirely clear because the handling is not uniform.Make the 3 variables be handled the same and at the same place in
options.c. Note that this affects only the internal usage of the variables.(
migrate_tls()had to be moved to be able to usemigrate_tls_creds())Signed-off-by Fabiano Rosas <[email protected]>--- migration/options.c | 81 ++++++++++++++++++++++++--------------------- migration/tls.c | 2 +- 2 files changed, 44 insertions(+), 39 deletions(-)migration/options.c +0
-0
migration/options.c static void migrate_params_test_apply(MigrateSetParameters *params, · line 1207
Quoted 10 lines
migration/options.c +5
-2@@ -1184,18 +1200,27 @@ static void migrate_params_test_apply(MigrateSetParameters *params, }if (params->tls_creds) { -assert (params->tls_creds->type ==QTYPE_QSTRING ); - dest->tls_creds = params->tls_creds->u.s; +if (params->tls_creds->type ==QTYPE_QNULL ) { + dest->tls_creds =NULL ; + }else { + dest->tls_creds = params->tls_creds->u.s; + }
Feels like it is still worth having the assert(QTYPE_QSTRING) in the else branch before we blindly reference the string pointer
Quoted 13 lines
migration/options.c +5
-2}if (params->tls_hostname) { -assert (params->tls_hostname->type ==QTYPE_QSTRING ); - dest->tls_hostname = params->tls_hostname->u.s; +if (params->tls_hostname->type ==QTYPE_QNULL ) { + dest->tls_hostname =NULL ; + }else { + dest->tls_hostname = params->tls_hostname->u.s; + } }if (params->tls_authz) {
migration/options.c static void migrate_params_test_apply(MigrateSetParameters *params, · line 1226
Quoted 10 lines
migration/options.c +5
-2-assert (params->tls_authz->type ==QTYPE_QSTRING ); - dest->tls_authz = params->tls_authz->u.s; +if (params->tls_authz->type ==QTYPE_QNULL ) { + dest->tls_authz =NULL ; + }else { + dest->tls_authz = params->tls_authz->u.s; + } }if (params->has_max_bandwidth) {
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 3/13 migration: Run a post update routine after setting parameters
In reply to Fabiano RosasSome migration parameters are updated immediately once they are set via migrate-set-parameters. Move that work outside of migrate_params_apply() and leave that function with the single responsibility of setting s->parameters and not doing any side-effects.
--- migration/options.c | 39 +++++++++++++++++++++++++++++---------- migration/ram.c | 2 +- 2 files changed, 30 insertions(+), 11 deletions(-)
migration/options.c +29 -10
@@ -1302,7 +1302,7 @@ static void migrate_params_test_apply(MigrateSetParameters *params, } } -static void migrate_params_apply (MigrateSetParameters *params,Error **errp) +static void migrate_params_apply (MigrateSetParameters *params) {MigrationState *s =migrate_get_current ();
@@ -1344,9 +1344,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_max_bandwidth) { s->parameters.max_bandwidth = params->max_bandwidth; -if (s->to_dst_file && !migration_in_postcopy ()) { -migration_rate_set (s->parameters.max_bandwidth); - } }if (params->has_avail_switchover_bandwidth) {
@@ -1359,7 +1356,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_x_checkpoint_delay) { s->parameters.x_checkpoint_delay = params->x_checkpoint_delay; -colo_checkpoint_delay_set (); }if (params->has_multifd_channels) {
@@ -1379,13 +1375,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) }if (params->has_xbzrle_cache_size) { s->parameters.xbzrle_cache_size = params->xbzrle_cache_size; -xbzrle_cache_resize (params->xbzrle_cache_size, errp); }if (params->has_max_postcopy_bandwidth) { s->parameters.max_postcopy_bandwidth = params->max_postcopy_bandwidth; -if (s->to_dst_file &&migration_in_postcopy ()) { -migration_rate_set (s->parameters.max_postcopy_bandwidth); - } }if (params->has_max_cpu_throttle) { s->parameters.max_cpu_throttle = params->max_cpu_throttle;
@@ -1434,6 +1426,32 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) } } +static void migrate_post_update_params (MigrateSetParameters *new,Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (new->has_max_bandwidth) { +if (s->to_dst_file && !migration_in_postcopy ()) { +migration_rate_set (new->max_bandwidth); + } + } + +if (new->has_x_checkpoint_delay) { +colo_checkpoint_delay_set (); + } + +if (new->has_xbzrle_cache_size) { +xbzrle_cache_resize (new->xbzrle_cache_size, errp); + } + +if (new->has_max_postcopy_bandwidth) { +if (s->to_dst_file &&migration_in_postcopy ()) { +migration_rate_set (new->max_postcopy_bandwidth); + } + } + +} +void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp) {MigrationParameters tmp;
@@ -1445,5 +1463,6 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)return ; } -migrate_params_apply (params, errp); +migrate_params_apply (params); +migrate_post_update_params (params, errp); }
migration/ram.c +1 -1
@@ -144,7 +144,7 @@ static void XBZRLE_cache_unlock(void)/** * xbzrle_cache_resize: resize the xbzrle cache * -* This function is called from migrate_params_apply in main +* This function is called from migrate_post_update_config in main * thread, possibly while a migration is in progress. A running * migration may be using the cache and might finish during this call, * hence changes to the cache are protected by XBZRLE.lock().
Signature · 1 line
2.35.3
Re: [RFC PATCH 03/13] migration: Run a post update routine after setting parameters
Quoted 7 lines · Fabiano Rosas
Some migration parameters are updated immediately once they are set via migrate-set-parameters. Move that work outside of
migrate_params_apply()and leave that function with the single responsibility of settings->parametersand not doing any side-effects.Signed-off-by Fabiano Rosas <[email protected]>
Signature · 1 line
Peter Xu
RFC PATCH 4/13 migration: Fix parameter validation
In reply to Fabiano RosasThe migration parameters validation involves producing a temporary structure which merges the current parameter values with the new parameters set by the user.
The has_ boolean fields of MigrateSetParameter are taken into consideration when writing the temporary structure, however the copy of the current parameters also copies all the has_ fields of s->parameters and those are (almost) all true due to being initialized by migrate_params_init().
Since the temporary structure copy does not carry over the has_ fields from MigrateSetParameters, only the values which were initialized in migrate_params_init() will end up being validated. This causes (almost) all of the migration parameters to be validated again every time a parameter is set, which could be considered a bug. But it also skips validation of those values which are not set in migrate_params_init(), which is a worse issue.
Fix by initializing the missing values in migrate_params_init(). Currently 'avail_switchover_bandwidth' and 'block_bitmap_mapping' are affected.
--- migration/options.c | 2 ++ 1 file changed, 2 insertions(+)
migration/options.c +2 -0
@@ -987,6 +987,8 @@ void migrate_params_init(MigrationParameters *params) params->has_mode = true; params->has_zero_page_detection = true; params->has_direct_io = true; + params->has_avail_switchover_bandwidth = true; + params->has_block_bitmap_mapping = true; }/*
Signature · 1 line
2.35.3
Re: [RFC PATCH 04/13] migration: Fix parameter validation
Quoted 10 lines · Fabiano Rosas
The migration parameters validation involves producing a temporary structure which merges the current parameter values with the new parameters set by the user.
The
has_boolean fields of MigrateSetParameter are taken into consideration when writing the temporary structure, however the copy of the current parameters also copies all thehas_fields ofs->parametersand those are (almost) all true due to being initialized bymigrate_params_init().
Quoted 7 lines
Since the temporary structure copy does not carry over the
has_fields from MigrateSetParameters, only the values which were initialized inmigrate_params_init()will end up being validated. This causes (almost) all of the migration parameters to be validated again every time a parameter is set, which could be considered a bug. But it also skips validation of those values which are not set inmigrate_params_init(), which is a worse issue.
IMHO it's ok to double check all parameters in slow path. Definitely not ok to skip them.. So now the question is, if migrate_params_test_apply() so far should check all params anyway...
Shall we drop the checking for all has_ there, then IIUC we also don't need any initializations for has_* in migrate_params_init() here?
So, admittedly s->parameters.has_* is still ugly to be present.. we declare all of them not used and ignore them always at least in s->parameters.
Quoted 26 lines
Fix by initializing the missing values in
migrate_params_init(). Currently 'avail_switchover_bandwidth' and 'block_bitmap_mapping' are affected.Signed-off-by Fabiano Rosas <[email protected]>--- migration/options.c | 2 ++ 1 file changed, 2 insertions(+)migration/options.c +2
-0@@ -987,6 +987,8 @@ void migrate_params_init(MigrationParameters *params) params->has_mode = true; params->has_zero_page_detection = true; params->has_direct_io = true; + params->has_avail_switchover_bandwidth = true; + params->has_block_bitmap_mapping = true; }/* Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [RFC PATCH 04/13] migration: Fix parameter validation
Quoted 19 lines · Peter Xu
Quoted 10 lines · Fabiano Rosas
The migration parameters validation involves producing a temporary structure which merges the current parameter values with the new parameters set by the user.
The
has_boolean fields of MigrateSetParameter are taken into consideration when writing the temporary structure, however the copy of the current parameters also copies all thehas_fields ofs->parametersand those are (almost) all true due to being initialized bymigrate_params_init().Quoted 7 lines
Since the temporary structure copy does not carry over the
has_fields from MigrateSetParameters, only the values which were initialized inmigrate_params_init()will end up being validated. This causes (almost) all of the migration parameters to be validated again every time a parameter is set, which could be considered a bug. But it also skips validation of those values which are not set inmigrate_params_init(), which is a worse issue.
Quoted 4 lines
IMHO it's ok to double check all parameters in slow path. Definitely not ok to skip them.. So now the question is, if
migrate_params_test_apply()so far should check all params anyway...
Well, either way is fine by me. In the current code, I can't tell what the intention was, unfortunately.
We could check all params, but then we need to make sure they never change in between calls to migrate-set-params. Looking at the code I don't see any place where we allow s->parameters to change.
But then I worry about checks that:
- might be too costly
- only make sense the first time
- depend on the order of setting (a param/cap that should only be set before/after some other param/cap)
Quoted 3 lines
Shall we drop the checking for all
has_there, then IIUC we also don't need any initializations for has_* inmigrate_params_init()here?
It'd be nice to not have to touch the has_ fields, yes. I'll experiment with it.
Quoted 29 lines
So, admittedly s->parameters.has_* is still ugly to be present.. we declare all of them not used and ignore them always at least in
s->parameters.Quoted 26 lines
Fix by initializing the missing values in
migrate_params_init(). Currently 'avail_switchover_bandwidth' and 'block_bitmap_mapping' are affected.Signed-off-by Fabiano Rosas <[email protected]>--- migration/options.c | 2 ++ 1 file changed, 2 insertions(+)migration/options.c +2
-0@@ -987,6 +987,8 @@ void migrate_params_init(MigrationParameters *params) params->has_mode = true; params->has_zero_page_detection = true; params->has_direct_io = true; + params->has_avail_switchover_bandwidth = true; + params->has_block_bitmap_mapping = true; }/* Signature · 1 line
2.35.3
Re: [RFC PATCH 04/13] migration: Fix parameter validation
Quoted 21 lines · Fabiano Rosas
Quoted 19 lines · Peter Xu
Quoted 17 lines · Fabiano Rosas
The migration parameters validation involves producing a temporary structure which merges the current parameter values with the new parameters set by the user.
The
has_boolean fields of MigrateSetParameter are taken into consideration when writing the temporary structure, however the copy of the current parameters also copies all thehas_fields ofs->parametersand those are (almost) all true due to being initialized bymigrate_params_init().Since the temporary structure copy does not carry over the
has_fields from MigrateSetParameters, only the values which were initialized inmigrate_params_init()will end up being validated. This causes (almost) all of the migration parameters to be validated again every time a parameter is set, which could be considered a bug. But it also skips validation of those values which are not set inmigrate_params_init(), which is a worse issue.
Quoted 3 lines
Quoted 3 lines
IMHO it's ok to double check all parameters in slow path. Definitely not ok to skip them.. So now the question is, if
migrate_params_test_apply()so far should check all params anyway...
Actually, this doesn't work...
The migrate-set-* commands have optional fields, so we need some form of checking has_* to know which fields the user is setting. Otherwise MigrationSetParameters will have zeros all over that will trip the check.
Then, we need some form of checking has_* to be able to enventually get the values into s->config (former s->parameters/capabilities), otherwise we'll overwrite the already-set values with the potentially empty ones coming from QAPI.
Then there's also the issue of knowing whether a value is 0 because the user set it 0 or because it was never set.
We also can't apply an invalid value to s->config and validate it after because some parameters are allowed to be changed during migration.
Re: [RFC PATCH 04/13] migration: Fix parameter validation
Quoted 12 lines · Fabiano Rosas
Actually, this doesn't work...
The migrate-set-* commands have optional fields, so we need some form of checking has_* to know which fields the user is setting. Otherwise MigrationSetParameters will have zeros all over that will trip the check.
Then, we need some form of checking has_* to be able to enventually get the values into
s->config(former s->parameters/capabilities), otherwise we'll overwrite the already-set values with the potentially empty ones coming from QAPI.
Quoted 5 lines
Then there's also the issue of knowing whether a value is 0 because the user set it 0 or because it was never set.
We also can't apply an invalid value to
s->configand validate it after because some parameters are allowed to be changed during migration.
What I meant was we only conditionally ignore the has_* fields in below:
(1) migrate_params_check(), so that QEMU always checks all parameters in the MigrationParameters* specified when invoking the function.
(2) MigrationState.parameters, so that as long as the parameters are applied (it should only happen after sanity check all pass..) then we ignore these has_* fields (until MigrationState.parameters can have a better struct to not include these has_* fields).
We can keep the has_* checks in migrate_params_test_apply() and migrate_params_apply(), so that we won't touch the ones the user didn't specify in the QMP commands as you said.
The benefits of having above 1/2 ignoring has_* is some code removal where we assume has_* always are TRUEs.
This can be still a bit confusing, but at least we don't need to init has_* fields in migrate_params_init() anymore as they'll be all ignored, then there's no chance we forget set TRUEs to any new params either.
Signature · 1 line
Peter Xu
Re: [RFC PATCH 04/13] migration: Fix parameter validation
Quoted 16 lines · Peter Xu
Quoted 15 lines · Fabiano Rosas
Actually, this doesn't work...
The migrate-set-* commands have optional fields, so we need some form of checking has_* to know which fields the user is setting. Otherwise MigrationSetParameters will have zeros all over that will trip the check.
Then, we need some form of checking has_* to be able to enventually get the values into
s->config(former s->parameters/capabilities), otherwise we'll overwrite the already-set values with the potentially empty ones coming from QAPI.Then there's also the issue of knowing whether a value is 0 because the user set it 0 or because it was never set.
Quoted 8 lines
Quoted 2 lines
We also can't apply an invalid value to
s->configand validate it after because some parameters are allowed to be changed during migration.What I meant was we only conditionally ignore the has_* fields in below:
(1)
migrate_params_check(), so that QEMU always checks all parameters in the MigrationParameters* specified when invoking the function.
Yes, I realised later that's what you meant. I'm looking into it. Hitting some issues with the block_bitmap_mapping option, which seems to be able to become NULL even if has_block_bitmap_mapping is true. According to commit 3cba22c9ad ("migration: Fix block_bitmap_mapping migration").
Quoted 15 lines
(2) MigrationState.parameters, so that as long as the parameters are applied (it should only happen after sanity check all pass..) then we ignore these has_* fields (until MigrationState.parameters can have a better struct to not include these has_* fields).
We can keep the has_* checks in
migrate_params_test_apply()andmigrate_params_apply(), so that we won't touch the ones the user didn't specify in the QMP commands as you said.The benefits of having above 1/2 ignoring has_* is some code removal where we assume has_* always are TRUEs.
This can be still a bit confusing, but at least we don't need to init has_* fields in
migrate_params_init()anymore as they'll be all ignored, then there's no chance we forget set TRUEs to any new params either.
RFC PATCH 5/13 migration: Reduce a bit of duplication in migration.json
In reply to Fabiano RosasIntroduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)
qapi/migration.json +108 -250
@@ -914,202 +914,6 @@ 'zero-page-detection', 'direct-io'] } -## -# @MigrateSetParameters: -# -# @announce-initial: Initial delay (in milliseconds) before sending -# the first announce (Since 4.0) -# -# @announce-max: Maximum delay (in milliseconds) between packets in -# the announcement (Since 4.0) -# -# @announce-rounds: Number of self-announce packets sent after -# migration (Since 4.0) -# -# @announce-step: Increase in delay (in milliseconds) between -# subsequent packets in the announcement (Since 4.0) -# -# @throttle-trigger-threshold: The ratio of bytes_dirty_period and -# bytes_xfer_period to trigger throttling. It is expressed as -# percentage. The default value is 50. (Since 5.0) -# -# @cpu-throttle-initial: Initial percentage of time guest cpus are -# throttled when migration auto-converge is activated. The -# default value is 20. (Since 2.7) -# -# @cpu-throttle-increment: throttle percentage increase each time -# auto-converge detects that migration is not making progress. -# The default value is 10. (Since 2.7) -# -# @cpu-throttle-tailslow: Make CPU throttling slower at tail stage At -# the tail stage of throttling, the Guest is very sensitive to CPU -# percentage while the @cpu-throttle -increment is excessive -# usually at tail stage. If this parameter is true, we will -# compute the ideal CPU percentage used by the Guest, which may -# exactly make the dirty rate match the dirty rate threshold. -# Then we will choose a smaller throttle increment between the one -# specified by @cpu-throttle-increment and the one generated by -# ideal CPU percentage. Therefore, it is compatible to -# traditional throttling, meanwhile the throttle increment won't -# be excessive at tail stage. The default value is false. (Since -# 5.1) -# -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. Setting this to a -# non-empty string enables TLS for all migrations. An empty -# string means that QEMU will use plain text mode for migration, -# rather than TLS. This is the default. (Since 2.7) -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. A non-empty value is -# required when using x509 based TLS credentials and the migration -# URI does not include a hostname, such as fd: or exec: based -# migration. (Since 2.7) -# -# Note: empty value works only since 2.9. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# This object is only resolved at time of use, so can be deleted -# and recreated on the fly while the migration server is active. -# If missing, it will default to denying access (Since 4.0) -# -# @max-bandwidth: maximum speed for migration, in bytes per second. -# (Since 2.8) -# -# @avail-switchover-bandwidth: to set the available bandwidth that -# migration can use during switchover phase. NOTE! This does not -# limit the bandwidth during switchover, but only for calculations -# when making decisions to switchover. By default, this value is -# zero, which means QEMU will estimate the bandwidth -# automatically. This can be set when the estimated value is not -# accurate, while the user is able to guarantee such bandwidth is -# available when switching over. When specified correctly, this -# can make the switchover decision much more accurate. -# (Since 8.2) -# -# @downtime-limit: set maximum tolerated downtime for migration. -# maximum downtime in milliseconds (Since 2.8) -# -# @x-checkpoint-delay: The delay time (in ms) between two COLO -# checkpoints in periodic mode. (Since 2.8) -# -# @multifd-channels: Number of channels used to migrate data in -# parallel. This is the same number that the number of sockets -# used for migration. The default value is 2 (since 4.0) -# -# @xbzrle-cache-size: cache size to be used by XBZRLE migration. It -# needs to be a multiple of the target page size and a power of 2 -# (Since 2.11) -# -# @max-postcopy-bandwidth: Background transfer bandwidth during -# postcopy. Defaults to 0 (unlimited). In bytes per second. -# (Since 3.0) -# -# @max-cpu-throttle: maximum cpu throttle percentage. Defaults to 99. -# (Since 3.1) -# -# @multifd-compression: Which compression method to use. Defaults to -# none. (Since 5.0) -# -# @multifd-zlib-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 9, -# where 0 means no compression, 1 means the best compression -# speed, and 9 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @multifd-qatzip-level: Set the compression level to be used in live -# migration. The level is an integer between 1 and 9, where 1 means -# the best compression speed, and 9 means the best compression -# ratio which will consume more CPU. Defaults to 1. (Since 9.2) -# -# @multifd-zstd-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 20, -# where 0 means no compression, 1 means the best compression -# speed, and 20 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @block-bitmap-mapping: Maps block nodes and bitmaps on them to -# aliases for the purpose of dirty bitmap migration. Such aliases -# may for example be the corresponding names on the opposite site. -# The mapping must be one-to-one, but not necessarily complete: On -# the source, unmapped bitmaps and all bitmaps on unmapped nodes -# will be ignored. On the destination, encountering an unmapped -# alias in the incoming migration stream will result in a report, -# and all further bitmap migration data will then be discarded. -# Note that the destination does not know about bitmaps it does -# not receive, so there is no limitation or requirement regarding -# the number of bitmaps received, or how they are named, or on -# which nodes they are placed. By default (when this parameter -# has never been set), bitmap names are mapped to themselves. -# Nodes are mapped to their block device name if there is one, and -# to their node name otherwise. (Since 5.2) -# -# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty -# limit during live migration. Should be in the range 1 to -# 1000ms. Defaults to 1000ms. (Since 8.1) -# -# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. -# Defaults to 1. (Since 8.1) -# -# @mode: Migration mode. See description in @MigMode. Default is -# 'normal'. (Since 8.2) -# -# @zero-page-detection: Whether and how to detect zero pages. -# See description in @ZeroPageDetection. Default is 'multifd'. -# (since 9.0) -# -# @direct-io: Open migration files with O_DIRECT when possible. This -# only has effect if the @mapped-ram capability is enabled. -# (Since 9.1) -# -# Features: -# -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. -# -# TODO: either fuse back into MigrationParameters, or make -# MigrationParameters members mandatory -# -# Since: 2.4 -## -{ 'struct': 'MigrateSetParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'StrOrNull', - '*tls-hostname': 'StrOrNull', - '*tls-authz': 'StrOrNull', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } - ## # @migrate-set-parameters: #
@@ -1127,9 +931,7 @@ 'data': 'MigrateSetParameters' } ## -# @MigrationParameters: -# -# The optional members aren't actually optional. +# @MigrationConfigBase: # # @announce-initial: Initial delay (in milliseconds) before sending # the first announce (Since 4.0)
@@ -1168,26 +970,6 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. An empty string -# means that QEMU will use plain text mode for migration, rather -# than TLS. (Since 2.7) -# -# Note: 2.8 omits empty @tls-creds instead. -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. (Since 2.7) -# -# Note: 2.8 omits empty @tls-hostname instead. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# (Since 4.0) -# # @max-bandwidth: maximum speed for migration, in bytes per second. # (Since 2.8) #
@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } + +## +# @MigrationParameters: +# +# The optional members of the base type aren't actually optional. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# # Since: 2.4 ## { 'struct': 'MigrationParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'str', - '*tls-hostname': 'str', - '*tls-authz': 'str', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } } + +## +# @MigrateSetParameters: +# +# Compatibility layer to accept null values for the TLS options. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. This is the default. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# +# Since: 2.4 +## +{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } } ## # @query-migrate-parameters:
Signature · 1 line
2.35.3
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 32 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +0
-0@@ -914,202 +914,6 @@
qapi/migration.json line 1065
Quoted 7 lines
qapi/migration.json +4
-0@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +#
I'm not really finding it compelling to add a bool @tls as it is just a denormalization of !!@tls-creds.
Incidentally the docs here are wrong - TLS can be used by only setting @tls-creds. The @tls-authz & @tls-hostname params are always optional.
Quoted 44 lines
qapi/migration.json +36
-0# Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } { 'struct': 'MigrationParameters', + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } }
snip
qapi/migration.json line 1176
Quoted 5 lines
qapi/migration.json +5
-0+{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } }
I recall we discussed this difference a year or two ago, but can't recall what the outcome was.
Making the TLS params optional is a back compatible change for MigrationParameters. I would think replacing 'str' with 'StrOrNull' is also back compatible. So I'm wondering if we can't just unify the sttructs fully for TLS, even if one usage scenario never actually needs the "OrNull" bit nor needs the optionality
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 32 lines · Daniel P. Berrangé
Quoted 30 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +0
-0@@ -914,202 +914,6 @@
Quoted 12 lines
qapi/migration.json line 1065
Quoted 8 lines
qapi/migration.json +4
-0@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +#I'm not really finding it compelling to add a bool @tls as it is just a denormalization of !!@tls-creds.
This is here by mistake.
I remember Markus mentioning that implying TLS usage from tls-creds was undesirable. I was prototyping a way of requiring an explicit opt-in.
Quoted 57 lines
Incidentally the docs here are wrong - TLS can be used by only setting @tls-creds. The @tls-authz & @tls-hostname params are always optional.
Quoted 44 lines
qapi/migration.json +36
-0# Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } { 'struct': 'MigrationParameters', + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } }snip
qapi/migration.json line 1176
Quoted 5 lines
qapi/migration.json +5
-0+{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } }
Quoted 9 lines
I recall we discussed this difference a year or two ago, but can't recall what the outcome was.
Making the TLS params optional is a back compatible change for MigrationParameters. I would think replacing 'str' with 'StrOrNull' is also back compatible. So I'm wondering if we can't just unify the sttructs fully for TLS, even if one usage scenario never actually needs the "OrNull" bit nor needs the optionality
MigrationParameters is the output type for query-migrate-parameters. I belive it must be all non-optional to keep compatibility. The docs on master say: "The optional members aren't really optional"
For MigrateSetParameters they've always been optional and continue to be. There we need to keep StrOrNull for compat.
Quoted 3 lines
With regards,
Daniel
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 47 lines · Fabiano Rosas
Quoted 32 lines · Daniel P. Berrangé
Quoted 30 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +0
-0@@ -914,202 +914,6 @@Quoted 12 lines
qapi/migration.json line 1065
Quoted 8 lines
qapi/migration.json +4
-0@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +#I'm not really finding it compelling to add a bool @tls as it is just a denormalization of !!@tls-creds.
Quoted 4 lines
This is here by mistake.
I remember Markus mentioning that implying TLS usage from tls-creds was undesirable. I was prototyping a way of requiring an explicit opt-in.
I don't remember a thing :)
Quoted 67 lines
Quoted 57 lines
Incidentally the docs here are wrong - TLS can be used by only setting @tls-creds. The @tls-authz & @tls-hostname params are always optional.
Quoted 44 lines
qapi/migration.json +36
-0# Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } { 'struct': 'MigrationParameters', + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } }snip
qapi/migration.json line 1176
Quoted 5 lines
qapi/migration.json +5
-0+{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } }Quoted 9 lines
I recall we discussed this difference a year or two ago, but can't recall what the outcome was.
Making the TLS params optional is a back compatible change for MigrationParameters. I would think replacing 'str' with 'StrOrNull' is also back compatible. So I'm wondering if we can't just unify the sttructs fully for TLS, even if one usage scenario never actually needs the "OrNull" bit nor needs the optionality
Quoted 3 lines
MigrationParameters is the output type for query-migrate-parameters. I belive it must be all non-optional to keep compatibility. The docs on master say: "The optional members aren't really optional"
To be precise: even though the members are declared optional, they must always be present.
The members became optional when commit de63ab61241 (migrate: Share common MigrationParameters struct) reused MigrationParameters as migrate-set-parameters argument type.
Making mandatory members of some type optional is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Making its members optional was not kosher. Because the optional members are always present, QMP didn't actually change, except for introspection.
Quoted 2 lines
For MigrateSetParameters they've always been optional and continue to be. There we need to keep StrOrNull for compat.
StrOrNull goes back to commit 01fa5598269 (migration: Use JSON null instead of "" to reset parameter to default).
Similar argument: adding values to a member is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Adding value null is won't be kosher. However, as long as as the value is never actually used there, QMP won't actually change, except for introspection.
If this was a clean and obvious interface, I'd argue against such trickery and for keeping it clean and obvious. But the migration configuration interface isn't.
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 49 lines · Markus Armbruster
Quoted 47 lines · Fabiano Rosas
Quoted 44 lines · Daniel P. Berrangé
Quoted 32 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +0
-0@@ -914,202 +914,6 @@qapi/migration.json line 1065
Quoted 7 lines
qapi/migration.json +4
-0@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +#I'm not really finding it compelling to add a bool @tls as it is just a denormalization of !!@tls-creds.
Quoted 7 lines
Quoted 4 lines
This is here by mistake.
I remember Markus mentioning that implying TLS usage from tls-creds was undesirable. I was prototyping a way of requiring an explicit opt-in.
I don't remember a thing :)
I hope I interpreted you correctly:
"We have three syntactically independent parameters: @tls-creds, @tls-hostname, @tls-authz. The latter two are meaningless (and silently ignored) unless the first one is given. I hate that.
TLS is off by default. To enable it, set @tls-creds. Except setting it to the otherwise invalid value "" disables it. That's because all we have for configuration is the "set migration parameter to value" interface. Only works because we have an invalid value we can abuse. I hate that." -- https://lore.kernel.org/all/[email protected]/
I still think it might be an improvement to define a new interface that requires explicitly enabling TLS. IOW, make tls a capability. But I really don't want to get caught on that right now, one mess at a time.
Quoted 71 lines
Quoted 67 lines
Quoted 57 lines
Incidentally the docs here are wrong - TLS can be used by only setting @tls-creds. The @tls-authz & @tls-hostname params are always optional.
Quoted 44 lines
qapi/migration.json +36
-0# Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } { 'struct': 'MigrationParameters', + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } }snip
qapi/migration.json line 1176
Quoted 5 lines
qapi/migration.json +5
-0+{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } }Quoted 9 lines
I recall we discussed this difference a year or two ago, but can't recall what the outcome was.
Making the TLS params optional is a back compatible change for MigrationParameters. I would think replacing 'str' with 'StrOrNull' is also back compatible. So I'm wondering if we can't just unify the sttructs fully for TLS, even if one usage scenario never actually needs the "OrNull" bit nor needs the optionality
Quoted 3 lines
MigrationParameters is the output type for query-migrate-parameters. I belive it must be all non-optional to keep compatibility. The docs on master say: "The optional members aren't really optional"
Quoted 7 lines
To be precise: even though the members are declared optional, they must always be present.
The members became optional when commit
de63ab61241(migrate: Share common MigrationParameters struct) reused MigrationParameters as migrate-set-parameters argument type.
Then, commit 1bda8b3c69 (migration: Unshare MigrationParameters struct for now, 2017-07-18) partially undid that change and introduced MigrateSetParameters.
At that point we might have been able to make the MigrationParameters' members mandatory, but it was chosen not too, probably with good reason. I can't tell from the commit message.
Strictly speaking, we cannot make them mandatory now because commit 31e4c354b3 (migration: Add block-bitmap-mapping parameter, 2020-08-20) added the block-bitmap parameter, but made its output in query-migrate-parameters truly optional:
if (s->parameters.has_block_bitmap_mapping) {
params->has_block_bitmap_mapping = true;
params->block_bitmap_mapping =
QAPI_CLONE(BitmapMigrationNodeAliasList,
s->parameters.block_bitmap_mapping);
}Quoted 21 lines
Making mandatory members of some type optional is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Making its members optional was not kosher. Because the optional members are always present, QMP didn't actually change, except for introspection.
Quoted 2 lines
For MigrateSetParameters they've always been optional and continue to be. There we need to keep StrOrNull for compat.
StrOrNull goes back to commit
01fa5598269(migration: Use JSON null instead of "" to reset parameter to default).Similar argument: adding values to a member is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Adding value null is won't be kosher. However, as long as as the value is never actually used there, QMP won't actually change, except for introspection.
If this was a clean and obvious interface, I'd argue against such trickery and for keeping it clean and obvious. But the migration configuration interface isn't.
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 65 lines · Fabiano Rosas
Quoted 49 lines · Markus Armbruster
Quoted 47 lines · Fabiano Rosas
Quoted 44 lines · Daniel P. Berrangé
Quoted 32 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +0
-0@@ -914,202 +914,6 @@qapi/migration.json line 1065
Quoted 7 lines
qapi/migration.json +4
-0@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +#I'm not really finding it compelling to add a bool @tls as it is just a denormalization of !!@tls-creds.
Quoted 7 lines
Quoted 4 lines
This is here by mistake.
I remember Markus mentioning that implying TLS usage from tls-creds was undesirable. I was prototyping a way of requiring an explicit opt-in.
I don't remember a thing :)
I hope I interpreted you correctly:
"We have three syntactically independent parameters: @tls-creds, @tls-hostname, @tls-authz. The latter two are meaningless (and silently ignored) unless the first one is given. I hate that.
Quoted 5 lines
TLS is off by default. To enable it, set @tls-creds. Except setting it to the otherwise invalid value "" disables it. That's because all we have for configuration is the "set migration parameter to value" interface. Only works because we have an invalid value we can abuse. I hate that." -- https://lore.kernel.org/all/[email protected]/
Ah, now I remember!
Possible minimally invasive improvement here:
* Reject @tls-hostname and @tls-authz when the value of @tls-creds means "TLS is off".
* When @tls-set-creds get set to a value that means "TLS is off", @tls-hostname and @tls-authz get unset.
This could break applications. I'm not at all sure the slight reduction in ugliness would be worth the risk.
I do not mean to discourage pursuing your RFC! The TLS configuration is ugly and confusing to use internally. Getting that streamlined would be nice.
Quoted 3 lines
I still think it might be an improvement to define a new interface that requires explicitly enabling TLS. IOW, make tls a capability. But I really don't want to get caught on that right now, one mess at a time.
First: one mess at a time is commonly the sensible approach.
Second: I'm not sure attempts to make the existing migration configuration interfaces somewhat less ugly and confusing are worthwhile.
I rarely advocate starting over just because something is ugly. However, migration configuration is not only ugly; its use of global state feels fundamentally misguided to me. Evidence: the surprising side effects on savevm Daniel mentioned. I'll add: when a management application connects to an existing QEMU, it needs to laboriously set the entire global migration configuration state, or else rely on unsound assumptions. Complete non-problem when the entire configuration is passed to the command, like we do elsewhere.
Quoted 79 lines
Quoted 71 lines
Quoted 67 lines
Quoted 57 lines
Incidentally the docs here are wrong - TLS can be used by only setting @tls-creds. The @tls-authz & @tls-hostname params are always optional.
Quoted 44 lines
qapi/migration.json +36
-0# Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } { 'struct': 'MigrationParameters', + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } }snip
qapi/migration.json line 1176
Quoted 5 lines
qapi/migration.json +5
-0+{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } }Quoted 9 lines
I recall we discussed this difference a year or two ago, but can't recall what the outcome was.
Making the TLS params optional is a back compatible change for MigrationParameters. I would think replacing 'str' with 'StrOrNull' is also back compatible. So I'm wondering if we can't just unify the sttructs fully for TLS, even if one usage scenario never actually needs the "OrNull" bit nor needs the optionality
Quoted 3 lines
MigrationParameters is the output type for query-migrate-parameters. I belive it must be all non-optional to keep compatibility. The docs on master say: "The optional members aren't really optional"
Quoted 7 lines
To be precise: even though the members are declared optional, they must always be present.
The members became optional when commit
de63ab61241(migrate: Share common MigrationParameters struct) reused MigrationParameters as migrate-set-parameters argument type.
Quoted 7 lines
Then, commit
1bda8b3c69(migration: Unshare MigrationParameters struct for now, 2017-07-18) partially undid that change and introduced MigrateSetParameters.At that point we might have been able to make the MigrationParameters' members mandatory, but it was chosen not too, probably with good reason. I can't tell from the commit message.
As far as I remember: to minimize last minute churn, and to not complicate a future re-sharing.
Quoted 11 lines
Strictly speaking, we cannot make them mandatory now because commit
31e4c354b3(migration: Add block-bitmap-mapping parameter, 2020-08-20) added the block-bitmap parameter, but made its output in query-migrate-parameters truly optional:if (s->parameters.has_block_bitmap_mapping) { params->has_block_bitmap_mapping = true; params->block_bitmap_mapping = QAPI_CLONE(BitmapMigrationNodeAliasList, s->parameters.block_bitmap_mapping); }
We're consistently inconsistent ;)
Quoted 21 lines
Quoted 21 lines
Making mandatory members of some type optional is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Making its members optional was not kosher. Because the optional members are always present, QMP didn't actually change, except for introspection.
Quoted 2 lines
For MigrateSetParameters they've always been optional and continue to be. There we need to keep StrOrNull for compat.
StrOrNull goes back to commit
01fa5598269(migration: Use JSON null instead of "" to reset parameter to default).Similar argument: adding values to a member is compatible as long as the type occurs only in command arguments. But MigrationParameters is a command return type. Adding value null is won't be kosher. However, as long as as the value is never actually used there, QMP won't actually change, except for introspection.
If this was a clean and obvious interface, I'd argue against such trickery and for keeping it clean and obvious. But the migration configuration interface isn't.
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
In reply to Fabiano RosasQuoted 12 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Quoted 7 lines
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
I double-checked the external interface. There's a new member @tls, but you already told us it's an accident.
Documentation does change. More on that below.
Quoted 393 lines
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +101
-250@@ -914,202 +914,6 @@ 'zero-page-detection', 'direct-io'] } -## -# @MigrateSetParameters: -# -# @announce-initial: Initial delay (in milliseconds) before sending -# the first announce (Since 4.0) -# -# @announce-max: Maximum delay (in milliseconds) between packets in -# the announcement (Since 4.0) -# -# @announce-rounds: Number of self-announce packets sent after -# migration (Since 4.0) -# -# @announce-step: Increase in delay (in milliseconds) between -# subsequent packets in the announcement (Since 4.0) -# -# @throttle-trigger-threshold: The ratio of bytes_dirty_period and -# bytes_xfer_period to trigger throttling. It is expressed as -# percentage. The default value is 50. (Since 5.0) -# -# @cpu-throttle-initial: Initial percentage of time guest cpus are -# throttled when migration auto-converge is activated. The -# default value is 20. (Since 2.7) -# -# @cpu-throttle-increment: throttle percentage increase each time -# auto-converge detects that migration is not making progress. -# The default value is 10. (Since 2.7) -# -# @cpu-throttle-tailslow: Make CPU throttling slower at tail stage At -# the tail stage of throttling, the Guest is very sensitive to CPU -# percentage while the @cpu-throttle -increment is excessive -# usually at tail stage. If this parameter is true, we will -# compute the ideal CPU percentage used by the Guest, which may -# exactly make the dirty rate match the dirty rate threshold. -# Then we will choose a smaller throttle increment between the one -# specified by @cpu-throttle-increment and the one generated by -# ideal CPU percentage. Therefore, it is compatible to -# traditional throttling, meanwhile the throttle increment won't -# be excessive at tail stage. The default value is false. (Since -# 5.1) -# -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. Setting this to a -# non-empty string enables TLS for all migrations. An empty -# string means that QEMU will use plain text mode for migration, -# rather than TLS. This is the default. (Since 2.7) -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. A non-empty value is -# required when using x509 based TLS credentials and the migration -# URI does not include a hostname, such as fd: or exec: based -# migration. (Since 2.7) -# -# Note: empty value works only since 2.9. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# This object is only resolved at time of use, so can be deleted -# and recreated on the fly while the migration server is active. -# If missing, it will default to denying access (Since 4.0) -# -# @max-bandwidth: maximum speed for migration, in bytes per second. -# (Since 2.8) -# -# @avail-switchover-bandwidth: to set the available bandwidth that -# migration can use during switchover phase. NOTE! This does not -# limit the bandwidth during switchover, but only for calculations -# when making decisions to switchover. By default, this value is -# zero, which means QEMU will estimate the bandwidth -# automatically. This can be set when the estimated value is not -# accurate, while the user is able to guarantee such bandwidth is -# available when switching over. When specified correctly, this -# can make the switchover decision much more accurate. -# (Since 8.2) -# -# @downtime-limit: set maximum tolerated downtime for migration. -# maximum downtime in milliseconds (Since 2.8) -# -# @x-checkpoint-delay: The delay time (in ms) between two COLO -# checkpoints in periodic mode. (Since 2.8) -# -# @multifd-channels: Number of channels used to migrate data in -# parallel. This is the same number that the number of sockets -# used for migration. The default value is 2 (since 4.0) -# -# @xbzrle-cache-size: cache size to be used by XBZRLE migration. It -# needs to be a multiple of the target page size and a power of 2 -# (Since 2.11) -# -# @max-postcopy-bandwidth: Background transfer bandwidth during -# postcopy. Defaults to 0 (unlimited). In bytes per second. -# (Since 3.0) -# -# @max-cpu-throttle: maximum cpu throttle percentage. Defaults to 99. -# (Since 3.1) -# -# @multifd-compression: Which compression method to use. Defaults to -# none. (Since 5.0) -# -# @multifd-zlib-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 9, -# where 0 means no compression, 1 means the best compression -# speed, and 9 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @multifd-qatzip-level: Set the compression level to be used in live -# migration. The level is an integer between 1 and 9, where 1 means -# the best compression speed, and 9 means the best compression -# ratio which will consume more CPU. Defaults to 1. (Since 9.2) -# -# @multifd-zstd-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 20, -# where 0 means no compression, 1 means the best compression -# speed, and 20 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @block-bitmap-mapping: Maps block nodes and bitmaps on them to -# aliases for the purpose of dirty bitmap migration. Such aliases -# may for example be the corresponding names on the opposite site. -# The mapping must be one-to-one, but not necessarily complete: On -# the source, unmapped bitmaps and all bitmaps on unmapped nodes -# will be ignored. On the destination, encountering an unmapped -# alias in the incoming migration stream will result in a report, -# and all further bitmap migration data will then be discarded. -# Note that the destination does not know about bitmaps it does -# not receive, so there is no limitation or requirement regarding -# the number of bitmaps received, or how they are named, or on -# which nodes they are placed. By default (when this parameter -# has never been set), bitmap names are mapped to themselves. -# Nodes are mapped to their block device name if there is one, and -# to their node name otherwise. (Since 5.2) -# -# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty -# limit during live migration. Should be in the range 1 to -# 1000ms. Defaults to 1000ms. (Since 8.1) -# -# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. -# Defaults to 1. (Since 8.1) -# -# @mode: Migration mode. See description in @MigMode. Default is -# 'normal'. (Since 8.2) -# -# @zero-page-detection: Whether and how to detect zero pages. -# See description in @ZeroPageDetection. Default is 'multifd'. -# (since 9.0) -# -# @direct-io: Open migration files with O_DIRECT when possible. This -# only has effect if the @mapped-ram capability is enabled. -# (Since 9.1) -# -# Features: -# -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. -# -# TODO: either fuse back into MigrationParameters, or make -# MigrationParameters members mandatory -# -# Since: 2.4 -## -{ 'struct': 'MigrateSetParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'StrOrNull', - '*tls-hostname': 'StrOrNull', - '*tls-authz': 'StrOrNull', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } - ## # @migrate-set-parameters: #@@ -1127,9 +931,7 @@ 'data': 'MigrateSetParameters' } ## -# @MigrationParameters: -# -# The optional members aren't actually optional. +# @MigrationConfigBase: # # @announce-initial: Initial delay (in milliseconds) before sending # the first announce (Since 4.0)@@ -1168,26 +970,6 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. An empty string -# means that QEMU will use plain text mode for migration, rather -# than TLS. (Since 2.7) -# -# Note: 2.8 omits empty @tls-creds instead. -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. (Since 2.7) -# -# Note: 2.8 omits empty @tls-hostname instead. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# (Since 4.0) -# # @max-bandwidth: maximum speed for migration, in bytes per second. # (Since 2.8) #@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } + +## +# @MigrationParameters: +# +# The optional members of the base type aren't actually optional. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# # Since: 2.4 ## { 'struct': 'MigrationParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'str', - '*tls-hostname': 'str', - '*tls-authz': 'str', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } } + +## +# @MigrateSetParameters: +# +# Compatibility layer to accept null values for the TLS options. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. This is the default. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +#
qapi/migration.json line 1179
Quoted 10 lines
qapi/migration.json +7
-0+# Since: 2.4 +## +{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } } ## # @query-migrate-parameters:
Your change is fairly simple, but that's not obvious from the diff.
1. Move everything but the TLS stuff from MigrationParameters to its new base type MigrationConfigBase.
No change to the interface, obviously.
Introspection hides the base type, and shows members of MigrationParameters in a different order, but order doesn't matter.
The doc generator isn't smart enough (yet) to hide the base type. Apart from that, documentation is unchanged.
2. Replace everything but the TLS stuff from MigrateSetParameters by base type MigrateSetParameters.
Because the base type's members are identical to the members it replaces, no change to the interface.
Introspection hides the base type, and shows members of MigrateSetParameters in a different order, but order doesn't matter.
The base type's member documentation is *not* identical to the member documentation it replaces. To see the differences, I ran sphinx-build with -b text before and after, and fed its output to diff:
* **cpu-throttle-initial** ("int", *optional*) -- Initial percentage of time guest cpus are throttled when migration
- auto-converge is activated. The default value is 20. (Since - 2.7) + auto-converge is activated. (Since 2.7)
We no longer document the default value. This is wrong.
The difference before the patch makes some sense. The members of MigrateSetParameters are actually optional command arguments, and their defaults must be documented. The members of MigrationParameters aren't actually optional, and talking about defaults is misleading there. We do it anyway in a few places.
Factoring out MigrationConfigBase leads to a conflict: in its role as base of MigrationParameters, it shouldn't talk about defaults, and in its role as base of MigrateSetParameters, it should document the defaults supplied by migrate-set-parameters.
There is no perfectly satisfactory solution with our current infrastructure.
We can elect to ignore the problem for now. Point it out in a TODO comment.
We can try to explain in the doc text that default values apply only to migrate-set-parameters. I expect this to be awkward. More awkward once the doc generator inlines stuff.
* **cpu-throttle-increment** ("int", *optional*) -- throttle percentage increase each time auto-converge detects that
- migration is not making progress. The default value is 10. - (Since 2.7) + migration is not making progress. (Since 2.7)
Likewise.
- * **x-checkpoint-delay** ("int", *optional*) -- The delay time
- (in ms) between two COLO checkpoints in periodic mode. (Since
- 2.8)
+ * **x-checkpoint-delay** ("int", *optional*) -- the delay time
+ between two COLO checkpoints. (Since 2.8)
Inconsistent before the patch. The deleted version is better. Easy enough to fix.
+ * **tls** ("boolean", *optional*) -- Whether to use TLS. If this
+ is set the options "tls-authz", "tls-creds", "tls-hostname"
+ are mandatory and a valid string is expected. (Since 10.1)
+
Accident.
Re: [RFC PATCH 05/13] migration: Reduce a bit of duplication in migration.json
Quoted 500 lines · Markus Armbruster
Quoted 12 lines · Fabiano Rosas
Introduce a new MigrationConfigBase, to allow most of the duplication in migration.json to be eliminated.
The reason we need MigrationParameters and MigrationSetParameters is that the internal parameter representation in the migration code, as well as the user-facing return of query-migrate-parameters use one type for the TLS options (tls-creds, tls-hostname, tls-authz), while the user-facing input from migrate-set-parameters uses another.
The difference is in whether the NULL values is accepted. The former considers NULL as invalid, while the latter doesn't.
Quoted 7 lines
Move all other (non-TLS) options into the new type and make it a base type for the two diverging types so that each child type can declare the TLS options in its own way.
Nothing changes in the user API, nothing changes in the internal representation, but we save several lines of duplication in migration.json.
I double-checked the external interface. There's a new member @tls, but you already told us it's an accident.
Documentation does change. More on that below.
Quoted 393 lines
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 358 +++++++++++++------------------------------- 1 file changed, 108 insertions(+), 250 deletions(-)qapi/migration.json +101
-250@@ -914,202 +914,6 @@ 'zero-page-detection', 'direct-io'] } -## -# @MigrateSetParameters: -# -# @announce-initial: Initial delay (in milliseconds) before sending -# the first announce (Since 4.0) -# -# @announce-max: Maximum delay (in milliseconds) between packets in -# the announcement (Since 4.0) -# -# @announce-rounds: Number of self-announce packets sent after -# migration (Since 4.0) -# -# @announce-step: Increase in delay (in milliseconds) between -# subsequent packets in the announcement (Since 4.0) -# -# @throttle-trigger-threshold: The ratio of bytes_dirty_period and -# bytes_xfer_period to trigger throttling. It is expressed as -# percentage. The default value is 50. (Since 5.0) -# -# @cpu-throttle-initial: Initial percentage of time guest cpus are -# throttled when migration auto-converge is activated. The -# default value is 20. (Since 2.7) -# -# @cpu-throttle-increment: throttle percentage increase each time -# auto-converge detects that migration is not making progress. -# The default value is 10. (Since 2.7) -# -# @cpu-throttle-tailslow: Make CPU throttling slower at tail stage At -# the tail stage of throttling, the Guest is very sensitive to CPU -# percentage while the @cpu-throttle -increment is excessive -# usually at tail stage. If this parameter is true, we will -# compute the ideal CPU percentage used by the Guest, which may -# exactly make the dirty rate match the dirty rate threshold. -# Then we will choose a smaller throttle increment between the one -# specified by @cpu-throttle-increment and the one generated by -# ideal CPU percentage. Therefore, it is compatible to -# traditional throttling, meanwhile the throttle increment won't -# be excessive at tail stage. The default value is false. (Since -# 5.1) -# -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. Setting this to a -# non-empty string enables TLS for all migrations. An empty -# string means that QEMU will use plain text mode for migration, -# rather than TLS. This is the default. (Since 2.7) -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. A non-empty value is -# required when using x509 based TLS credentials and the migration -# URI does not include a hostname, such as fd: or exec: based -# migration. (Since 2.7) -# -# Note: empty value works only since 2.9. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# This object is only resolved at time of use, so can be deleted -# and recreated on the fly while the migration server is active. -# If missing, it will default to denying access (Since 4.0) -# -# @max-bandwidth: maximum speed for migration, in bytes per second. -# (Since 2.8) -# -# @avail-switchover-bandwidth: to set the available bandwidth that -# migration can use during switchover phase. NOTE! This does not -# limit the bandwidth during switchover, but only for calculations -# when making decisions to switchover. By default, this value is -# zero, which means QEMU will estimate the bandwidth -# automatically. This can be set when the estimated value is not -# accurate, while the user is able to guarantee such bandwidth is -# available when switching over. When specified correctly, this -# can make the switchover decision much more accurate. -# (Since 8.2) -# -# @downtime-limit: set maximum tolerated downtime for migration. -# maximum downtime in milliseconds (Since 2.8) -# -# @x-checkpoint-delay: The delay time (in ms) between two COLO -# checkpoints in periodic mode. (Since 2.8) -# -# @multifd-channels: Number of channels used to migrate data in -# parallel. This is the same number that the number of sockets -# used for migration. The default value is 2 (since 4.0) -# -# @xbzrle-cache-size: cache size to be used by XBZRLE migration. It -# needs to be a multiple of the target page size and a power of 2 -# (Since 2.11) -# -# @max-postcopy-bandwidth: Background transfer bandwidth during -# postcopy. Defaults to 0 (unlimited). In bytes per second. -# (Since 3.0) -# -# @max-cpu-throttle: maximum cpu throttle percentage. Defaults to 99. -# (Since 3.1) -# -# @multifd-compression: Which compression method to use. Defaults to -# none. (Since 5.0) -# -# @multifd-zlib-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 9, -# where 0 means no compression, 1 means the best compression -# speed, and 9 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @multifd-qatzip-level: Set the compression level to be used in live -# migration. The level is an integer between 1 and 9, where 1 means -# the best compression speed, and 9 means the best compression -# ratio which will consume more CPU. Defaults to 1. (Since 9.2) -# -# @multifd-zstd-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 20, -# where 0 means no compression, 1 means the best compression -# speed, and 20 means best compression ratio which will consume -# more CPU. Defaults to 1. (Since 5.0) -# -# @block-bitmap-mapping: Maps block nodes and bitmaps on them to -# aliases for the purpose of dirty bitmap migration. Such aliases -# may for example be the corresponding names on the opposite site. -# The mapping must be one-to-one, but not necessarily complete: On -# the source, unmapped bitmaps and all bitmaps on unmapped nodes -# will be ignored. On the destination, encountering an unmapped -# alias in the incoming migration stream will result in a report, -# and all further bitmap migration data will then be discarded. -# Note that the destination does not know about bitmaps it does -# not receive, so there is no limitation or requirement regarding -# the number of bitmaps received, or how they are named, or on -# which nodes they are placed. By default (when this parameter -# has never been set), bitmap names are mapped to themselves. -# Nodes are mapped to their block device name if there is one, and -# to their node name otherwise. (Since 5.2) -# -# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty -# limit during live migration. Should be in the range 1 to -# 1000ms. Defaults to 1000ms. (Since 8.1) -# -# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. -# Defaults to 1. (Since 8.1) -# -# @mode: Migration mode. See description in @MigMode. Default is -# 'normal'. (Since 8.2) -# -# @zero-page-detection: Whether and how to detect zero pages. -# See description in @ZeroPageDetection. Default is 'multifd'. -# (since 9.0) -# -# @direct-io: Open migration files with O_DIRECT when possible. This -# only has effect if the @mapped-ram capability is enabled. -# (Since 9.1) -# -# Features: -# -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. -# -# TODO: either fuse back into MigrationParameters, or make -# MigrationParameters members mandatory -# -# Since: 2.4 -## -{ 'struct': 'MigrateSetParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'StrOrNull', - '*tls-hostname': 'StrOrNull', - '*tls-authz': 'StrOrNull', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } - ## # @migrate-set-parameters: #@@ -1127,9 +931,7 @@ 'data': 'MigrateSetParameters' } ## -# @MigrationParameters: -# -# The optional members aren't actually optional. +# @MigrationConfigBase: # # @announce-initial: Initial delay (in milliseconds) before sending # the first announce (Since 4.0)@@ -1168,26 +970,6 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # -# @tls-creds: ID of the 'tls-creds' object that provides credentials -# for establishing a TLS connection over the migration data -# channel. On the outgoing side of the migration, the credentials -# must be for a 'client' endpoint, while for the incoming side the -# credentials must be for a 'server' endpoint. An empty string -# means that QEMU will use plain text mode for migration, rather -# than TLS. (Since 2.7) -# -# Note: 2.8 omits empty @tls-creds instead. -# -# @tls-hostname: migration target's hostname for validating the -# server's x509 certificate identity. If empty, QEMU will use the -# hostname from the migration URI, if any. (Since 2.7) -# -# Note: 2.8 omits empty @tls-hostname instead. -# -# @tls-authz: ID of the 'authz' object subclass that provides access -# control checking of the TLS x509 certificate distinguished name. -# (Since 4.0) -# # @max-bandwidth: maximum speed for migration, in bytes per second. # (Since 2.8) #@@ -1277,45 +1059,121 @@ # only has effect if the @mapped-ram capability is enabled. # (Since 9.1) # +# @tls: Whether to use TLS. If this is set the options @tls-authz, +# @tls-creds, @tls-hostname are mandatory and a valid string is +# expected. (Since 10.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # +# Since: 10.1 +## +{ 'struct': 'MigrationConfigBase', + 'data': { '*announce-initial': 'size', + '*announce-max': 'size', + '*announce-rounds': 'size', + '*announce-step': 'size', + '*throttle-trigger-threshold': 'uint8', + '*cpu-throttle-initial': 'uint8', + '*cpu-throttle-increment': 'uint8', + '*cpu-throttle-tailslow': 'bool', + '*max-bandwidth': 'size', + '*avail-switchover-bandwidth': 'size', + '*downtime-limit': 'uint64', + '*x-checkpoint-delay': { 'type': 'uint32', + 'features': [ 'unstable' ] }, + '*multifd-channels': 'uint8', + '*xbzrle-cache-size': 'size', + '*max-postcopy-bandwidth': 'size', + '*max-cpu-throttle': 'uint8', + '*multifd-compression': 'MultiFDCompression', + '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-zstd-level': 'uint8', + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64', + '*mode': 'MigMode', + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool', + '*tls': 'bool' } } + +## +# @MigrationParameters: +# +# The optional members of the base type aren't actually optional. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# # Since: 2.4 ## { 'struct': 'MigrationParameters', - 'data': { '*announce-initial': 'size', - '*announce-max': 'size', - '*announce-rounds': 'size', - '*announce-step': 'size', - '*throttle-trigger-threshold': 'uint8', - '*cpu-throttle-initial': 'uint8', - '*cpu-throttle-increment': 'uint8', - '*cpu-throttle-tailslow': 'bool', - '*tls-creds': 'str', - '*tls-hostname': 'str', - '*tls-authz': 'str', - '*max-bandwidth': 'size', - '*avail-switchover-bandwidth': 'size', - '*downtime-limit': 'uint64', - '*x-checkpoint-delay': { 'type': 'uint32', - 'features': [ 'unstable' ] }, - '*multifd-channels': 'uint8', - '*xbzrle-cache-size': 'size', - '*max-postcopy-bandwidth': 'size', - '*max-cpu-throttle': 'uint8', - '*multifd-compression': 'MultiFDCompression', - '*multifd-zlib-level': 'uint8', - '*multifd-qatzip-level': 'uint8', - '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], - '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] }, - '*vcpu-dirty-limit': 'uint64', - '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection', - '*direct-io': 'bool' } } + 'base': 'MigrationConfigBase', + 'data': { 'tls-creds': 'str', + 'tls-hostname': 'str', + 'tls-authz': 'str' } } + +## +# @MigrateSetParameters: +# +# Compatibility layer to accept null values for the TLS options. +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. This is the default. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +#qapi/migration.json line 1179
Quoted 10 lines
qapi/migration.json +7
-0+# Since: 2.4 +## +{ 'struct': 'MigrateSetParameters', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'StrOrNull', + '*tls-hostname': 'StrOrNull', + '*tls-authz': 'StrOrNull' } } ## # @query-migrate-parameters:Your change is fairly simple, but that's not obvious from the diff.
1. Move everything but the TLS stuff from MigrationParameters to its new base type MigrationConfigBase.
No change to the interface, obviously.
Introspection hides the base type, and shows members of MigrationParameters in a different order, but order doesn't matter.
The doc generator isn't smart enough (yet) to hide the base type. Apart from that, documentation is unchanged.
2. Replace everything but the TLS stuff from MigrateSetParameters by base type MigrateSetParameters.
Because the base type's members are identical to the members it replaces, no change to the interface.
Introspection hides the base type, and shows members of MigrateSetParameters in a different order, but order doesn't matter.
The base type's member documentation is *not* identical to the member documentation it replaces. To see the differences, I ran sphinx-build with -b text before and after, and fed its output to diff:
* **cpu-throttle-initial** ("int", *optional*) -- Initial percentage of time guest cpus are throttled when migration
(quoted diff) +4
-7- auto-converge is activated. The default value is 20. (Since - 2.7) + auto-converge is activated. (Since 2.7) We no longer document the default value. This is wrong. The difference before the patch makes some sense. The members of MigrateSetParameters are actually optional command arguments, and their defaults must be documented. The members of MigrationParameters aren't actually optional, and talking about defaults is misleading there. We do it anyway in a few places. Factoring out MigrationConfigBase leads to a conflict: in its role as base of MigrationParameters, it shouldn't talk about defaults, and in its role as base of MigrateSetParameters, it should document the defaults supplied by migrate-set-parameters. There is no perfectly satisfactory solution with our current infrastructure. We can elect to ignore the problem for now. Point it out in a TODO comment. We can try to explain in the doc text that default values apply only to migrate-set-parameters. I expect this to be awkward. More awkward once the doc generator inlines stuff. * **cpu-throttle-increment** ("int", *optional*) -- throttle percentage increase each time auto-converge detects that - migration is not making progress. The default value is 10. - (Since 2.7) + migration is not making progress. (Since 2.7) Likewise. - * **x-checkpoint-delay** ("int", *optional*) -- The delay time - (in ms) between two COLO checkpoints in periodic mode. (Since - 2.8) + * **x-checkpoint-delay** ("int", *optional*) -- the delay time + between two COLO checkpoints. (Since 2.8)
Review
Quoted 10 lines
(quoted diff) +4
-0Inconsistent before the patch. The deleted version is better. Easy enough to fix. + * **tls** ("boolean", *optional*) -- Whether to use TLS. If this + is set the options "tls-authz", "tls-creds", "tls-hostname" + are mandatory and a valid string is expected. (Since 10.1) + Accident.
I missed
Quoted 55 lines
base type MigrateSetParameters.
Because the base type's members are identical to the members it replaces, no change to the interface.
Introspection hides the base type, and shows members of MigrateSetParameters in a different order, but order doesn't matter.
The base type's member documentation is *not* identical to the member documentation it replaces. To see the differences, I ran sphinx-build with -b text before and after, and fed its output to diff:
* **cpu-throttle-initial** ("int", *optional*) -- Initial percentage of time guest cpus are throttled when migration
(quoted diff) +4
-7- auto-converge is activated. The default value is 20. (Since - 2.7) + auto-converge is activated. (Since 2.7) We no longer document the default value. This is wrong. The difference before the patch makes some sense. The members of MigrateSetParameters are actually optional command arguments, and their defaults must be documented. The members of MigrationParameters aren't actually optional, and talking about defaults is misleading there. We do it anyway in a few places. Factoring out MigrationConfigBase leads to a conflict: in its role as base of MigrationParameters, it shouldn't talk about defaults, and in its role as base of MigrateSetParameters, it should document the defaults supplied by migrate-set-parameters. There is no perfectly satisfactory solution with our current infrastructure. We can elect to ignore the problem for now. Point it out in a TODO comment. We can try to explain in the doc text that default values apply only to migrate-set-parameters. I expect this to be awkward. More awkward once the doc generator inlines stuff. * **cpu-throttle-increment** ("int", *optional*) -- throttle percentage increase each time auto-converge detects that - migration is not making progress. The default value is 10. - (Since 2.7) + migration is not making progress. (Since 2.7) Likewise. - * **x-checkpoint-delay** ("int", *optional*) -- The delay time - (in ms) between two COLO checkpoints in periodic mode. (Since - 2.8) + * **x-checkpoint-delay** ("int", *optional*) -- the delay time + between two COLO checkpoints. (Since 2.8)
Review
Quoted 10 lines
(quoted diff) +4
-0Inconsistent before the patch. The deleted version is better. Easy enough to fix. + * **tls** ("boolean", *optional*) -- Whether to use TLS. If this + is set the options "tls-authz", "tls-creds", "tls-hostname" + are mandatory and a valid string is expected. (Since 10.1) + Accident.
I missed
3. Make @tls-creds, @tls-hostname, @tls-authz non-optional in MigrationParameters.
Compatible change to the interface.
When my description of what a patch does goes 1. 2. 3., then splitting it up could be in order. Especially when I miss 3. initially :)
RFC PATCH 6/13 migration: Remove the parameters copy during validation
In reply to Fabiano RosasWhile validating migration parameters, a temporary structure is used to apply the values to be validated. Only when validation succeeds on the temporary values, they are actually copied into the internal s->parameters.
There is however an extra step, which is to copy s->parameters into the temporary structure prior to merging the new values in. This is not documented.
The validation function is intended to only validate new values by checking the QAPI generated has_* boolean fields for each struct member. However, nothing ever sets the has_* fields on the temporary structure, so the reason that copy is there is to take advantage of the has_* fields of s->parameters, which _have_ been set by migrate_params_init() for the same reason of allowing migrate_params_check() to interpret the values as newly added.
This is convoluted and has led to the current state of validating _every_ parameter, even the ones who were not set by the user.
migrate_params_init() sets _all_ has_* fields because there are global qdev properties defined for every parameter and those need to be validated at migration_object_check().
Detangle the validation done at object creation from the one done after user input, remove the s->parameters copy and move the migrate_params_init() function close to the migrate_params_check(). Add comments.
With the removal of the copy at migrate_set_parameters, there's still the need to convert MigrateSetParameters into MigrationParameters to satisfy the signature of migrate_params_check(). This is fine. Keep it, but replace the inline copy of every member (done at migrate_params_test_apply) with a QAPI_CLONE call as long prophecized by the code comments.
This last step cannot be in a separate patch because with the removal of the s->parameters copy, something needs to be done to set the has_* fields of the temporary structure so it can be validated. That something is already embedded in the MigrateSetParameters -> MigrationParameters conversion due to the former having the booleans properly set by QAPI.
MigrateSetParameters and MigrationParameters are obviously not of the same type, so the QAPI_CLONE call will use the common base MigrationConfigBase. The only remaining fields will be the TLS options which will be set inline.
For the validation done at migration_check_object(), introduce a new copy, to make sure we're not operating on the s->parameters and leaving it dirty with the has_* fields set.
--- migration/migration.c | 22 +++++- migration/options.c | 175 ++++++++++++------------------------------ 2 files changed, 67 insertions(+), 130 deletions(-)
migration/migration.c +19 -3
@@ -4051,8 +4051,6 @@ static void migration_instance_init(Object *obj)qemu_sem_init (&ms->pause_sem,0 );qemu_mutex_init (&ms->error_mutex); -migrate_params_init (&ms->parameters); -qemu_sem_init (&ms->postcopy_pause_sem,0 );qemu_sem_init (&ms->rp_state.rp_sem,0 );qemu_sem_init (&ms->rp_state.rp_pong_acks,0 );
@@ -4070,11 +4068,29 @@ static bool migration_object_check(MigrationState *ms, Error **errp) {/* Assuming all off */ bool old_caps[MIGRATION_CAPABILITY__MAX ] = {0 }; +g_autoptr (MigrationParameters) globals =NULL ; -if (!migrate_params_check (&ms->parameters, errp)) { +/* +* Copy the values that were already set via qdev properties +* (-global). +*/ + globals =QAPI_CLONE (MigrationParameters, &ms->parameters); + +/* +* Set the has_* fields because migrate_params_check() only +* validates new fields. +*/ +migrate_params_init (globals); + +if (!migrate_params_check (globals, errp)) {return false; } +/* +* After the validation succeeds, there's no need to apply the +* 'globals' because the values are already in s->config. +*/ +return migrate_caps_check (old_caps, ms->capabilities, errp); }
migration/options.c +48 -127
226 lines not on this page · raw message
Signature · 1 line
2.35.3
RFC PATCH 7/13 migration: Introduce new MigrationConfig structure
In reply to Fabiano RosasAdd a new migration structure to consolidate the capabilities and parameters. This structure will be used in place of the s->parameters and s->capabilities data structures in the next few patches.
The QAPI migration types now look like this:
/* options previously known as parameters */
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
} }
/* for compat with query-migrate-parameters */
{ 'struct': 'MigrationParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'str' format>
} }
/* for compat with migrate-set-parameters */
{ 'struct': 'MigrateSetParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'StrOrNull' format>
} }
/* to replace MigrationParameters in the MigrationState */ { 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
} }
The above keeps the query/set-parameters commands stable. For the capabilities as well as the options added in the future, we have a choice of where to put them:
1) In MigrationConfigBase, this means that the existing query/set-parameters commands will be updated to deal with capabilities/new options.
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
<capabilities>
<new opts>
} }
{ 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
} }2) In MigrationConfig, this means that the existing commands will be frozen in time.
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
} }
{ 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
<capabilities>
<new opts>
} }For now, I've chosen the option 1, all capabilities and new options go into MigrationConfigBase. This gives the option to keep the existing commands for as long as we'd like.
Note that the query/set capabilities commands will have to go, we can treat parameters as generic configuration options, but capabilities are just too different.
--- qapi/migration.json | 163 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 160 insertions(+), 3 deletions(-)
qapi/migration.json +160 -3
@@ -1063,10 +1063,108 @@ # @tls-creds, @tls-hostname are mandatory and a valid string is # expected. (Since 10.1) # +# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length +# Encoding). This feature allows us to minimize migration traffic +# for certain work loads, by sending compressed difference of the +# pages +# +# @rdma-pin-all: Controls whether or not the entire VM memory +# footprint is mlock()'d on demand or all at once. Refer to +# docs/rdma.txt for usage. Disabled by default. (since 2.0) +# +# @zero-blocks: During storage migration encode blocks of zeroes +# efficiently. This essentially saves 1MB of zeroes per block on +# the wire. Enabling requires source and target VM to support +# this feature. To enable it is sufficient to enable the +# capability on the source VM. The feature is disabled by +# default. (since 1.6) +# +# @events: generate events for each migration state change (since 2.4) +# +# @auto-converge: If enabled, QEMU will automatically throttle down +# the guest to speed up convergence of RAM migration. (since 1.6) +# +# @postcopy-ram: Start executing on the migration target before all of +# RAM has been migrated, pulling the remaining pages along as +# needed. The capacity must have the same setting on both source +# and target or migration will not even start. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.6) +# +# @x-colo: If enabled, migration will never end, and the state of the +# VM on the primary side will be migrated continuously to the VM +# on secondary side, this process is called COarse-Grain LOck +# Stepping (COLO) for Non-stop Service. (since 2.8) +# +# @release-ram: if enabled, qemu will free the migrated ram pages on +# the source during postcopy-ram migration. (since 2.9) +# +# @return-path: If enabled, migration will use the return path even +# for precopy. (since 2.10) +# +# @pause-before-switchover: Pause outgoing migration before +# serialising device state and before disabling block IO (since +# 2.11) +# +# @multifd: Use more than one fd for migration (since 4.0) +# +# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. +# (since 2.12) +# +# @postcopy-blocktime: Calculate downtime for postcopy live migration +# (since 3.0) +# +# @late-block-activate: If enabled, the destination will not activate +# block devices (and thus take locks) immediately at the end of +# migration. (since 3.0) +# +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory +# that is accessible on the destination machine. (since 4.0) +# +# @validate-uuid: Send the UUID of the source to allow the destination +# to ensure it is the same. (since 4.2) +# +# @background-snapshot: If enabled, the migration stream will be a +# snapshot of the VM exactly at the point when the migration +# procedure starts. The VM RAM is saved with running VM. +# (since 6.0) +# +# @zero-copy-send: Controls behavior on sending memory pages on +# migration. When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. Requires that QEMU be +# permitted to use locked memory for guest RAM pages. (since 7.1) +# +# @postcopy-preempt: If enabled, the migration process will allow +# postcopy requests to preempt precopy stream, so postcopy +# requests will be handled faster. This is a performance feature +# and should not affect the correctness of postcopy migration. +# (since 7.1) +# +# @switchover-ack: If enabled, migration will not stop the source VM +# and complete the migration until an ACK is received from the +# destination that it's OK to do so. Exactly when this ACK is +# sent depends on the migrated devices that use this feature. For +# example, a device can use it to make sure some of its data is +# sent and loaded in the destination before doing switchover. +# This can reduce downtime if devices that support this capability +# are present. 'return-path' capability must be enabled to use +# it. (since 8.1) +# +# @dirty-limit: If enabled, migration will throttle vCPUs as needed to +# keep their dirty page rate within @vcpu-dirty-limit. This can +# improve responsiveness of large guests during live migration, +# and can result in more stable read performance. Requires KVM +# with accelerator property "dirty-ring-size" set. (Since 8.1) +# +# @mapped-ram: Migrate using fixed offsets in the migration file for +# each RAM page. Requires a migration URI that supports seeking, +# such as a file. (since 9.0) +# # Features: # -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. +# @unstable: Members @x-checkpoint-delay, @x-vcpu-dirty-limit-period, +# @x-colo and @x-ignore-shared are experimental. +# @deprecated: Member @zero-blocks is deprecated as being part of +# block migration which was already removed. # # Since: 10.1 ##
@@ -1099,7 +1197,29 @@ '*mode': 'MigMode', '*zero-page-detection': 'ZeroPageDetection', '*direct-io': 'bool', - '*tls': 'bool' } } + '*tls': 'bool', + '*xbzrle': 'bool', + '*rdma-pin-all': 'bool', + '*auto-converge': 'bool', + '*zero-blocks': { 'type': 'bool', 'features': [ 'deprecated' ] }, + '*events': 'bool', + '*postcopy-ram': 'bool', + '*x-colo': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*release-ram': 'bool', + '*return-path': 'bool', + '*pause-before-switchover': 'bool', + '*multifd': 'bool', + '*dirty-bitmaps': 'bool', + '*postcopy-blocktime': 'bool', + '*late-block-activate': 'bool', + '*x-ignore-shared': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*validate-uuid': 'bool', + '*background-snapshot': 'bool', + '*zero-copy-send': 'bool', + '*postcopy-preempt': 'bool', + '*switchover-ack': 'bool', + '*dirty-limit': 'bool', + '*mapped-ram': 'bool' } } ## # @MigrationParameters:
@@ -2395,3 +2515,40 @@ 'data': { 'job-id': 'str', 'tag': 'str', 'devices': ['str'] } } + +## +# @MigrationConfig: +# +# Migration configuration options +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7) +# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9. +# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# +# Since: 10.1 +## +{ 'struct': 'MigrationConfig', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'str', + '*tls-hostname': 'str', + '*tls-authz': 'str' } }
Signature · 1 line
2.35.3
Re: [RFC PATCH 07/13] migration: Introduce new MigrationConfig structure
Quoted 7 lines · Fabiano Rosas
Add a new migration structure to consolidate the capabilities and parameters. This structure will be used in place of the
s->parametersands->capabilitiesdata structures in the next few patches.The QAPI migration types now look like this:
/* options previously known as parameters */
Configuration previously known as parameters less the TLS stuff.
Quoted 13 lines
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
} }/* for compat with query-migrate-parameters */
{ 'struct': 'MigrationParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'str' format>
} }
Quoted 6 lines
/* for compat with migrate-set-parameters */
{ 'struct': 'MigrateSetParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'StrOrNull' format>
} }
Yes, this is the state since PATCH 05.
Quoted 6 lines
/* to replace MigrationParameters in the MigrationState */ { 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
} }
This is new in this patch.
Your description doesn't cover optionalness. Here's my understanding:
* MigrationSetParameters has optional members, because migrate-set-parameters needs that.
* MigrationParameters would ideally have these members non-optional, because query-migrate-parameters wants that.
* But to enable sharing via common base type MigrationConfigBase, we accept unwanted optional in MigrationParameters and thus query-migrate-parameters.
* This doesn't apply to the non-shared members of MigrationParameters, so you made them non-optional. These are @tls-creds, @tls-hostname, @tls-authz.
* But in MigrationConfig they're optional again, because "empty string means absent" is silly; we want "NULL means absent".
Correct?
Up to here, this enables cleanup of some "empty string means absent" silliness in later patches.
The remainder is about unifying capabilities into parameters. I'd split the patch (but I'm a compulsive patch splitter).
Quoted 37 lines
The above keeps the query/set-parameters commands stable. For the capabilities as well as the options added in the future, we have a choice of where to put them:
1) In MigrationConfigBase, this means that the existing query/set-parameters commands will be updated to deal with capabilities/new options.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> <capabilities> <new opts> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> } }2) In MigrationConfig, this means that the existing commands will be frozen in time.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> <capabilities> <new opts> } }
Quoted 3 lines
For now, I've chosen the option 1, all capabilities and new options go into MigrationConfigBase. This gives the option to keep the existing commands for as long as we'd like.
Perhaps this would be slightly easier to digest for the reader if you talked just about capabilities at first. Once that's understood, mention we have the same choice for new configuration bits.
Quoted 3 lines
Note that the query/set capabilities commands will have to go, we can treat parameters as generic configuration options, but capabilities are just too different.
I think the argument is that migration capabilities are a pointless interface complication. One mechanism (parameters) is better than two (parameters and capabilities).
Quoted 138 lines
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 163 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 160 insertions(+), 3 deletions(-)qapi/migration.json +113
-3@@ -1063,10 +1063,108 @@ # @tls-creds, @tls-hostname are mandatory and a valid string is # expected. (Since 10.1) # +# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length +# Encoding). This feature allows us to minimize migration traffic +# for certain work loads, by sending compressed difference of the +# pages +# +# @rdma-pin-all: Controls whether or not the entire VM memory +# footprint is mlock()'d on demand or all at once. Refer to +# docs/rdma.txt for usage. Disabled by default. (since 2.0) +# +# @zero-blocks: During storage migration encode blocks of zeroes +# efficiently. This essentially saves 1MB of zeroes per block on +# the wire. Enabling requires source and target VM to support +# this feature. To enable it is sufficient to enable the +# capability on the source VM. The feature is disabled by +# default. (since 1.6) +# +# @events: generate events for each migration state change (since 2.4) +# +# @auto-converge: If enabled, QEMU will automatically throttle down +# the guest to speed up convergence of RAM migration. (since 1.6) +# +# @postcopy-ram: Start executing on the migration target before all of +# RAM has been migrated, pulling the remaining pages along as +# needed. The capacity must have the same setting on both source +# and target or migration will not even start. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.6) +# +# @x-colo: If enabled, migration will never end, and the state of the +# VM on the primary side will be migrated continuously to the VM +# on secondary side, this process is called COarse-Grain LOck +# Stepping (COLO) for Non-stop Service. (since 2.8) +# +# @release-ram: if enabled, qemu will free the migrated ram pages on +# the source during postcopy-ram migration. (since 2.9) +# +# @return-path: If enabled, migration will use the return path even +# for precopy. (since 2.10) +# +# @pause-before-switchover: Pause outgoing migration before +# serialising device state and before disabling block IO (since +# 2.11) +# +# @multifd: Use more than one fd for migration (since 4.0) +# +# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. +# (since 2.12) +# +# @postcopy-blocktime: Calculate downtime for postcopy live migration +# (since 3.0) +# +# @late-block-activate: If enabled, the destination will not activate +# block devices (and thus take locks) immediately at the end of +# migration. (since 3.0) +# +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory +# that is accessible on the destination machine. (since 4.0) +# +# @validate-uuid: Send the UUID of the source to allow the destination +# to ensure it is the same. (since 4.2) +# +# @background-snapshot: If enabled, the migration stream will be a +# snapshot of the VM exactly at the point when the migration +# procedure starts. The VM RAM is saved with running VM. +# (since 6.0) +# +# @zero-copy-send: Controls behavior on sending memory pages on +# migration. When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. Requires that QEMU be +# permitted to use locked memory for guest RAM pages. (since 7.1) +# +# @postcopy-preempt: If enabled, the migration process will allow +# postcopy requests to preempt precopy stream, so postcopy +# requests will be handled faster. This is a performance feature +# and should not affect the correctness of postcopy migration. +# (since 7.1) +# +# @switchover-ack: If enabled, migration will not stop the source VM +# and complete the migration until an ACK is received from the +# destination that it's OK to do so. Exactly when this ACK is +# sent depends on the migrated devices that use this feature. For +# example, a device can use it to make sure some of its data is +# sent and loaded in the destination before doing switchover. +# This can reduce downtime if devices that support this capability +# are present. 'return-path' capability must be enabled to use +# it. (since 8.1) +# +# @dirty-limit: If enabled, migration will throttle vCPUs as needed to +# keep their dirty page rate within @vcpu-dirty-limit. This can +# improve responsiveness of large guests during live migration, +# and can result in more stable read performance. Requires KVM +# with accelerator property "dirty-ring-size" set. (Since 8.1) +# +# @mapped-ram: Migrate using fixed offsets in the migration file for +# each RAM page. Requires a migration URI that supports seeking, +# such as a file. (since 9.0) +# # Features: # -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. +# @unstable: Members @x-checkpoint-delay, @x-vcpu-dirty-limit-period, +# @x-colo and @x-ignore-shared are experimental. +# @deprecated: Member @zero-blocks is deprecated as being part of +# block migration which was already removed. # # Since: 10.1 ##@@ -1099,7 +1197,29 @@ '*mode': 'MigMode', '*zero-page-detection': 'ZeroPageDetection', '*direct-io': 'bool', - '*tls': 'bool' } } + '*tls': 'bool', + '*xbzrle': 'bool', + '*rdma-pin-all': 'bool', + '*auto-converge': 'bool', + '*zero-blocks': { 'type': 'bool', 'features': [ 'deprecated' ] }, + '*events': 'bool', + '*postcopy-ram': 'bool', + '*x-colo': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*release-ram': 'bool', + '*return-path': 'bool', + '*pause-before-switchover': 'bool', + '*multifd': 'bool', + '*dirty-bitmaps': 'bool',
qapi/migration.json line 1222
Quoted 10 lines
qapi/migration.json +10
-0+ '*postcopy-blocktime': 'bool', + '*late-block-activate': 'bool', + '*x-ignore-shared': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*validate-uuid': 'bool', + '*background-snapshot': 'bool', + '*zero-copy-send': 'bool', + '*postcopy-preempt': 'bool', + '*switchover-ack': 'bool', + '*dirty-limit': 'bool', + '*mapped-ram': 'bool' } }
This is part 2 "unify capabilities into parameters".
Missing in your series: deprecate migrate-set-capabilities and query-migrate-capabilities.
Quoted 11 lines
qapi/migration.json +4
-0## # @MigrationParameters:@@ -2395,3 +2515,40 @@ 'data': { 'job-id': 'str', 'tag': 'str', 'devices': ['str'] } } + +## +# @MigrationConfig: +#
qapi/migration.json line 2531
Quoted 10 lines
qapi/migration.json +10
-0+# Migration configuration options +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data +# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7)
Is "empty string" expected to occur here?
qapi/migration.json line 2540
Quoted 9 lines
qapi/migration.json +9
-0+# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9.
Likewise.
Quoted 4 lines
qapi/migration.json +4
-0+# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted
qapi/migration.json line 2554
Quoted 10 lines
qapi/migration.json +10
-0+# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# +# Since: 10.1 +## +{ 'struct': 'MigrationConfig', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'str', + '*tls-hostname': 'str', + '*tls-authz': 'str' } }
This is part 1 "enable cleanup".
Re: [RFC PATCH 07/13] migration: Introduce new MigrationConfig structure
Markus Armbruster <[email protected]> writes:
Markus, sorry for the delay here. I had vacations and holidays, plus a pile of patches to review.
Quoted 41 lines
Quoted 7 lines · Fabiano Rosas
Add a new migration structure to consolidate the capabilities and parameters. This structure will be used in place of the
s->parametersands->capabilitiesdata structures in the next few patches.The QAPI migration types now look like this:
/* options previously known as parameters */
Configuration previously known as parameters less the TLS stuff.
Quoted 13 lines
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
} }/* for compat with query-migrate-parameters */
{ 'struct': 'MigrationParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'str' format>
} }Quoted 6 lines
/* for compat with migrate-set-parameters */
{ 'struct': 'MigrateSetParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'StrOrNull' format>
} }Yes, this is the state since PATCH 05.
Quoted 6 lines
/* to replace MigrationParameters in the MigrationState */ { 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
} }
Quoted 7 lines
This is new in this patch.
Your description doesn't cover optionalness. Here's my understanding:
* MigrationSetParameters has optional members, because migrate-set-parameters needs that.
Yes.
Quoted 3 lines
* MigrationParameters would ideally have these members non-optional, because query-migrate-parameters wants that.
Yes.
Quoted 4 lines
* But to enable sharing via common base type MigrationConfigBase, we accept unwanted optional in MigrationParameters and thus query-migrate-parameters.
Yes.
Quoted 4 lines
* This doesn't apply to the non-shared members of MigrationParameters, so you made them non-optional. These are @tls-creds, @tls-hostname, @tls-authz.
Yes.
Quoted 3 lines
* But in MigrationConfig they're optional again, because "empty string means absent" is silly; we want "NULL means absent".
Yes. But mostly because MigrationConfig will become the type for the new '*config' argument to migrate/migrate_incoming (patches 12 & 13) and we want to keep all members optional. Otherwise the user would have to pass all ~50 migration options in every migrate command, which is bad IMO.
Quoted 45 lines
Correct?
Up to here, this enables cleanup of some "empty string means absent" silliness in later patches.
The remainder is about unifying capabilities into parameters. I'd split the patch (but I'm a compulsive patch splitter).
Quoted 37 lines
The above keeps the query/set-parameters commands stable. For the capabilities as well as the options added in the future, we have a choice of where to put them:
1) In MigrationConfigBase, this means that the existing query/set-parameters commands will be updated to deal with capabilities/new options.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> <capabilities> <new opts> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> } }2) In MigrationConfig, this means that the existing commands will be frozen in time.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> <capabilities> <new opts> } }
Quoted 8 lines
Quoted 3 lines
For now, I've chosen the option 1, all capabilities and new options go into MigrationConfigBase. This gives the option to keep the existing commands for as long as we'd like.
Perhaps this would be slightly easier to digest for the reader if you talked just about capabilities at first. Once that's understood, mention we have the same choice for new configuration bits.
Ok, I'll reorganize, along with the other comments you've made.
Quoted 8 lines
Quoted 3 lines
Note that the query/set capabilities commands will have to go, we can treat parameters as generic configuration options, but capabilities are just too different.
I think the argument is that migration capabilities are a pointless interface complication. One mechanism (parameters) is better than two (parameters and capabilities).
Yes, that's the main point indeed.
Quoted 151 lines
Quoted 138 lines
Signed-off-by Fabiano Rosas <[email protected]>--- qapi/migration.json | 163 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 160 insertions(+), 3 deletions(-)qapi/migration.json +113
-3@@ -1063,10 +1063,108 @@ # @tls-creds, @tls-hostname are mandatory and a valid string is # expected. (Since 10.1) # +# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length +# Encoding). This feature allows us to minimize migration traffic +# for certain work loads, by sending compressed difference of the +# pages +# +# @rdma-pin-all: Controls whether or not the entire VM memory +# footprint is mlock()'d on demand or all at once. Refer to +# docs/rdma.txt for usage. Disabled by default. (since 2.0) +# +# @zero-blocks: During storage migration encode blocks of zeroes +# efficiently. This essentially saves 1MB of zeroes per block on +# the wire. Enabling requires source and target VM to support +# this feature. To enable it is sufficient to enable the +# capability on the source VM. The feature is disabled by +# default. (since 1.6) +# +# @events: generate events for each migration state change (since 2.4) +# +# @auto-converge: If enabled, QEMU will automatically throttle down +# the guest to speed up convergence of RAM migration. (since 1.6) +# +# @postcopy-ram: Start executing on the migration target before all of +# RAM has been migrated, pulling the remaining pages along as +# needed. The capacity must have the same setting on both source +# and target or migration will not even start. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.6) +# +# @x-colo: If enabled, migration will never end, and the state of the +# VM on the primary side will be migrated continuously to the VM +# on secondary side, this process is called COarse-Grain LOck +# Stepping (COLO) for Non-stop Service. (since 2.8) +# +# @release-ram: if enabled, qemu will free the migrated ram pages on +# the source during postcopy-ram migration. (since 2.9) +# +# @return-path: If enabled, migration will use the return path even +# for precopy. (since 2.10) +# +# @pause-before-switchover: Pause outgoing migration before +# serialising device state and before disabling block IO (since +# 2.11) +# +# @multifd: Use more than one fd for migration (since 4.0) +# +# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. +# (since 2.12) +# +# @postcopy-blocktime: Calculate downtime for postcopy live migration +# (since 3.0) +# +# @late-block-activate: If enabled, the destination will not activate +# block devices (and thus take locks) immediately at the end of +# migration. (since 3.0) +# +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory +# that is accessible on the destination machine. (since 4.0) +# +# @validate-uuid: Send the UUID of the source to allow the destination +# to ensure it is the same. (since 4.2) +# +# @background-snapshot: If enabled, the migration stream will be a +# snapshot of the VM exactly at the point when the migration +# procedure starts. The VM RAM is saved with running VM. +# (since 6.0) +# +# @zero-copy-send: Controls behavior on sending memory pages on +# migration. When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. Requires that QEMU be +# permitted to use locked memory for guest RAM pages. (since 7.1) +# +# @postcopy-preempt: If enabled, the migration process will allow +# postcopy requests to preempt precopy stream, so postcopy +# requests will be handled faster. This is a performance feature +# and should not affect the correctness of postcopy migration. +# (since 7.1) +# +# @switchover-ack: If enabled, migration will not stop the source VM +# and complete the migration until an ACK is received from the +# destination that it's OK to do so. Exactly when this ACK is +# sent depends on the migrated devices that use this feature. For +# example, a device can use it to make sure some of its data is +# sent and loaded in the destination before doing switchover. +# This can reduce downtime if devices that support this capability +# are present. 'return-path' capability must be enabled to use +# it. (since 8.1) +# +# @dirty-limit: If enabled, migration will throttle vCPUs as needed to +# keep their dirty page rate within @vcpu-dirty-limit. This can +# improve responsiveness of large guests during live migration, +# and can result in more stable read performance. Requires KVM +# with accelerator property "dirty-ring-size" set. (Since 8.1) +# +# @mapped-ram: Migrate using fixed offsets in the migration file for +# each RAM page. Requires a migration URI that supports seeking, +# such as a file. (since 9.0) +# # Features: # -# @unstable: Members @x-checkpoint-delay and -# @x-vcpu-dirty-limit-period are experimental. +# @unstable: Members @x-checkpoint-delay, @x-vcpu-dirty-limit-period, +# @x-colo and @x-ignore-shared are experimental. +# @deprecated: Member @zero-blocks is deprecated as being part of +# block migration which was already removed. # # Since: 10.1 ##@@ -1099,7 +1197,29 @@ '*mode': 'MigMode', '*zero-page-detection': 'ZeroPageDetection', '*direct-io': 'bool', - '*tls': 'bool' } } + '*tls': 'bool', + '*xbzrle': 'bool', + '*rdma-pin-all': 'bool', + '*auto-converge': 'bool', + '*zero-blocks': { 'type': 'bool', 'features': [ 'deprecated' ] }, + '*events': 'bool', + '*postcopy-ram': 'bool', + '*x-colo': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*release-ram': 'bool', + '*return-path': 'bool', + '*pause-before-switchover': 'bool', + '*multifd': 'bool', + '*dirty-bitmaps': 'bool',qapi/migration.json line 1222
Quoted 10 lines
qapi/migration.json +10
-0+ '*postcopy-blocktime': 'bool', + '*late-block-activate': 'bool', + '*x-ignore-shared': { 'type': 'bool', 'features': [ 'unstable' ] }, + '*validate-uuid': 'bool', + '*background-snapshot': 'bool', + '*zero-copy-send': 'bool', + '*postcopy-preempt': 'bool', + '*switchover-ack': 'bool', + '*dirty-limit': 'bool', + '*mapped-ram': 'bool' } }This is part 2 "unify capabilities into parameters".
Quoted 3 lines
Missing in your series: deprecate migrate-set-capabilities and query-migrate-capabilities.
I'll add when repost.
Quoted 15 lines
Quoted 15 lines
qapi/migration.json +8
-0## # @MigrationParameters:@@ -2395,3 +2515,40 @@ 'data': { 'job-id': 'str', 'tag': 'str', 'devices': ['str'] } } + +## +# @MigrationConfig: +# +# Migration configuration options +# +# @tls-creds: ID of the 'tls-creds' object that provides credentials +# for establishing a TLS connection over the migration data
Quoted 9 lines
qapi/migration.json line 2531
Quoted 6 lines
qapi/migration.json +6
-0+# channel. On the outgoing side of the migration, the credentials +# must be for a 'client' endpoint, while for the incoming side the +# credentials must be for a 'server' endpoint. Setting this to a +# non-empty string enables TLS for all migrations. An empty +# string means that QEMU will use plain text mode for migration, +# rather than TLS. (Since 2.7)Is "empty string" expected to occur here?
Yes. Up until this point I'm keeping everything compatible, only switching to use MigrationConfig internally. Come patch 12, when we expose this type externally, it will be time to decide whether to declare that the new way of setting migration configuration options treats the empty string as invalid. I haven't made the call yet because I wanted some feedback on the new commands, backward compatibility, etc.
Quoted 12 lines
qapi/migration.json line 2540
Quoted 9 lines
qapi/migration.json +9
-0+# +# @tls-hostname: migration target's hostname for validating the +# server's x509 certificate identity. If empty, QEMU will use the +# hostname from the migration URI, if any. A non-empty value is +# required when using x509 based TLS credentials and the migration +# URI does not include a hostname, such as fd: or exec: based +# migration. (Since 2.7) +# +# Note: empty value works only since 2.9.Likewise.
Same here.
Quoted 8 lines
Quoted 8 lines
qapi/migration.json +8
-0+# +# @tls-authz: ID of the 'authz' object subclass that provides access +# control checking of the TLS x509 certificate distinguished name. +# This object is only resolved at time of use, so can be deleted +# and recreated on the fly while the migration server is active. +# If missing, it will default to denying access (Since 4.0) +# +# Since: 10.1
Quoted 8 lines
qapi/migration.json line 2554
Quoted 6 lines
qapi/migration.json +6
-0+## +{ 'struct': 'MigrationConfig', + 'base': 'MigrationConfigBase', + 'data': { '*tls-creds': 'str', + '*tls-hostname': 'str', + '*tls-authz': 'str' } }This is part 1 "enable cleanup".
Ok.
Thanks for the review!
Re: [RFC PATCH 07/13] migration: Introduce new MigrationConfig structure
Quoted 4 lines · Fabiano Rosas
Markus Armbruster <[email protected]> writes:
Markus, sorry for the delay here. I had vacations and holidays, plus a pile of patches to review.
No problem. Hope you enjoyed your time off!
Quoted 71 lines
Quoted 41 lines
Quoted 7 lines · Fabiano Rosas
Add a new migration structure to consolidate the capabilities and parameters. This structure will be used in place of the
s->parametersands->capabilitiesdata structures in the next few patches.The QAPI migration types now look like this:
/* options previously known as parameters */
Configuration previously known as parameters less the TLS stuff.
Quoted 13 lines
{ 'struct': 'MigrationConfigBase',
'data': {
<parameters>
} }/* for compat with query-migrate-parameters */
{ 'struct': 'MigrationParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'str' format>
} }Quoted 6 lines
/* for compat with migrate-set-parameters */
{ 'struct': 'MigrateSetParameters',
'base': 'MigrationConfigBase',
'data': {
<TLS options in 'StrOrNull' format>
} }Yes, this is the state since PATCH 05.
Quoted 6 lines
/* to replace MigrationParameters in the MigrationState */ { 'struct': 'MigrationConfig',
'base': 'MigrationConfigBase'
'data': {
<TLS options in 'str' format>
} }Quoted 7 lines
This is new in this patch.
Your description doesn't cover optionalness. Here's my understanding:
* MigrationSetParameters has optional members, because migrate-set-parameters needs that.
Yes.
Quoted 3 lines
* MigrationParameters would ideally have these members non-optional, because query-migrate-parameters wants that.
Yes.
Quoted 4 lines
* But to enable sharing via common base type MigrationConfigBase, we accept unwanted optional in MigrationParameters and thus query-migrate-parameters.
Yes.
Quoted 4 lines
* This doesn't apply to the non-shared members of MigrationParameters, so you made them non-optional. These are @tls-creds, @tls-hostname, @tls-authz.
Yes.
Quoted 8 lines
Quoted 3 lines
* But in MigrationConfig they're optional again, because "empty string means absent" is silly; we want "NULL means absent".
Yes. But mostly because MigrationConfig will become the type for the new '*config' argument to migrate/migrate_incoming (patches 12 & 13) and we want to keep all members optional. Otherwise the user would have to pass all ~50 migration options in every migrate command, which is bad IMO.
Got it.
Quoted 60 lines
Quoted 45 lines
Correct?
Up to here, this enables cleanup of some "empty string means absent" silliness in later patches.
The remainder is about unifying capabilities into parameters. I'd split the patch (but I'm a compulsive patch splitter).
Quoted 37 lines
The above keeps the query/set-parameters commands stable. For the capabilities as well as the options added in the future, we have a choice of where to put them:
1) In MigrationConfigBase, this means that the existing query/set-parameters commands will be updated to deal with capabilities/new options.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> <capabilities> <new opts> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> } }2) In MigrationConfig, this means that the existing commands will be frozen in time.
{ 'struct': 'MigrationConfigBase', 'data': { <parameters> } } { 'struct': 'MigrationConfig', 'base': 'MigrationConfigBase' 'data': { <TLS options in 'str' format> <capabilities> <new opts> } }Quoted 8 lines
Quoted 3 lines
For now, I've chosen the option 1, all capabilities and new options go into MigrationConfigBase. This gives the option to keep the existing commands for as long as we'd like.
Perhaps this would be slightly easier to digest for the reader if you talked just about capabilities at first. Once that's understood, mention we have the same choice for new configuration bits.
Ok, I'll reorganize, along with the other comments you've made.
Quoted 4 lines
Quoted 3 lines
Note that the query/set capabilities commands will have to go, we can treat parameters as generic configuration options, but capabilities are just too different.
Quoted 6 lines
Quoted 4 lines
I think the argument is that migration capabilities are a pointless interface complication. One mechanism (parameters) is better than two (parameters and capabilities).
Yes, that's the main point indeed.
Perhaps you can make this point more prominently.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Signed-off-by Fabiano Rosas <[email protected]>
[...]
RFC PATCH 8/13 migration: Replace s->parameters with s->config
In reply to Fabiano RosasThe concepts of 'parameters' and 'capabilities' are almost functionally identical and there is a consensus that their usage is confusing, particularly when it comes to determine to which category a new option should belong.
The new MigrationConfig type was introduced to eliminate the distinction between parameters and capabilities and contains the union of both groups.
Replace internal usages of MigrationParameters with MigrationConfig.
For the user-facing migrate-set-parameters and query-migrate-parameters, the old types MigrateSetParameters and MigrationParameters are still used.
Note that migrate_params_check (now migrate_config_check) will be used in the future also for capabilities validation and for any future command that uses MigrationConfig.
--- migration/migration.c | 25 ++- migration/migration.h | 4 +- migration/options.c | 371 ++++++++++++++++++++---------------------- migration/options.h | 8 +- migration/ram.c | 4 +- 5 files changed, 192 insertions(+), 220 deletions(-)
migration/migration.c +12 -13
@@ -119,7 +119,7 @@ static void migration_downtime_start(MigrationState *s)/* * This is unfortunate: incoming migration actually needs the outgoing * migration state (MigrationState) to be there too, e.g. to query -* capabilities, parameters, using locks, setup errors, etc. +* options, using locks, setup errors, etc. * * NOTE: when calling this, making sure current_migration exists and not * been freed yet! Otherwise trying to access the refcount is already
@@ -1664,7 +1664,7 @@ void migration_remove_notifier(NotifierWithReturn *notify)int migration_call_notifiers (MigrationState *s,MigrationEventType type,Error **errp) { -MigMode mode = s->parameters.mode; +MigMode mode = s->config.mode;MigrationEvent e;int ret;
@@ -1735,7 +1735,7 @@ bool migration_thread_is_self(void)bool migrate_mode_is_cpr (MigrationState *s) { -MigMode mode = s->parameters.mode; +MigMode mode = s->config.mode;return mode ==MIG_MODE_CPR_REBOOT || mode ==MIG_MODE_CPR_TRANSFER ; }
@@ -1750,9 +1750,8 @@ int migrate_init(MigrationState *s, Error **errp) }/* -* Reinitialise all migration state, except -* parameters/capabilities that the user set, and -* locks. +* Reinitialise all migration state, except options that the user +* set, and locks. */ s->to_dst_file =NULL ; s->state =MIGRATION_STATUS_NONE ;
@@ -2212,7 +2211,7 @@ void qmp_migrate(const char *uri, bool has_channels,return ; } -if (s->parameters.mode ==MIG_MODE_CPR_TRANSFER && !cpr_channel) { +if (s->config.mode ==MIG_MODE_CPR_TRANSFER && !cpr_channel) {error_setg (errp,"missing 'cpr' migration channel" );return ; }
@@ -2237,7 +2236,7 @@ void qmp_migrate(const char *uri, bool has_channels,* in which case the target will not listen for the incoming migration * connection, so qmp_migrate_finish will fail to connect, and then recover. */ -if (s->parameters.mode ==MIG_MODE_CPR_TRANSFER ) { +if (s->config.mode ==MIG_MODE_CPR_TRANSFER ) {migrate_hup_add (s,cpr_state_ioc (), (GSourceFunc )qmp_migrate_finish_cb,QAPI_CLONE (MigrationAddress, addr));
@@ -4068,21 +4067,21 @@ static bool migration_object_check(MigrationState *ms, Error **errp) {/* Assuming all off */ bool old_caps[MIGRATION_CAPABILITY__MAX ] = {0 }; -g_autoptr (MigrationParameters) globals =NULL ; +g_autoptr (MigrationConfig) globals =NULL ;/* * Copy the values that were already set via qdev properties * (-global). */ - globals =QAPI_CLONE (MigrationParameters, &ms->parameters); + globals =QAPI_CLONE (MigrationConfig, &ms->config);/* -* Set the has_* fields because migrate_params_check() only +* Set the has_* fields because migrate_config_check() only * validates new fields. */ -migrate_params_init (globals); +migrate_config_init (globals); -if (!migrate_params_check (globals, errp)) { +if (!migrate_config_check (globals, errp)) {return false; }
migration/migration.h +1 -3
@@ -317,9 +317,7 @@ struct MigrationState {*/ uint64_t threshold_size; -/* params from 'migrate-set-parameters' */ -MigrationParameters parameters; - +MigrationConfig config;MigrationStatus state;/* State related to return path */
migration/options.c +175 -196
733 lines not on this page · raw message
migration/options.h +2 -6
@@ -59,8 +59,6 @@ bool migrate_tls(void);bool migrate_caps_check (bool *old_caps,bool *new_caps,Error **errp); -/* parameters */ -const BitmapMigrationNodeAliasList *migrate_block_bitmap_mapping (void );bool migrate_has_block_bitmap_mapping (void );
@@ -86,8 +84,6 @@ const char *migrate_tls_hostname(void);uint64_t migrate_xbzrle_cache_size (void );ZeroPageDetection migrate_zero_page_detection (void ); -/* parameters helpers */ - -bool migrate_params_check (MigrationParameters *params,Error **errp); -void migrate_params_init (MigrationParameters *params); +bool migrate_config_check (MigrationConfig *params,Error **errp); +void migrate_config_init (MigrationConfig *params);#endif
migration/ram.c +2 -2
@@ -973,11 +973,11 @@ static void migration_dirty_limit_guest(void)* vcpu-dirty-limit untouched. */ if (dirtylimit_in_service () && - quota_dirtyrate == s->parameters.vcpu_dirty_limit) { + quota_dirtyrate == s->config.vcpu_dirty_limit) {return ; } - quota_dirtyrate = s->parameters.vcpu_dirty_limit; + quota_dirtyrate = s->config.vcpu_dirty_limit;/* * Set all vCPU a quota dirtyrate, note that the second
Signature · 1 line
2.35.3
RFC PATCH 9/13 migration: Do away with usage of QERR_INVALID_PARAMETER_VALUE
In reply to Fabiano RosasNow that we're removing mentions of 'parameters' from migration code, the "Parameter X expects..." message is a little confusing, although the "parameter" there refers to the QMP command parameter. There's also some instances which don't make sense grammatically.
Remove all the usage of QERR_INVALID_PARAMETER_VALUE.
--- migration/migration.c | 3 +-- migration/options.c | 56 +++++++++++++++--------------------------- migration/page_cache.c | 6 ++--- migration/ram.c | 3 +-- 4 files changed, 24 insertions(+), 44 deletions(-)
migration/migration.c +1 -2
@@ -2281,8 +2281,7 @@ static void qmp_migrate_finish(MigrationAddress *addr, bool resume_requested, }else if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {file_start_outgoing_migration (s, &addr->u.file, &local_err); }else { -error_setg (&local_err,QERR_INVALID_PARAMETER_VALUE ,"uri" , -"a valid migration protocol" ); +error_setg (&local_err,"uri is not a valid migration protocol" );migrate_set_state (&s->state,MIGRATION_STATUS_SETUP ,MIGRATION_STATUS_FAILED ); }
migration/options.c +20 -36
154 lines not on this page · raw message
migration/page_cache.c +2 -4
@@ -45,15 +45,13 @@ PageCache *cache_init(uint64_t new_size, size_t page_size, Error **errp)PageCache *cache;if (new_size < page_size) { -error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"cache size" , -"is smaller than one target page size" ); +error_setg (errp,"cache size is smaller than target page size" );return NULL ; }/* round down to the nearest power of 2 */ if (!is_power_of_2 (num_pages)) { -error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"cache size" , -"is not a power of two number of pages" ); +error_setg (errp,"number of pages is not a power of two" );return NULL ; }
migration/ram.c +1 -2
@@ -161,8 +161,7 @@ int xbzrle_cache_resize(uint64_t new_size, Error **errp)/* Check for truncation */ if (new_size != (size_t )new_size) { -error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"cache size" , -"exceeding address space" ); +error_setg (errp,"xbzrle cache size integer overflow" );return -1 ; }
Signature · 1 line
2.35.3
RFC PATCH 10/13 migration: Replace s->capabilities with s->config
In reply to Fabiano RosasContinuing with the removal of parameters and capabilities, replace the internal usage of s->capabilities with MigrationConfig.
To keep compatibility with the special format of the user-facing migrate-set-capabilities and query-migrate-capabilities, introduce a few compatibility routines to convert between MigrationCapabilityStatusList and MigrationConfig.
--- migration/migration.c | 4 +- migration/migration.h | 1 - migration/options.c | 361 +++++++++++++++++++++++++----------------- migration/options.h | 16 +- migration/savevm.c | 8 +- 5 files changed, 227 insertions(+), 163 deletions(-)
migration/migration.c +1 -3
@@ -4064,8 +4064,6 @@ static void migration_instance_init(Object *obj)*/ static bool migration_object_check (MigrationState *ms,Error **errp) { -/* Assuming all off */ -bool old_caps[MIGRATION_CAPABILITY__MAX ] = {0 };g_autoptr (MigrationConfig) globals =NULL ;/*
@@ -4089,7 +4087,7 @@ static bool migration_object_check(MigrationState *ms, Error **errp)* 'globals' because the values are already in s->config. */ -return migrate_caps_check (old_caps, ms->capabilities, errp); +return migrate_caps_check (&ms->config, errp); }static const TypeInfo migration_type = {
migration/migration.h +0 -1
@@ -356,7 +356,6 @@ struct MigrationState {int64_t downtime_start;int64_t downtime;int64_t expected_downtime; -bool capabilities[MIGRATION_CAPABILITY__MAX ];int64_t setup_time;/*
migration/options.c +218 -143
633 lines not on this page · raw message
migration/options.h +3 -13
@@ -1,5 +1,5 @@/* -* QEMU migration capabilities +* QEMU migration options * * Copyright (c) 2012-2023 Red Hat Inc *
@@ -23,8 +23,6 @@extern const Property migration_properties[];extern const size_t migration_properties_count; -/* capabilities */ -bool migrate_auto_converge (void );bool migrate_colo (void );bool migrate_dirty_bitmaps (void );
@@ -43,21 +41,11 @@ bool migrate_validate_uuid(void);bool migrate_xbzrle (void );bool migrate_zero_copy_send (void ); -/* -* pseudo capabilities -* -* These are functions that are used in a similar way to capabilities -* check, but they are not a capability. -*/ -bool migrate_multifd_flush_after_each_section (void );bool migrate_postcopy (void );bool migrate_rdma (void );bool migrate_tls (void ); -/* capabilities helpers */ - -bool migrate_caps_check (bool *old_caps,bool *new_caps,Error **errp);const BitmapMigrationNodeAliasList *migrate_block_bitmap_mapping (void );bool migrate_has_block_bitmap_mapping (void );
@@ -86,4 +74,6 @@ ZeroPageDetection migrate_zero_page_detection(void);bool migrate_config_check (MigrationConfig *params,Error **errp);void migrate_config_init (MigrationConfig *params); +bool migrate_config_get_cap_compat (MigrationConfig *config,int i); +bool migrate_caps_check (MigrationConfig *new,Error **errp);#endif
migration/savevm.c +5 -3
@@ -290,7 +290,8 @@ static uint32_t get_validatable_capabilities_count(void)uint32_t result =0 ;int i;for (i =0 ; i <MIGRATION_CAPABILITY__MAX ; i++) { -if (should_validate_capability (i) && s->capabilities[i]) { +if (should_validate_capability (i) && +migrate_config_get_cap_compat (&s->config, i)) { result++; } }
@@ -312,7 +313,8 @@ static int configuration_pre_save(void *opaque) state->capabilities =g_renew (MigrationCapability, state->capabilities, state->caps_count);for (i = j =0 ; i <MIGRATION_CAPABILITY__MAX ; i++) { -if (should_validate_capability (i) && s->capabilities[i]) { +if (should_validate_capability (i) && +migrate_config_get_cap_compat (&s->config, i)) { state->capabilities[j++] = i; } }
@@ -362,7 +364,7 @@ static bool configuration_validate_capabilities(SaveState *state)continue ; } source_state =test_bit (i, source_caps_bm); - target_state = s->capabilities[i]; + target_state =migrate_config_get_cap_compat (&s->config, i);if (source_state != target_state) { error_report("Capability %s is %s, but received capability is %s", MigrationCapability_str(i),
Signature · 1 line
2.35.3
RFC PATCH 11/13 migration: Merge parameters and capability checks
In reply to Fabiano RosasSince parameters and capabilities are now stored in the same structure and both use the same method to validate the input (i.e. copy, set a temporary, validate the temporary and overwrite), it's possible to use the same routine for both.
With this all distinction from capabilities and parameters is removed, except for the user-facing pieces that have to maintain compatibility.
Note that capabilities need to be validated against conflicting capabilities that are already set and also against capabilities that the user might have set on the same command. Therefore they cannot check the has_* fields, otherwise the amount of combinations would be too large.
--- migration/migration.c | 6 +- migration/options.c | 528 ++++++++++++++++++++++-------------------- migration/options.h | 1 - 3 files changed, 272 insertions(+), 263 deletions(-)
migration/migration.c +1 -5
@@ -4078,16 +4078,12 @@ static bool migration_object_check(MigrationState *ms, Error **errp)*/ migrate_config_init (globals); -if (!migrate_config_check (globals, errp)) { -return false; - } -/* * After the validation succeeds, there's no need to apply the * 'globals' because the values are already in s->config. */ -return migrate_caps_check (&ms->config, errp); +return migrate_config_check (globals, errp); }static const TypeInfo migration_type = {
migration/options.c +271 -257
689 lines not on this page · raw message
migration/options.h +0 -1
@@ -46,7 +46,6 @@ bool migrate_postcopy(void);bool migrate_rdma (void );bool migrate_tls (void ); -const BitmapMigrationNodeAliasList *migrate_block_bitmap_mapping (void );bool migrate_has_block_bitmap_mapping (void );
Signature · 1 line
2.35.3
RFC PATCH 12/13 migration: Add query/set commands for MigrationConfig
In reply to Fabiano RosasAdd the QMP commands query-migrate-config and migrate-set-config to read and write the migration configuration options.
--- migration/options.c | 24 ++++++++++++++++++++++++ migration/options.h | 2 +- qapi/migration.json | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-)
migration/options.c +24 -0
@@ -1441,3 +1441,27 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)migrate_config_apply (&tmp);migrate_post_update_config (&tmp, errp); } + +void qmp_migrate_set_config (MigrationConfig *config,Error **errp) +{ +if (!migrate_config_check (config, errp)) { +/* Invalid parameter */ +return ; + } + +migrate_config_apply (config); +migrate_post_update_config (config, errp); +} + +MigrationConfig *qmp_query_migrate_config (Error **errp) +{ +MigrationState *s =migrate_get_current (); +MigrationConfig *config =g_new0 (MigrationConfig,1 ); + +QAPI_CLONE_MEMBERS (MigrationConfig, config, &s->config); + +/* set the has_* fields for every option */ +migrate_config_init (config); + +return config; +}
migration/options.h +1 -1
@@ -72,7 +72,7 @@ uint64_t migrate_xbzrle_cache_size(void);ZeroPageDetection migrate_zero_page_detection (void );bool migrate_config_check (MigrationConfig *params,Error **errp); -void migrate_config_init (MigrationConfig *params); +void migrate_config_init (MigrationConfig *config);bool migrate_config_get_cap_compat (MigrationConfig *config,int i);bool migrate_caps_check (MigrationConfig *new,Error **errp);#endif
qapi/migration.json +42 -0
45 lines not on this page · raw message
Signature · 1 line
2.35.3
Re: [RFC PATCH 12/13] [PoC] migration: Add query/set commands for MigrationConfig
Quoted 2 lines · Fabiano Rosas
Add the QMP commands query-migrate-config and migrate-set-config to read and write the migration configuration options.
These supersede query-migrate-capabilities, query-migrate-parameters, migrate-set-capabilities, and migrate-set-parameters, right?
Quoted 44 lines
Signed-off-by Fabiano Rosas <[email protected]>--- migration/options.c | 24 ++++++++++++++++++++++++ migration/options.h | 2 +- qapi/migration.json | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-)migration/options.c +24
-0@@ -1441,3 +1441,27 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)migrate_config_apply (&tmp);migrate_post_update_config (&tmp, errp); } + +void qmp_migrate_set_config (MigrationConfig *config,Error **errp) +{ +if (!migrate_config_check (config, errp)) { +/* Invalid parameter */ +return ; + } + +migrate_config_apply (config); +migrate_post_update_config (config, errp); +} + +MigrationConfig *qmp_query_migrate_config (Error **errp) +{ +MigrationState *s =migrate_get_current (); +MigrationConfig *config =g_new0 (MigrationConfig,1 ); + +QAPI_CLONE_MEMBERS (MigrationConfig, config, &s->config); + +/* set the has_* fields for every option */ +migrate_config_init (config); + +return config; +}migration/options.h +0
-0
migration/options.h uint64_t migrate_xbzrle_cache_size(void); · line 75
Quoted 5 lines
migration/options.h +1
-1@@ -72,7 +72,7 @@ uint64_t migrate_xbzrle_cache_size(void);ZeroPageDetection migrate_zero_page_detection (void );bool migrate_config_check (MigrationConfig *params,Error **errp); -void migrate_config_init (MigrationConfig *params); +void migrate_config_init (MigrationConfig *config);
Have you considered renaming the declaration's parameter when you change its type in PATCH 08, or when you rename the definition's parameter in PATCH 11?
Quoted 43 lines
migration/options.h +0
-0bool migrate_config_get_cap_compat (MigrationConfig *config,int i);bool migrate_caps_check (MigrationConfig *new,Error **errp);#endif qapi/migration.json +32
-0@@ -2552,3 +2552,45 @@ 'data': { '*tls-creds': 'str', '*tls-hostname': 'str', '*tls-authz': 'str' } } + +## +# @query-migrate-config: +# +# Returns information about the current migration configuration +# options +# +# Returns: @MigrationConfig +# +# Since: 10.1 +# +# .. qmp-example:: +# +# -> { "execute": "query-migrate-config" } +# <- { "return": { +# "multifd-channels": 2, +# "cpu-throttle-increment": 10, +# "cpu-throttle-initial": 20, +# "max-bandwidth": 33554432, +# "downtime-limit": 300 +# } +# } +## +{ 'command': 'query-migrate-config', + 'returns': 'MigrationConfig' } + +## +# @migrate-set-config: +# +# Set various migration configuration options. +# +# Since: 10.1
qapi/migration.json line 2596
Quoted 10 lines
qapi/migration.json +10
-0+# +# .. qmp-example:: +# +# -> { "execute": "migrate-set-config" , +# "arguments": { "max-bandwidth": 33554432, +# "downtime-limit": 300 } } +# <- { "return": {} } +## +{ 'command': 'migrate-set-config', 'boxed': true, + 'data': 'MigrationConfig' }
This patch exposes MigrationConfig externally. We should double-check its documentation to make sure it's what we want there. Known issue:
how to reset @tls-creds & friends. Touched in my review of PATCH 07.
Re: [RFC PATCH 12/13] [PoC] migration: Add query/set commands for MigrationConfig
Quoted 8 lines · Markus Armbruster
Quoted 2 lines · Fabiano Rosas
Add the QMP commands query-migrate-config and migrate-set-config to read and write the migration configuration options.
These supersede query-migrate-capabilities, query-migrate-parameters, migrate-set-capabilities, and migrate-set-parameters, right?
They could. We haven't made a decision at this moment.
Quoted 46 lines
Quoted 46 lines
Signed-off-by Fabiano Rosas <[email protected]>--- migration/options.c | 24 ++++++++++++++++++++++++ migration/options.h | 2 +- qapi/migration.json | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-)migration/options.c +24
-0@@ -1441,3 +1441,27 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)migrate_config_apply (&tmp);migrate_post_update_config (&tmp, errp); } + +void qmp_migrate_set_config (MigrationConfig *config,Error **errp) +{ +if (!migrate_config_check (config, errp)) { +/* Invalid parameter */ +return ; + } + +migrate_config_apply (config); +migrate_post_update_config (config, errp); +} + +MigrationConfig *qmp_query_migrate_config (Error **errp) +{ +MigrationState *s =migrate_get_current (); +MigrationConfig *config =g_new0 (MigrationConfig,1 ); + +QAPI_CLONE_MEMBERS (MigrationConfig, config, &s->config); + +/* set the has_* fields for every option */ +migrate_config_init (config); + +return config; +}migration/options.h +0
-0@@ -72,7 +72,7 @@ uint64_t migrate_xbzrle_cache_size(void);ZeroPageDetection migrate_zero_page_detection (void );
Quoted 8 lines
migration/options.h uint64_t migrate_xbzrle_cache_size(void); · line 75
Quoted 3 lines
migration/options.h +1
-1bool migrate_config_check (MigrationConfig *params,Error **errp); -void migrate_config_init (MigrationConfig *params); +void migrate_config_init (MigrationConfig *config);Have you considered renaming the declaration's parameter when you change its type in PATCH 08, or when you rename the definition's parameter in PATCH 11?
Yes, that looks like a mistake during rebase.
Quoted 49 lines
Quoted 49 lines
migration/options.h +0
-0bool migrate_config_get_cap_compat (MigrationConfig *config,int i);bool migrate_caps_check (MigrationConfig *new,Error **errp);#endif qapi/migration.json +38
-0@@ -2552,3 +2552,45 @@ 'data': { '*tls-creds': 'str', '*tls-hostname': 'str', '*tls-authz': 'str' } } + +## +# @query-migrate-config: +# +# Returns information about the current migration configuration +# options +# +# Returns: @MigrationConfig +# +# Since: 10.1 +# +# .. qmp-example:: +# +# -> { "execute": "query-migrate-config" } +# <- { "return": { +# "multifd-channels": 2, +# "cpu-throttle-increment": 10, +# "cpu-throttle-initial": 20, +# "max-bandwidth": 33554432, +# "downtime-limit": 300 +# } +# } +## +{ 'command': 'query-migrate-config', + 'returns': 'MigrationConfig' } + +## +# @migrate-set-config: +# +# Set various migration configuration options. +# +# Since: 10.1 +# +# .. qmp-example:: +# +# -> { "execute": "migrate-set-config" , +# "arguments": { "max-bandwidth": 33554432, +# "downtime-limit": 300 } }
Quoted 8 lines
qapi/migration.json line 2596
Quoted 4 lines
qapi/migration.json +4
-0+# <- { "return": {} } +## +{ 'command': 'migrate-set-config', 'boxed': true, + 'data': 'MigrationConfig' }This patch exposes MigrationConfig externally. We should double-check its documentation to make sure it's what we want there. Known issue:
how to reset @tls-creds & friends. Touched in my review of PATCH 07.
I'll take a closer look at tls options after Daniel suggested changing MigrationParameters' TLS options type from str to StrOrNull. That allows us to drop MigrateSetParameters entirely.
RFC PATCH 13/13 migration: Allow migrate commands to provide the migration config
In reply to Fabiano RosasAllow the migrate and migrate_incoming commands to pass the migration configuration options all at once, dispensing the use of migrate-set-parameters and migrate-set-capabilities.
The motivation of this is to simplify the interface with the management layer and avoid the usage of several command invocations to configure a migration. It also avoids stale parameters from a previous migration to influence the current migration.
The options that are changed during the migration can still be set with the existing commands.
--- migration/migration-hmp-cmds.c | 5 +++-- migration/migration.c | 8 ++++---- qapi/migration.json | 10 ++++++++++ system/vl.c | 3 ++- 4 files changed, 19 insertions(+), 7 deletions(-)
migration/migration-hmp-cmds.c +3 -2
@@ -429,7 +429,7 @@ void hmp_migrate_incoming(Monitor *mon, const QDict *qdict) }QAPI_LIST_PREPEND (caps,g_steal_pointer (&channel)); -qmp_migrate_incoming (NULL , true, caps, true, false, &err); +qmp_migrate_incoming (NULL , true, caps,NULL , true, false, &err);qapi_free_MigrationChannelList (caps); end:
@@ -715,7 +715,8 @@ void hmp_migrate(Monitor *mon, const QDict *qdict) }QAPI_LIST_PREPEND (caps,g_steal_pointer (&channel)); -qmp_migrate (NULL , true, caps, false, false, true, resume, &err); +qmp_migrate (NULL , true, caps,NULL , false, false, true, resume, + &err);if (hmp_handle_error (mon, err)) {return ; }
migration/migration.c +4 -4
@@ -1894,8 +1894,8 @@ void migrate_del_blocker(Error **reasonp)void qmp_migrate_incoming (const char *uri,bool has_channels,MigrationChannelList *channels, -bool has_exit_on_error,bool exit_on_error, -Error **errp) +MigrationConfig *config,bool has_exit_on_error, +bool exit_on_error,Error **errp) {Error *local_err =NULL ;static bool once = true;
@@ -2159,8 +2159,8 @@ static gboolean qmp_migrate_finish_cb(QIOChannel *channel,return G_SOURCE_REMOVE ; } -void qmp_migrate (const char *uri,bool has_channels, -MigrationChannelList *channels,bool has_detach,bool detach, +void qmp_migrate (const char *uri,bool has_channels,MigrationChannelList *channels, +MigrationConfig *config,bool has_detach,bool detach,bool has_resume,bool resume,Error **errp) {bool resume_requested;
qapi/migration.json +10 -0
@@ -1638,6 +1638,10 @@ # # @resume: resume one paused migration, default "off". (since 3.0) # +# @config: migration configuration options, previously set via +# @migrate-set-parameters and @migrate-set-capabilities. (since +# 10.1) +# # Since: 0.14 # # .. admonition:: Notes
@@ -1702,6 +1706,7 @@ { 'command': 'migrate', 'data': {'*uri': 'str', '*channels': [ 'MigrationChannel' ], + '*config': 'MigrationConfig', '*detach': 'bool', '*resume': 'bool' } } ##
@@ -1721,6 +1726,10 @@ # error details could be retrieved with query-migrate. # (since 9.1) # +# @config: migration configuration options, previously set via +# @migrate-set-parameters and @migrate-set-capabilities. (since +# 10.1) +# # Since: 2.3 # # .. admonition:: Notes
@@ -1774,6 +1783,7 @@ { 'command': 'migrate-incoming', 'data': {'*uri': 'str', '*channels': [ 'MigrationChannel' ], + '*config': 'MigrationConfig', '*exit-on-error': 'bool' } } ##
system/vl.c +2 -1
@@ -2826,7 +2826,8 @@ void qmp_x_exit_preconfig(Error **errp)g_new0 (MigrationChannelList,1 ); channels->value = incoming_channels[MIGRATION_CHANNEL_TYPE_MAIN ]; -qmp_migrate_incoming (NULL , true, channels, true, true, &local_err); +qmp_migrate_incoming (NULL , true, channels,NULL , true, true, + &local_err);if (local_err) {error_reportf_err (local_err,"-incoming %s: " , incoming);exit (1 );
Signature · 1 line
2.35.3
Re: [RFC PATCH 13/13] [PoC] migration: Allow migrate commands to provide the migration config
qapi/migration.json
Quoted 74 lines · Fabiano Rosas
Allow the migrate and
migrate_incomingcommands to pass the migration configuration options all at once, dispensing the use of migrate-set-parameters and migrate-set-capabilities.The motivation of this is to simplify the interface with the management layer and avoid the usage of several command invocations to configure a migration. It also avoids stale parameters from a previous migration to influence the current migration.
The options that are changed during the migration can still be set with the existing commands.
Signed-off-by Fabiano Rosas <[email protected]>--- migration/migration-hmp-cmds.c | 5 +++-- migration/migration.c | 8 ++++---- qapi/migration.json | 10 ++++++++++ system/vl.c | 3 ++- 4 files changed, 19 insertions(+), 7 deletions(-)migration/migration-hmp-cmds.c +3
-2@@ -429,7 +429,7 @@ void hmp_migrate_incoming(Monitor *mon, const QDict *qdict) }QAPI_LIST_PREPEND (caps,g_steal_pointer (&channel)); -qmp_migrate_incoming (NULL , true, caps, true, false, &err); +qmp_migrate_incoming (NULL , true, caps,NULL , true, false, &err);qapi_free_MigrationChannelList (caps); end:@@ -715,7 +715,8 @@ void hmp_migrate(Monitor *mon, const QDict *qdict) }QAPI_LIST_PREPEND (caps,g_steal_pointer (&channel)); -qmp_migrate (NULL , true, caps, false, false, true, resume, &err); +qmp_migrate (NULL , true, caps,NULL , false, false, true, resume, + &err);if (hmp_handle_error (mon, err)) {return ; }migration/migration.c +4
-4@@ -1894,8 +1894,8 @@ void migrate_del_blocker(Error **reasonp)void qmp_migrate_incoming (const char *uri,bool has_channels,MigrationChannelList *channels, -bool has_exit_on_error,bool exit_on_error, -Error **errp) +MigrationConfig *config,bool has_exit_on_error, +bool exit_on_error,Error **errp) {Error *local_err =NULL ;static bool once = true;@@ -2159,8 +2159,8 @@ static gboolean qmp_migrate_finish_cb(QIOChannel *channel,return G_SOURCE_REMOVE ; } -void qmp_migrate (const char *uri,bool has_channels, -MigrationChannelList *channels,bool has_detach,bool detach, +void qmp_migrate (const char *uri,bool has_channels,MigrationChannelList *channels, +MigrationConfig *config,bool has_detach,bool detach,bool has_resume,bool resume,Error **errp) {bool resume_requested;qapi/migration.json +0
-0@@ -1638,6 +1638,10 @@
##
# @migrate:
#
# Migrates the current running guest to another Virtual Machine. #
# @uri: the Uniform Resource Identifier of the destination VM #
# @channels: list of migration stream channels with each stream in the
# list connected to a destination interface endpoint.
#
# @detach: this argument exists only for compatibility reasons and is
# ignored by QEMU
qapi/migration.json line 1639
Quoted 2 lines
qapi/migration.json +0
-0# # @resume: resume one paused migration, default "off". (since 3.0)
Unrelated to this patch, but here goes anyway. What happens if I pass @uri and @channels with I "resume": true, and they differ from the ones passed originally?
Quoted 9 lines
qapi/migration.json +4
-0# +# @config: migration configuration options, previously set via +# @migrate-set-parameters and @migrate-set-capabilities. (since +# 10.1) +# # Since: 0.14 # # .. admonition:: Notes
qapi/migration.json line 1711
Quoted 6 lines
qapi/migration.json +1
-0@@ -1702,6 +1706,7 @@ { 'command': 'migrate', 'data': {'*uri': 'str', '*channels': [ 'MigrationChannel' ], + '*config': 'MigrationConfig', '*detach': 'bool', '*resume': 'bool' } }
Some configuration is passed in individual arguments, the remainder wrapped in a struct.
Here's how we could avoid this. 'data': 'MigrateArguments' where MigrateArguments is a struct with base MigrationConfig that adds the other arguments. Add 'boxed': true if you like.
Not sure it's wortwhile, but I wanted to show you for your consideration.
Quoted 13 lines
qapi/migration.json +4
-0##@@ -1721,6 +1726,10 @@ # error details could be retrieved with query-migrate. # (since 9.1) # +# @config: migration configuration options, previously set via +# @migrate-set-parameters and @migrate-set-capabilities. (since +# 10.1) +# # Since: 2.3 # # .. admonition:: Notes
qapi/migration.json line 1789
Quoted 7 lines
qapi/migration.json +1
-0@@ -1774,6 +1783,7 @@ { 'command': 'migrate-incoming', 'data': {'*uri': 'str', '*channels': [ 'MigrationChannel' ], + '*config': 'MigrationConfig', '*exit-on-error': 'bool' } } ##
Likewise.
Quoted 14 lines
system/vl.c +2
-1@@ -2826,7 +2826,8 @@ void qmp_x_exit_preconfig(Error **errp)g_new0 (MigrationChannelList,1 ); channels->value = incoming_channels[MIGRATION_CHANNEL_TYPE_MAIN ]; -qmp_migrate_incoming (NULL , true, channels, true, true, &local_err); +qmp_migrate_incoming (NULL , true, channels,NULL , true, true, + &local_err);if (local_err) {error_reportf_err (local_err,"-incoming %s: " , incoming);exit (1 );
Re: [RFC PATCH 13/13] [PoC] migration: Allow migrate commands to provide the migration config
Quoted 17 lines · Markus Armbruster
qapi/migration.json
Quoted 5 lines
qapi/migration.json +0
-0@@ -1638,6 +1638,10 @@##
# @migrate:
#
# Migrates the current running guest to another Virtual Machine. #
# @uri: the Uniform Resource Identifier of the destination VM #
# @channels: list of migration stream channels with each stream in the
# list connected to a destination interface endpoint.
#
# @detach: this argument exists only for compatibility reasons and is
# ignored by QEMU
Quoted 6 lines
qapi/migration.json line 1639
Quoted 2 lines
qapi/migration.json +0
-0# # @resume: resume one paused migration, default "off". (since 3.0)Unrelated to this patch, but here goes anyway. What happens if I pass @uri and @channels with I "resume": true, and they differ from the ones passed originally?
I can answer this one - "resume" was designed to work always with new channels/URI passed in. It was currently only used for postcopy to resume a paused postcopy migration where the old URI/channels stopped working already.
The doc is indeed not obvious to show that, and can be confusing. If anyone thinks worthwhile, I can send a patch to touch it up:
qapi/migration.json +4 -1
@@ -1658,7 +1658,10 @@ # @detach: this argument exists only for compatibility reasons and is # ignored by QEMU # -# @resume: resume one paused migration, default "off". (since 3.0) +# @resume: when set, resume one paused postcopy migration, using the new +# URI/channels specified to replace the old/broken channels. The user +# should make sure the migration is in "postcopy-paused" state before +# the resume request. Default "off". (since 3.0) # # Since: 0.14 #
Thanks,
Signature · 1 line
Peter Xu
Quoted 26 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.
Quoted 7 lines
- What about savevm?
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
For savevm/loadvm in particular it is very much a foot-gun that 'migrate-set-*' will affect them, because savevm/loadvm aren't obviously connected to 'migrate-*' commands unless you're aware of how QEMU implements savevm internally.
Quoted 6 lines
- HMP?
Can we convert the strings passed via
hmp_set_parameterswithout having an enum of parameters? Duplication problem again.
Quoted 6 lines
- incoming defer?
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from
qmp_migrateon source and send it to the destination for negotiation.
I'm not sure I understand why the QAPI design changes are tied to the new protocol handshake ? I guess you're wanting to avoid updating 'migrate_incoming' to accept the new parameters directly ?
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
Quoted 9 lines
Quoted 5 lines
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Well, except the options that are adjusted during migration. But yes, I agree. It all depends on how we proceed with keeping the old commands around and for how long. If they're still around we can't stop people from using them and later invoking "savevm" for instance.
Quoted 5 lines
For savevm/loadvm in particular it is very much a foot-gun that 'migrate-set-*' will affect them, because savevm/loadvm aren't obviously connected to 'migrate-*' commands unless you're aware of how QEMU implements savevm internally.
Yes, I could perhaps reset all options once savevm is called, maybe that would be acceptable, then we don't need to check and block every single one. Once we add support to migration options to savevm, then they'd be set in the savevm command-line from day 1 and those wouldn't be reset. We could also keep HMP restricted to savevm without any migration options. That's be easy to enforce. If the user wants fancy savevm, they can invoke via QMP.
Quoted 8 lines
Quoted 8 lines
- HMP?
Can we convert the strings passed via
hmp_set_parameterswithout having an enum of parameters? Duplication problem again.- incoming defer?
Quoted 9 lines
Quoted 4 lines
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from
qmp_migrateon source and send it to the destination for negotiation.I'm not sure I understand why the QAPI design changes are tied to the new protocol handshake ? I guess you're wanting to avoid updating '
migrate_incoming' to accept the new parameters directly ?
Yes, without migrate-set-capabilities, we'd need to pass an enormous command line to -incoming defer to be able to enable capabilities on the destination. With the handshake, we could transfer them over the wire somehow. Does that make sense?
Quoted 3 lines
With regards,
Daniel
Quoted 52 lines · Fabiano Rosas
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
Quoted 9 lines
Quoted 5 lines
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Well, except the options that are adjusted during migration. But yes, I agree. It all depends on how we proceed with keeping the old commands around and for how long. If they're still around we can't stop people from using them and later invoking "savevm" for instance.
Quoted 5 lines
For savevm/loadvm in particular it is very much a foot-gun that 'migrate-set-*' will affect them, because savevm/loadvm aren't obviously connected to 'migrate-*' commands unless you're aware of how QEMU implements savevm internally.
Quoted 7 lines
Yes, I could perhaps reset all options once savevm is called, maybe that would be acceptable, then we don't need to check and block every single one. Once we add support to migration options to savevm, then they'd be set in the savevm command-line from day 1 and those wouldn't be reset. We could also keep HMP restricted to savevm without any migration options. That's be easy to enforce. If the user wants fancy savevm, they can invoke via QMP.
Can we make the two approaches mutually exclusive ? Taking your 'migrate' command example addition:
{ 'command': 'migrate',
'data': {'*uri': 'str',
'*channels': [ 'MigrationChannel' ],
+ '*config': 'MigrationConfig',
'*detach': 'bool', '*resume': 'bool' } }
if 'migrate' is invoked with the '*config' data being non-nil, then we should ignore *all* global state previously set with migrate-set-XXXX, and exclusively use '*config'.
That gives a clean semantic break between old and new approaches, without us having to worry about removing the existing commands quickly.
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
- incoming defer?
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from
qmp_migrateon source and send it to the destination for negotiation.I'm not sure I understand why the QAPI design changes are tied to the new protocol handshake ? I guess you're wanting to avoid updating '
migrate_incoming' to accept the new parameters directly ?
Quoted 4 lines
Yes, without migrate-set-capabilities, we'd need to pass an enormous command line to -incoming defer to be able to enable capabilities on the destination. With the handshake, we could transfer them over the wire somehow. Does that make sense?
'-incoming defer' still gets paired with 'migrate-incoming' on the target, so no matter what, there's no reason to ever pass parameters on the CLI with '-incoming defer'.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 70 lines · Daniel P. Berrangé
Quoted 52 lines · Fabiano Rosas
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
Quoted 9 lines
Quoted 5 lines
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Well, except the options that are adjusted during migration. But yes, I agree. It all depends on how we proceed with keeping the old commands around and for how long. If they're still around we can't stop people from using them and later invoking "savevm" for instance.
Quoted 5 lines
For savevm/loadvm in particular it is very much a foot-gun that 'migrate-set-*' will affect them, because savevm/loadvm aren't obviously connected to 'migrate-*' commands unless you're aware of how QEMU implements savevm internally.
Quoted 7 lines
Yes, I could perhaps reset all options once savevm is called, maybe that would be acceptable, then we don't need to check and block every single one. Once we add support to migration options to savevm, then they'd be set in the savevm command-line from day 1 and those wouldn't be reset. We could also keep HMP restricted to savevm without any migration options. That's be easy to enforce. If the user wants fancy savevm, they can invoke via QMP.
Can we make the two approaches mutually exclusive ? Taking your 'migrate' command example addition:
{ 'command': 'migrate',
'data': {'*uri': 'str',
'*channels': [ 'MigrationChannel' ],
+ '*config': 'MigrationConfig','*detach': 'bool', '*resume': 'bool' } }
Quoted 8 lines
if 'migrate' is invoked with the '*config' data being non-nil, then we should ignore *all* global state previously set with migrate-set-XXXX, and exclusively use '*config'.
That gives a clean semantic break between old and new approaches, without us having to worry about removing the existing commands quickly.
Good idea. I will need to do something about the -global options because they also set the defaults for the various options. But we should be able to decouple setting defaults from -global. Or I could just apply -global again on top of what came in '*config'.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
- incoming defer?
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from
qmp_migrateon source and send it to the destination for negotiation.I'm not sure I understand why the QAPI design changes are tied to the new protocol handshake ? I guess you're wanting to avoid updating '
migrate_incoming' to accept the new parameters directly ?
Quoted 9 lines
Quoted 4 lines
Yes, without migrate-set-capabilities, we'd need to pass an enormous command line to -incoming defer to be able to enable capabilities on the destination. With the handshake, we could transfer them over the wire somehow. Does that make sense?
'-incoming defer' still gets paired with 'migrate-incoming' on the target, so no matter what, there's no reason to ever pass parameters on the CLI with '-incoming defer'.
Oops, I misread the strcmp in vl.c. I mean -incoming uri is the one that'll need a huge cmdline.
But if we follow your suggestion above we could just tie -incoming URI to the existing commands and make the new format require defer.
Quoted 3 lines
With regards,
Daniel
Quoted 109 lines · Fabiano Rosas
Quoted 70 lines · Daniel P. Berrangé
Quoted 52 lines · Fabiano Rosas
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
Quoted 9 lines
Quoted 5 lines
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Well, except the options that are adjusted during migration. But yes, I agree. It all depends on how we proceed with keeping the old commands around and for how long. If they're still around we can't stop people from using them and later invoking "savevm" for instance.
Quoted 5 lines
For savevm/loadvm in particular it is very much a foot-gun that 'migrate-set-*' will affect them, because savevm/loadvm aren't obviously connected to 'migrate-*' commands unless you're aware of how QEMU implements savevm internally.
Quoted 7 lines
Yes, I could perhaps reset all options once savevm is called, maybe that would be acceptable, then we don't need to check and block every single one. Once we add support to migration options to savevm, then they'd be set in the savevm command-line from day 1 and those wouldn't be reset. We could also keep HMP restricted to savevm without any migration options. That's be easy to enforce. If the user wants fancy savevm, they can invoke via QMP.
Can we make the two approaches mutually exclusive ? Taking your 'migrate' command example addition:
{ 'command': 'migrate',
'data': {'*uri': 'str',
'*channels': [ 'MigrationChannel' ],
+ '*config': 'MigrationConfig','*detach': 'bool', '*resume': 'bool' } }Quoted 8 lines
if 'migrate' is invoked with the '*config' data being non-nil, then we should ignore *all* global state previously set with migrate-set-XXXX, and exclusively use '*config'.
That gives a clean semantic break between old and new approaches, without us having to worry about removing the existing commands quickly.
Good idea. I will need to do something about the -global options because they also set the defaults for the various options. But we should be able to decouple setting defaults from -global. Or I could just apply -global again on top of what came in '*config'.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
- incoming defer?
It seems we cannot do the final step of removing migrate-set-capabilites before we have a form of handshake implemented. That would take the config from
qmp_migrateon source and send it to the destination for negotiation.I'm not sure I understand why the QAPI design changes are tied to the new protocol handshake ? I guess you're wanting to avoid updating '
migrate_incoming' to accept the new parameters directly ?Quoted 9 lines
Quoted 4 lines
Yes, without migrate-set-capabilities, we'd need to pass an enormous command line to -incoming defer to be able to enable capabilities on the destination. With the handshake, we could transfer them over the wire somehow. Does that make sense?
'-incoming defer' still gets paired with 'migrate-incoming' on the target, so no matter what, there's no reason to ever pass parameters on the CLI with '-incoming defer'.
Quoted 5 lines
Oops, I misread the strcmp in
vl.c. I mean -incoming uri is the one that'll need a huge cmdline.But if we follow your suggestion above we could just tie -incoming URI to the existing commands and make the new format require defer.
Yes, modern approach should require 'defer'.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 18 lines · Fabiano Rosas
Quoted 9 lines
Can we make the two approaches mutually exclusive ? Taking your 'migrate' command example addition:
{ 'command': 'migrate',
'data': {'*uri': 'str',
'*channels': [ 'MigrationChannel' ],
+ '*config': 'MigrationConfig','*detach': 'bool', '*resume': 'bool' } }Quoted 8 lines
if 'migrate' is invoked with the '*config' data being non-nil, then we should ignore *all* global state previously set with migrate-set-XXXX, and exclusively use '*config'.
That gives a clean semantic break between old and new approaches, without us having to worry about removing the existing commands quickly.
Quoted 4 lines
Good idea. I will need to do something about the -global options because they also set the defaults for the various options. But we should be able to decouple setting defaults from -global. Or I could just apply -global again on top of what came in '*config'.
Would it still be possible that we allow whatever options attached to the "migrate" command to only overwrite the globals (either set via -global or migrate-set-* QMP commands), rather than ignoring them completely?
If so, we don't need to do anything with current -global and it'll keep working. It's the same to most existing way to set the migration options (e.g., otherwise do we plan to disable HMP "migrate" usage?).
Making the above '*config' to only overwrite also do not stand against the mgmt using it to ignore the default options, after all the mgmt can decide to always set everything in the '*config' then it's the same as ignoring the globals?
Signature · 1 line
Peter Xu
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
Quoted 8 lines
Quoted 5 lines
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Yes.
Even better: make it a job.
[...]
Quoted 37 lines · Markus Armbruster
Quoted 35 lines · Daniel P. Berrangé
Quoted 33 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.- What about savevm?
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
Quoted 7 lines
Quoted 2 lines
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Yes.
Even better: make it a job.
What do we gain from that in relation to being able to pass ~50 parameters to a command? I don't see it. I think that's the actual crux here, too many options and how to get them from the QAPI into the migration core for consumption.
The current usage of MigrationParameters as both the return type for query-set-parameters and the global parameter store for the migration state is really dissonant. What do the has_* fields even mean when accessed via MigrationState::parameters? This series is not doing any better in that regard, mind you. I'm almost tempted to ask that we expose the QDict from the marshaling function directly to the migration code, at least that's a data type that makes sense internally.
Quoted 42 lines · Fabiano Rosas
Quoted 40 lines · Markus Armbruster
Quoted 37 lines · Daniel P. Berrangé
Quoted 26 lines · Fabiano Rosas
Open questions:
---------------- Deprecations/compat?
I think we should deprecate migrate-set/query-capabilities and everything to do with capabilities (specifically the validation in the JSON at the end of the stream).
For migrate-set/query-parameters, we could probably keep it around indefinitely, but it'd be convenient to introduce new commands so we can give them new semantics.
- How to restrict the options that should not be set when the migration is in progress?
i.e.:
all options can be set before migration (initial config) some options can be set during migration (runtime)I thought of adding another type at the top of the hierarchy, with just the options allowed to change at runtime, but that doesn't really stop the others being also set at runtime. I'd need a way to have a set of options that are rejected 'if
migration_is_running()', without adding more duplication all around.Quoted 7 lines
- What about savevm?
None of this solves the issue of random caps/params being set before calling savevm. We still need to special-case savevm and reject everything. Unless we entirely deprecate setting initial options via set-parameters (or set-config) and require all options to be set as savevm (and migrate) arguments.
I'd suggest we aim for a world where the commands take all options as direct args and try to remove the global state eventually.
Quoted 8 lines
Quoted 3 lines
Yes.
Even better: make it a job.
What do we gain from that in relation to being able to pass ~50 parameters to a command? I don't see it. I think that's the actual crux here, too many options and how to get them from the QAPI into the migration core for consumption.
Two separate interface design problems:
1. Query and manipulate complex configuration
2. Control and monitor long-running jobs
Let's start with 1.
Migration is not the only instance of complex configuration. Block devices are arguably even more complex. A difference: we can have many block devices, but just one migration.
Here's the usual way to configure things: commands creating a thing take the thing's configuration as arguments, other commands query and manipulate a thing's configuration. Works fine both for simple and complex configuration. Block devices work this way.
Migration doesn't. Instead, we have global configuration state, and commands to query and manipulate it. Also works.
Instead of passing the entire configuration to the "create" command, you can pass it piecemeal to "manipulate global configuration" commands. You pay for the convenience by having to reset the entire global configuration to a known state in certain situations. Since use of the global configuration state is implicit, it can be surprising, as Daniel pointed out for savevm.
Overall, this isn't simpler or more convenient than the usual way, just different.
Both ways use QAPI objects to hold configuration. Query commands can return such an object. Commands that deal with partial configuration can only take the same object when its members are optional.
In places where configuration is complete (internal configuration state, the query command that returns it), optional makes no sense, and is annoying. We either accept that, or duplicate the configuration object. Duplicating violates DRY. It does result in better command documentation, though.
On to 2.
Migration is not the only kind of long-running job. There's also block-commit, block-stream, drive-mirror, drive-backup, blockdev-create, x-blockdev-amend, snapshot-load, snapshot-save, snapshot-delete.
These all create a "job" object that represents the long-running job. We have commands and events to control and monitor such jobs: job-pause, job-resume, job-cancel, job-complete, job-dismiss, job-finalize, query-jobs, JOB_STATUS_CHANGE.
Migration does its own thing instead.
Quoted 4 lines
The current usage of MigrationParameters as both the return type for query-set-parameters and the global parameter store for the migration state is really dissonant. What do the has_* fields even mean when accessed via
MigrationState::parameters?
Yes, it's ugly, and the alternative is differently ugly, as discussed above.
Quoted 4 lines
This series is not doing any
better in that regard, mind you. I'm almost tempted to ask that we expose the QDict from the marshaling function directly to the migration code, at least that's a data type that makes sense internally.
Based on experience in the block layer, I predict you'd regret this :)
Use of native C data types is just so much easier on the eyes than messing with QObjects. Moreover, the compiler can't help you as much with QObjects.