migration: always initial ram_counters for a new migration
PATCH v3 1/1 migration: always initial ram_counters for a new migration
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run into migration_completion phase
Reason as follows:
Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:
current_bytes = migration_total_bytes(s);
transferred = current_bytes - s->iteration_initial_bytes;
time_spent = current_time - s->iteration_start_time;
bandwidth = (double)transferred / time_spent;
s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new migration, but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may lead pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.
---v2->v3:
- fix the bug of update_iteration_initial_status function prototype
v1->v2:
- Add interface update_iteration_initial_status to update statistic fields at the same time to avoid info mismatch lead wrong calculation result.
migration/migration.c | 25 +++++++++++++++++++------ migration/savevm.c | 1 + 2 files changed, 20 insertions(+), 6 deletions(-)
migration/migration.c +19 -6
@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, }migrate_init (s); +/* +* set ram_counters memory to zero for a +* new migration +*/ +memset (&ram_counters,0 ,sizeof (ram_counters));return true; }
@@ -3025,6 +3030,17 @@ static void migration_calculate_complete(MigrationState *s) } } +static void update_iteration_initial_status (MigrationState *s) +{ +/* +* Update these three fields at the same time to avoid mismatch info lead +* wrong speed calculation. +*/ + s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); + s->iteration_initial_bytes =migration_total_bytes (s); + s->iteration_initial_pages =ram_get_total_transferred_pages (); +} +static void migration_update_counters (MigrationState *s,int64_t current_time) {
@@ -3060,9 +3076,7 @@ static void migration_update_counters(MigrationState *s,qemu_file_reset_rate_limit (s->to_dst_file); - s->iteration_start_time = current_time; - s->iteration_initial_bytes = current_bytes; - s->iteration_initial_pages =ram_get_total_transferred_pages (); +update_iteration_initial_status (s);trace_migrate_transferred (transferred, time_spent, bandwidth, s->threshold_size);
@@ -3186,7 +3200,7 @@ static void *migration_thread(void *opaque)rcu_register_thread ();object_ref (OBJECT (s)); - s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +update_iteration_initial_status (s);qemu_savevm_state_header (s->to_dst_file);
@@ -3251,8 +3265,7 @@ static void *migration_thread(void *opaque)* the local variables. This is important to avoid * breaking transferred_bytes and bandwidth calculation */ - s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); - s->iteration_initial_bytes =0 ; +update_iteration_initial_status (s); } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );
migration/savevm.c +1 -0
@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) }migrate_init (ms); +memset (&ram_counters,0 ,sizeof (ram_counters)); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();
Signature · 1 line
2.17.2 (Apple Git-113)
Quoted 31 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.
Quoted 7 lines
Signed-off-by Ivan Ren <[email protected]>Reviewed-by Juan Quintela <[email protected]>Suggested-by Wei Yang <[email protected]>---
v2->v3:
- fix the bug ofupdate_iteration_initial_statusfunction prototype
Code looks good. Have you verified on this version?
BTW, you didn't address the multifd count in this patch, right?
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 41 lines · Wei Yang
Quoted 31 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.Quoted 7 lines
Signed-off-by Ivan Ren <[email protected]>Reviewed-by Juan Quintela <[email protected]>Suggested-by Wei Yang <[email protected]>---
v2->v3:
- fix the bug ofupdate_iteration_initial_statusfunction prototype
Quoted 1 line
Code looks good. Have you verified on this version?
Yes
Quoted 1 line
BTW, you didn't address the multifd count in this patch, right?
Yes.
Currently multifd page count has no harm, so I think it's better to optimize it in a new patch to make things clearer.
Thanks.
Wei Yang
Reviewed-byQuoted 44 lines · Ivan Ren
Quoted 41 lines · Wei Yang
Quoted 31 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.Quoted 7 lines
Signed-off-by Ivan Ren <[email protected]>Reviewed-by Juan Quintela <[email protected]>Suggested-by Wei Yang <[email protected]>---
v2->v3:
- fix the bug ofupdate_iteration_initial_statusfunction prototypeQuoted 1 line
Code looks good. Have you verified on this version?
Quoted 7 lines
Yes
Quoted 1 line
BTW, you didn't address the multifd count in this patch, right?
Yes.
Currently multifd page count has no harm, so I think it's better to optimize it in a new patch to make things clearer.
Fine.
Quoted 2 lines
Thanks.
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 31 lines · * Ivan Ren ([email protected]) wrote:
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.
Quoted 3 lines
Signed-off-by Ivan Ren <[email protected]>Reviewed-by Juan Quintela <[email protected]>Suggested-by Wei Yang <[email protected]>
Thank you,
Queued
Quoted 92 lines
---
v2->v3:
- fix the bug ofupdate_iteration_initial_statusfunction prototype
v1->v2:
- Add interfaceupdate_iteration_initial_statusto update statistic fields at the same time to avoid info mismatch lead wrong calculation result.migration/migration.c | 25 +++++++++++++++++++------ migration/savevm.c | 1 + 2 files changed, 20 insertions(+), 6 deletions(-)migration/migration.c +19
-6@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, }migrate_init (s); +/* +* set ram_counters memory to zero for a +* new migration +*/ +memset (&ram_counters,0 ,sizeof (ram_counters));return true; }@@ -3025,6 +3030,17 @@ static void migration_calculate_complete(MigrationState *s) } } +static void update_iteration_initial_status (MigrationState *s) +{ +/* +* Update these three fields at the same time to avoid mismatch info lead +* wrong speed calculation. +*/ + s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); + s->iteration_initial_bytes =migration_total_bytes (s); + s->iteration_initial_pages =ram_get_total_transferred_pages (); +} +static void migration_update_counters (MigrationState *s,int64_t current_time) {@@ -3060,9 +3076,7 @@ static void migration_update_counters(MigrationState *s,qemu_file_reset_rate_limit (s->to_dst_file); - s->iteration_start_time = current_time; - s->iteration_initial_bytes = current_bytes; - s->iteration_initial_pages =ram_get_total_transferred_pages (); +update_iteration_initial_status (s);trace_migrate_transferred (transferred, time_spent, bandwidth, s->threshold_size);@@ -3186,7 +3200,7 @@ static void *migration_thread(void *opaque)rcu_register_thread ();object_ref (OBJECT (s)); - s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +update_iteration_initial_status (s);qemu_savevm_state_header (s->to_dst_file);@@ -3251,8 +3265,7 @@ static void *migration_thread(void *opaque)* the local variables. This is important to avoid * breaking transferred_bytes and bandwidth calculation */ - s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); - s->iteration_initial_bytes =0 ; +update_iteration_initial_status (s); } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );migration/savevm.c +1
-0@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) }migrate_init (ms); +memset (&ram_counters,0 ,sizeof (ram_counters)); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();Signature · 1 line
2.17.2 (Apple Git-113)
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK