migration: always initial ram_counters for a new migration
PATCH 1/1 migration: always initial ram_counters for a new migration
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run into migration_completion phase
Reason as follows:
Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:
current_bytes = migration_total_bytes(s);
transferred = current_bytes - s->iteration_initial_bytes;
time_spent = current_time - s->iteration_start_time;
bandwidth = (double)transferred / time_spent;
s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new migration, but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may lead pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.
--- migration/migration.c | 15 ++++++++++++++- migration/savevm.c | 1 + 2 files changed, 15 insertions(+), 1 deletion(-)
migration/migration.c +14 -1
@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, }migrate_init (s); +/* +* set ram_counters memory to zero for a +* new migration +*/ +memset (&ram_counters,0 ,sizeof (ram_counters));return true; }
@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)object_ref (OBJECT (s)); s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +/* +* Update s->iteration_initial_bytes to match s->iteration_start_time. +*/ + s->iteration_initial_bytes =migration_total_bytes (s);qemu_savevm_state_header (s->to_dst_file);
@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)* breaking transferred_bytes and bandwidth calculation */ s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); - s->iteration_initial_bytes =0 ; +/* +* Update s->iteration_initial_bytes to current size to +* avoid historical data lead wrong bandwidth. +*/ + s->iteration_initial_bytes =migration_total_bytes (s); } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );
migration/savevm.c +1 -0
@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) }migrate_init (ms); +memset (&ram_counters,0 ,sizeof (ram_counters)); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();
Signature · 1 line
2.17.2 (Apple Git-113)
Quoted 31 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.
Quoted 1 line
Signed-off-by Ivan Ren <[email protected]>
Quoted 54 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run intomigration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every new migration, butram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may leadpending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.Signed-off-by Ivan Ren <[email protected]>---
migration/migration.c| 15 ++++++++++++++-migration/savevm.c| 1 +2 files changed, 15 insertions(+), 1 deletion(-)migration/migration.c +5
-0@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, }migrate_init (s); +/* +* set ram_counters memory to zero for a +* new migration +*/ +memset (&ram_counters,0 ,sizeof (ram_counters));return true; }
migration/migration.c static void *migration_thread(void *opaque) · line 3198
Quoted 7 lines
migration/migration.c +4
-0@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)object_ref (OBJECT (s)); s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +/* +* Update s->iteration_initial_bytes to match s->iteration_start_time. +*/ + s->iteration_initial_bytes =migration_total_bytes (s);
Is this one necessary? We have sent out nothing yet.
Quoted 31 lines
migration/migration.c +5
-1qemu_savevm_state_header (s->to_dst_file);@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)* breaking transferred_bytes and bandwidth calculation */ s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); - s->iteration_initial_bytes =0 ; +/* +* Update s->iteration_initial_bytes to current size to +* avoid historical data lead wrong bandwidth. +*/ + s->iteration_initial_bytes =migration_total_bytes (s); } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );migration/savevm.c +1
-0@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) }migrate_init (ms); +memset (&ram_counters,0 ,sizeof (ram_counters)); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();Signature · 1 line
2.17.2 (Apple Git-113)
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential missing?
Thanks.
Quoted 110 lines · Wei Yang
Quoted 3 lines · Ivan Ren
From: Ivan Ren <[email protected]>
This patch fix a multifd migration bug in migration speed calculation,
this
Quoted 5 lines
problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferredbytes]
Quoted 2 lines
4. migrate cancel
5. begin a new migration with multifd, the migration will directly runinto
Quoted 13 lines
migration_completionphaseReason as follows:
Migration update bandwidth and
s->threshold_sizein functionmigration_update_countersafterBUFFER_DELAYtime:current_bytes = migration_total_bytes(s); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit;Quoted 3 lines
In multifd migration,
migration_total_bytesfunction returnqemu_ftell(s->to_dst_file)+ ram_counters.multifd_bytes.s->iteration_initial_byteswill be initialized to 0 at every newmigration,
Quoted 2 lines
but
ram_countersis a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it maylead
migration/migration.c static bool migrate_prepare(MigrationState *s,
Quoted 14 lines
pending_size>=s->threshold_sizebecome false inmigration_iteration_runafter the firstmigration_update_counters.Signed-off-by Ivan Ren <[email protected]>---
migration/migration.c| 15 ++++++++++++++-migration/savevm.c| 1 +2 files changed, 15 insertions(+), 1 deletion(-)migration/migration.c +0
-0@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s,bool blk, bool
blk_inc,Quoted 12 lines
migration/migration.c +5
-0}migrate_init (s); +/* +* set ram_counters memory to zero for a +* new migration +*/ +memset (&ram_counters,0 ,sizeof (ram_counters));return true; }migration/migration.c static void *migration_thread(void *opaque) · line 3196
Quoted 5 lines
migration/migration.c +2
-0@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)object_ref (OBJECT (s)); s->iteration_start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +/* +* Update s->iteration_initial_bytes to match
s->iteration_start_time.migration/migration.c static void *migration_thread(void *opaque) · line 3198
Quoted 2 lines
migration/migration.c +2
-0+*/ + s->iteration_initial_bytes =migration_total_bytes (s);Is this one necessary? We have sent out nothing yet.
migration/migration.c static void *migration_thread(void *opaque) · line 3263
Quoted 7 lines
migration/migration.c +0
-0qemu_savevm_state_header (s->to_dst_file);@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)* breaking transferred_bytes and bandwidth calculation */ s->iteration_start_time =
qemu_clock_get_ms(QEMU_CLOCK_REALTIME);migration/savevm.c static int qemu_savevm_state(QEMUFile *f, Error
Quoted 14 lines
migration/migration.c +5
-1- s->iteration_initial_bytes =0 ; +/* +* Update s->iteration_initial_bytes to current size to +* avoid historical data lead wrong bandwidth. +*/ + s->iteration_initial_bytes =migration_total_bytes (s); } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );migration/savevm.c +0
-0@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error**errp)
Quoted 10 lines
migration/savevm.c +1
-1}migrate_init (ms); +memset (&ram_counters,0 ,sizeof (ram_counters)); ms->to_dst_file = f;qemu_mutex_unlock_iothread (); --2.17.2 (Apple Git-113)
--
Wei Yang
Help you, Help me
Quoted 7 lines · Ivan Ren
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.Quoted 3 lines
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 7 lines
Quoted 1 line
Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Is that better to always match the update of
iteration_initial_bytesanditeration_start_timein a explicit way to avoid some potential missing?
You may get some point. Well after a close look, we may find other potential problem.
1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a helper function. So each time all of them.
2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.Quoted 3 lines
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Is that better to always match the update of
iteration_initial_bytesanditeration_start_timein a explicit way to avoid some potential
missing?
Quoted 3 lines
Quoted 1 line
You may get some point. Well after a close look, we may find other
potential
Quoted 4 lines
problem.
1. To be consistency, we need to update
iteration_initial_pagestoo. So my opinion is to wrap the update of these three counters into a
helper
Quoted 2 lines
function. So each time all of them.
2. In functionram_get_total_transferred_pages, do we missedmultifd_bytes?
In function ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?
Thanks.
Quoted 32 lines · Wei Yang
Quoted 7 lines · Ivan Ren
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.Quoted 3 lines
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);Quoted 6 lines
Quoted 1 line
Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Is that better to always match the update of
iteration_initial_bytesanditeration_start_timein a explicit way to avoid some potentialmissing?
Quoted 1 line
You may get some point. Well after a close look, we may find other potential
problem.1. To be consistency, we need to update
iteration_initial_pagestoo. So my opinion is to wrap the update of these three counters into a
helper
function. So each time all of them.2. In function
ram_get_total_transferred_pages, do we missedmultifd_bytes?--
Wei Yang
Help you, Help me
Quoted 11 lines · Ivan Ren
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Is that better to always match the update of
iteration_initial_bytesanditeration_start_timein a explicit way to avoid some potentialmissing?
Quoted 3 lines
Quoted 1 line
You may get some point. Well after a close look, we may find other
potential
Quoted 2 lines
problem.
Well, I guess you need to use another tool to send mail. The format is corrupted.
Quoted 4 lines
Quoted 2 lines
1. To be consistency, we need to update
iteration_initial_pagestoo. So my opinion is to wrap the update of these three counters into ahelper
Quoted 1 line
function. So each time all of them.
I don't see you reply this one or the mail is corrupted.
If we don't update iteration_initial_pages, the initial_pages will mismatch the initial_bytes. Am I right?
Quoted 7 lines
Quoted 1 line
2. In function
ram_get_total_transferred_pages, do we missedmultifd_bytes?In function
ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?
Ok, it is counted in normal.
While if my understanding is correct, normal is used to count pages sent by save_normal_page(). Sounds this is misused?
Quoted 5 lines
Thanks.
On Fri, Aug 2, 2019 at 8:49 AM Wei Yang <[email protected]> wrote:
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 18 lines · Wei Yang
Quoted 16 lines · Ivan Ren
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Review
Quoted 3 lines
s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);(quoted diff) +2
-0+ /* + * Update s->iteration_initial_bytes to match
s->iteration_start_time.Quoted 3 lines
Quoted 2 lines
(quoted diff) +2
-0+ */ + s->iteration_initial_bytes = migration_total_bytes(s);Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Is this one necessary? We have sent out nothing yet.
Yes, currently nothing has been sent yet at this point.
Is that better to always match the update of
iteration_initial_bytesanditeration_start_timein a explicit way to avoid some potentialmissing?
Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
You may get some point. Well after a close look, we may find other
potential
Quoted 2 lines
problem.
Well, I guess you need to use another tool to send mail. The format is corrupted.
OK
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
1. To be consistency, we need to update
iteration_initial_pagestoo. So my opinion is to wrap the update of these three counters into ahelper
Quoted 1 line
function. So each time all of them.
I don't see you reply this one or the mail is corrupted.
If we don't update
iteration_initial_pages, theinitial_pageswill mismatch theinitial_bytes. Am I right?
Yes, agree, I'll send a new version, thanks.
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
2. In function
ram_get_total_transferred_pages, do we missedmultifd_bytes?In function
ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?
Quoted 5 lines
Ok, it is counted in normal.
While if my understanding is correct, normal is used to count pages sent by
save_normal_page(). Sounds this is misused?
Yes, current it is counted in normal, a specific counter is more accurate.
Thanks