Skip to contentSkip to conversation

migration: always initial ram_counters for a new migration

PATCH Superseded Ivan Ren 3 participants 8 messages Jul 30, 2019 – Aug 2, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ivan Ren

author
#0 lore raw

PATCH 1/1 migration: always initial ram_counters for a new migration

From: Ivan Ren <[email protected]>

This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run into migration_completion phase

Reason as follows:

Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:

    current_bytes = migration_total_bytes(s);
    transferred = current_bytes - s->iteration_initial_bytes;
    time_spent = current_time - s->iteration_start_time;
    bandwidth = (double)transferred / time_spent;
    s->threshold_size = bandwidth * s->parameters.downtime_limit;

In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new migration, but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may lead pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.

---
 migration/migration.c | 15 ++++++++++++++-
 migration/savevm.c    |  1 +
 2 files changed, 15 insertions(+), 1 deletion(-)
migration/migration.c +14 -1
index 8a607fe1e2..d35a6ae6f9 100644
@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
     }
 
     migrate_init(s);
+    /*
+     * set ram_counters memory to zero for a
+     * new migration
+     */
+    memset(&ram_counters, 0, sizeof(ram_counters));
 
     return true;
 }
@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)
 
     object_ref(OBJECT(s));
     s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    /*
+     * Update s->iteration_initial_bytes to match s->iteration_start_time.
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);
 
     qemu_savevm_state_header(s->to_dst_file);
 
@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)
              * breaking transferred_bytes and bandwidth calculation
              */
             s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-            s->iteration_initial_bytes = 0;
+            /*
+             * Update s->iteration_initial_bytes to current size to
+             * avoid historical data lead wrong bandwidth.
+             */
+            s->iteration_initial_bytes = migration_total_bytes(s);
         }
 
         current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
migration/savevm.c +1 -0
index 79ed44d475..480c511b19 100644
@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
     }
 
     migrate_init(ms);
+    memset(&ram_counters, 0, sizeof(ram_counters));
     ms->to_dst_file = f;
 
     qemu_mutex_unlock_iothread();
Signature · 1 line
2.17.2 (Apple Git-113)
Reviewed-by on v1 1/1
#1 lore raw
Quoted 31 lines · Ivan Ren

From: Ivan Ren <[email protected]>

This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run into migration_completion phase

Reason as follows:

Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:

    current_bytes = migration_total_bytes(s);
    transferred = current_bytes - s->iteration_initial_bytes;
    time_spent = current_time - s->iteration_start_time;
    bandwidth = (double)transferred / time_spent;
    s->threshold_size = bandwidth * s->parameters.downtime_limit;

In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new migration, but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may lead pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.

Quoted 1 line
#2 lore raw
In reply to Ivan Ren
Quoted 54 lines · Ivan Ren

From: Ivan Ren <[email protected]>

This patch fix a multifd migration bug in migration speed calculation, this problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred bytes]
4. migrate cancel
5. begin a new migration with multifd, the migration will directly run into migration_completion phase

Reason as follows:

Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:

   current_bytes = migration_total_bytes(s);
   transferred = current_bytes - s->iteration_initial_bytes;
   time_spent = current_time - s->iteration_start_time;
   bandwidth = (double)transferred / time_spent;
   s->threshold_size = bandwidth * s->parameters.downtime_limit;

In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new migration, but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may lead pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.

---
migration/migration.c | 15 ++++++++++++++-
migration/savevm.c | 1 +

2 files changed, 15 insertions(+), 1 deletion(-)
migration/migration.c +5 -0
index 8a607fe1e2..d35a6ae6f9 100644
@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
    }
 
    migrate_init(s);
+    /*
+     * set ram_counters memory to zero for a
+     * new migration
+     */
+    memset(&ram_counters, 0, sizeof(ram_counters));
 
    return true;
 }

migration/migration.c static void *migration_thread(void *opaque) · line 3198

Quoted 7 lines
migration/migration.c +4 -0
@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)
 
    object_ref(OBJECT(s));
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    /*
+     * Update s->iteration_initial_bytes to match s->iteration_start_time.
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);

Is this one necessary? We have sent out nothing yet.

Quoted 31 lines
migration/migration.c +5 -1
    qemu_savevm_state_header(s->to_dst_file);
 
@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)
             * breaking transferred_bytes and bandwidth calculation
             */
            s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-            s->iteration_initial_bytes = 0;
+            /*
+             * Update s->iteration_initial_bytes to current size to
+             * avoid historical data lead wrong bandwidth.
+             */
+            s->iteration_initial_bytes = migration_total_bytes(s);
        }
 
        current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
migration/savevm.c +1 -0
index 79ed44d475..480c511b19 100644
@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
    }
 
    migrate_init(ms);
+    memset(&ram_counters, 0, sizeof(ram_counters));
    ms->to_dst_file = f;
 
    qemu_mutex_unlock_iothread();
Signature · 1 line
2.17.2 (Apple Git-113)
Signature · 2 lines
Wei Yang
Help you, Help me
author
#3 lore raw
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential missing?

Thanks.

Quoted 110 lines · Wei Yang
Quoted 3 lines · Ivan Ren

From: Ivan Ren <[email protected]>

This patch fix a multifd migration bug in migration speed calculation,

this

Quoted 5 lines

problem can be reproduced as follows:
1. start a vm and give a heavy memory write stress to prevent the vm be successfully migrated to destination
2. begin a migration with multifd
3. migrate for a long time [actually, this can be measured by transferred

bytes]

Quoted 2 lines

4. migrate cancel
5. begin a new migration with multifd, the migration will directly run

into

Quoted 13 lines

migration_completion phase

Reason as follows:

Migration update bandwidth and s->threshold_size in function migration_update_counters after BUFFER_DELAY time:

   current_bytes = migration_total_bytes(s);
   transferred = current_bytes - s->iteration_initial_bytes;
   time_spent = current_time - s->iteration_start_time;
   bandwidth = (double)transferred / time_spent;
   s->threshold_size = bandwidth * s->parameters.downtime_limit;
Quoted 3 lines

In multifd migration, migration_total_bytes function return qemu_ftell(s->to_dst_file) + ram_counters.multifd_bytes. s->iteration_initial_bytes will be initialized to 0 at every new

migration,

Quoted 2 lines

but ram_counters is a global variable, and history migration data will be accumulated. So if the ram_counters.multifd_bytes is big enough, it may

lead

migration/migration.c static bool migrate_prepare(MigrationState *s,

Quoted 14 lines

pending_size >= s->threshold_size become false in migration_iteration_run after the first migration_update_counters.

---
migration/migration.c | 15 ++++++++++++++-
migration/savevm.c | 1 +

2 files changed, 15 insertions(+), 1 deletion(-)
migration/migration.c +0 -0
index 8a607fe1e2..d35a6ae6f9 100644
@@ -1908,6 +1908,11 @@ static bool migrate_prepare(MigrationState *s,

bool blk, bool blk_inc,

Quoted 12 lines
migration/migration.c +5 -0
    }
 
    migrate_init(s);
+    /*
+     * set ram_counters memory to zero for a
+     * new migration
+     */
+    memset(&ram_counters, 0, sizeof(ram_counters));
 
    return true;
 }

migration/migration.c static void *migration_thread(void *opaque) · line 3196

Quoted 5 lines
migration/migration.c +2 -0
@@ -3187,6 +3192,10 @@ static void *migration_thread(void *opaque)
 
    object_ref(OBJECT(s));
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

migration/migration.c static void *migration_thread(void *opaque) · line 3198

Quoted 2 lines
migration/migration.c +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);

Is this one necessary? We have sent out nothing yet.

migration/migration.c static void *migration_thread(void *opaque) · line 3263

Quoted 7 lines
migration/migration.c +0 -0
    qemu_savevm_state_header(s->to_dst_file);
 
@@ -3252,7 +3261,11 @@ static void *migration_thread(void *opaque)
             * breaking transferred_bytes and bandwidth calculation
             */
            s->iteration_start_time =

qemu_clock_get_ms(QEMU_CLOCK_REALTIME);

migration/savevm.c static int qemu_savevm_state(QEMUFile *f, Error

Quoted 14 lines
migration/migration.c +5 -1
-            s->iteration_initial_bytes = 0;
+            /*
+             * Update s->iteration_initial_bytes to current size to
+             * avoid historical data lead wrong bandwidth.
+             */
+            s->iteration_initial_bytes = migration_total_bytes(s);
        }
 
        current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
migration/savevm.c +0 -0
index 79ed44d475..480c511b19 100644
@@ -1424,6 +1424,7 @@ static int qemu_savevm_state(QEMUFile *f, Error

**errp)

Quoted 10 lines
migration/savevm.c +1 -1
    }
 
    migrate_init(ms);
+    memset(&ram_counters, 0, sizeof(ram_counters));
    ms->to_dst_file = f;
 
    qemu_mutex_unlock_iothread();
--

2.17.2 (Apple Git-113)

--
Wei Yang
Help you, Help me

#4 lore raw
Quoted 7 lines · Ivan Ren
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 3 lines
Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 7 lines
Quoted 1 line

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential missing?

You may get some point. Well after a close look, we may find other potential problem.

1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a helper function. So each time all of them.

2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?

Signature · 2 lines
Wei Yang
Help you, Help me
author
#5 lore raw
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 3 lines
Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential

missing?

Quoted 3 lines
Quoted 1 line

You may get some point. Well after a close look, we may find other

potential

Quoted 4 lines

problem.

1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a

helper

Quoted 2 lines

function. So each time all of them.
2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?

In function ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?

Thanks.

Quoted 32 lines · Wei Yang
Quoted 7 lines · Ivan Ren
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 3 lines
Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 6 lines
Quoted 1 line

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential

missing?

Quoted 1 line

You may get some point. Well after a close look, we may find other potential
problem.

1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a
helper
function. So each time all of them.

2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?

--
Wei Yang
Help you, Help me

#6 lore raw
Quoted 11 lines · Ivan Ren
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential

missing?

Quoted 3 lines
Quoted 1 line

You may get some point. Well after a close look, we may find other

potential

Quoted 2 lines

problem.

Well, I guess you need to use another tool to send mail. The format is corrupted.

Quoted 4 lines
Quoted 2 lines

1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a

helper

Quoted 1 line

function. So each time all of them.

I don't see you reply this one or the mail is corrupted.

If we don't update iteration_initial_pages, the initial_pages will mismatch the initial_bytes. Am I right?

Quoted 7 lines
Quoted 1 line

2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?

In function ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?

Ok, it is counted in normal.

While if my understanding is correct, normal is used to count pages sent by save_normal_page(). Sounds this is misused?

Quoted 5 lines

Thanks.

On Fri, Aug 2, 2019 at 8:49 AM Wei Yang <[email protected]> wrote:

Signature · 2 lines
Wei Yang
Help you, Help me
author
#7 lore raw
Quoted 18 lines · Wei Yang
Quoted 16 lines · Ivan Ren
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Review

Quoted 3 lines
    s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
(quoted diff) +2 -0
+    /*
+     * Update s->iteration_initial_bytes to match

s->iteration_start_time.

Quoted 3 lines
Quoted 2 lines
(quoted diff) +2 -0
+     */
+    s->iteration_initial_bytes = migration_total_bytes(s);
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Is this one necessary? We have sent out nothing yet.

Yes, currently nothing has been sent yet at this point.

Is that better to always match the update of iteration_initial_bytes and iteration_start_time in a explicit way to avoid some potential

missing?

Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

You may get some point. Well after a close look, we may find other

potential

Quoted 2 lines

problem.

Well, I guess you need to use another tool to send mail. The format is corrupted.

OK

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

1. To be consistency, we need to update iteration_initial_pages too. So my opinion is to wrap the update of these three counters into a

helper

Quoted 1 line

function. So each time all of them.

I don't see you reply this one or the mail is corrupted.

If we don't update iteration_initial_pages, the initial_pages will mismatch the initial_bytes. Am I right?

Yes, agree, I'll send a new version, thanks.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

2. In function ram_get_total_transferred_pages, do we missed multifd_bytes?

In function ram_save_multifd_page, ram pages transferred by multifd threads is
counted by ram_counters.normal.
You mean other multifd bytes like multifd packet or multifd sync info?

Quoted 5 lines

Ok, it is counted in normal.

While if my understanding is correct, normal is used to count pages sent by save_normal_page(). Sounds this is misused?

Yes, current it is counted in normal, a specific counter is more accurate.

Thanks