Skip to contentSkip to conversation

ram_save_live: add a no-progress convergence rule

PATCH In review Uri Lublin 6 participants 15 messages May 19, 2009 – May 20, 2009

lore
Share & tools
lore

Uri Lublin

author
#0 lore raw

PATCH 1/1 ram_save_live: add a no-progress convergence rule

Currently the live-part (section QEMU_VM_SECTION_PART) of ram_save_live has only one convergence rule, which is when the number of dirty pages is smaller than a threshold.

When the guest uses more memory pages than the threshold (e.g. playing a movie, copying files, sending/receiving many packets), it may take a very long time before convergence according to this rule.

This patch (re)introduces a no-progress convergence rule, which limit the number of times the migration process is not progressing (and even regressing), with regards to the number of dirty pages. No-progress means that the number of pages that got dirty is larger than the number of pages that got transferred to the destination during the last transfer.
This rule applies only after the first round (in which most memory pages are being transferred).

Also this patch enlarges the number-dirty-pages threshold (of the first convergence rule) to 50 pages (was 10)

---
 vl.c |   25 +++++++++++++++++++++++--
 1 files changed, 23 insertions(+), 2 deletions(-)
vl.c +23 -2
index 40b1d8b..5f145a0 100644
@@ -3181,6 +3181,11 @@ static void ram_decompress_close(RamDecompressState *s)
 #define RAM_SAVE_FLAG_PAGE	0x08
 #define RAM_SAVE_FLAG_EOS	0x10
 
+static ram_addr_t ram_save_threshold = 50;
+static unsigned ram_save_no_progress_max = 10;
+static unsigned ram_save_no_progress = 0;
+static ram_addr_t ram_save_rounds = 0;
+
 static int is_dup_page(uint8_t *page, uint8_t ch)
 {
     uint32_t val = ch << 24 | ch << 16 | ch << 8 | ch;
@@ -3225,12 +3230,13 @@ static int ram_save_block(QEMUFile *f)
         }
         addr += TARGET_PAGE_SIZE;
         current_addr = (saved_addr + addr) % last_ram_offset;
+        if (current_addr == 0)
+            ram_save_rounds++;
     }
 
     return found;
 }
 
-static ram_addr_t ram_save_threshold = 10;
 
 static ram_addr_t ram_save_remaining(void)
 {
@@ -3245,11 +3251,26 @@ static ram_addr_t ram_save_remaining(void)
     return count;
 }
 
+static int ram_save_is_converged(void)
+{
+    const ram_addr_t count = ram_save_remaining();
+    static ram_addr_t last_count = 0;
+
+    if ((count > last_count) && (ram_save_rounds > 0))
+        ram_save_no_progress++;
+    last_count = count;
+
+    return ((count < ram_save_threshold) ||
+            (ram_save_no_progress > ram_save_no_progress_max));
+}
+
 static int ram_save_live(QEMUFile *f, int stage, void *opaque)
 {
     ram_addr_t addr;
 
     if (stage == 1) {
+        ram_save_rounds = 0;
+        ram_save_no_progress = 0;
         /* Make sure all dirty bits are set */
         for (addr = 0; addr < last_ram_offset; addr += TARGET_PAGE_SIZE) {
             if (!cpu_physical_memory_get_dirty(addr, MIGRATION_DIRTY_FLAG))
@@ -3281,7 +3302,7 @@ static int ram_save_live(QEMUFile *f, int stage, void *opaque)
 
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
 
-    return (stage == 2) && (ram_save_remaining() < ram_save_threshold);
+    return (stage == 2) && ram_save_is_converged();
 }
 
 static int ram_load_dead(QEMUFile *f, void *opaque)
Signature · 1 line
1.6.0.6
#1 lore raw
Quoted 18 lines · Uri Lublin

Currently the live-part (section QEMU_VM_SECTION_PART) of ram_save_live has only one convergence rule, which is when the number of dirty pages is smaller than a threshold.

When the guest uses more memory pages than the threshold (e.g. playing a movie, copying files, sending/receiving many packets), it may take a very long time before convergence according to this rule.

This patch (re)introduces a no-progress convergence rule, which limit the number of times the migration process is not progressing (and even regressing), with regards to the number of dirty pages. No-progress means that the number of pages that got dirty is larger than the number of pages that got transferred to the destination during the last transfer.
This rule applies only after the first round (in which most memory pages are being transferred).

Quoted 5 lines

Also this patch enlarges the number-dirty-pages threshold (of the first convergence rule) to 50 pages (was 10)

The right place to do this is in a management tool. An arbitrary convergence rule of 50 can do more damage than good.

For some set of users, it's better that live migration fail than it cause an arbitrarily long pause in the guest which can result in dropped TCP connections, soft lock ups, and other badness.

A management tool can force convergence by issuing a "stop" command in the monitor. I suspect a management tool cares more about wall-clock time than number of iterations too so a valid metric would be something along the lines of if not converged after N seconds, issue stop monitor command where N is calculated from available network bandwidth and guest memory size.

Regards,

Anthony Liguori

#2 lore raw
Quoted 32 lines · Anthony Liguori
Quoted 18 lines · Uri Lublin

Currently the live-part (section QEMU_VM_SECTION_PART) of ram_save_live has only one convergence rule, which is when the number of dirty pages is smaller than a threshold.

When the guest uses more memory pages than the threshold (e.g. playing a movie, copying files, sending/receiving many packets), it may take a very long time before convergence according to this rule.

This patch (re)introduces a no-progress convergence rule, which limit the number of times the migration process is not progressing (and even regressing), with regards to the number of dirty pages. No-progress means that the number of pages that got dirty is larger than the number of pages that got transferred to the destination during the last transfer.
This rule applies only after the first round (in which most memory pages are being transferred).

Quoted 5 lines

Also this patch enlarges the number-dirty-pages threshold (of the first convergence rule) to 50 pages (was 10)

The right place to do this is in a management tool. An arbitrary convergence rule of 50 can do more damage than good.

For some set of users, it's better that live migration fail than it cause an arbitrarily long pause in the guest which can result in dropped TCP connections, soft lock ups, and other badness.

Quoted 6 lines

A management tool can force convergence by issuing a "stop" command in the monitor. I suspect a management tool cares more about wall-clock time than number of iterations too so a valid metric would be something along the lines of if not converged after N seconds, issue stop monitor command where N is calculated from available network bandwidth and guest memory size.

Another possibility is for the management tool to increase the bandwidth for little periods if it perceives that no progress is being made.

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about the internal state of migration, so the mgmt tool can take a more informed decision?

#3 lore raw
Quoted 34 lines · Glauber Costa
Quoted 23 lines · Anthony Liguori
Quoted 21 lines · Uri Lublin

Currently the live-part (section QEMU_VM_SECTION_PART) of ram_save_live has only one convergence rule, which is when the number of dirty pages is smaller than a threshold.

When the guest uses more memory pages than the threshold (e.g. playing a movie, copying files, sending/receiving many packets), it may take a very long time before convergence according to this rule.

This patch (re)introduces a no-progress convergence rule, which limit the number of times the migration process is not progressing (and even regressing), with regards to the number of dirty pages. No-progress means that the number of pages that got dirty is larger than the number of pages that got transferred to the destination during the last transfer.
This rule applies only after the first round (in which most memory pages are being transferred).

Also this patch enlarges the number-dirty-pages threshold (of the first convergence rule) to 50 pages (was 10)

Quoted 9 lines
Quoted 3 lines

The right place to do this is in a management tool. An arbitrary convergence rule of 50 can do more damage than good.

For some set of users, it's better that live migration fail than it cause an arbitrarily long pause in the guest which can result in dropped TCP connections, soft lock ups, and other badness.

A management tool can force convergence by issuing a "stop" command in the monitor. I suspect a management tool cares more about wall-clock time than number of iterations too so a valid metric would be something along the lines of if not converged after N seconds, issue stop monitor command where N is calculated from available network bandwidth and guest memory size.

Quoted 10 lines

Another possibility is for the management tool to increase the bandwidth for little periods if it perceives that no progress is being made.

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about the internal state of migration, so the mgmt tool can take a more informed decision?

The problem is that if migration is not progressing since the guest is dirtying pages faster than the migration protocol can send, than we just waist time and cpu. The minimum is to notify the monitor interface in order to let mgmt daemon to trap it. We can easily see this issue while running iperf in the guest or any other high load/dirty pages scenario.

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

#4 lore raw
Quoted 55 lines · Dor Laor
Quoted 46 lines · Glauber Costa
Quoted 33 lines · Anthony Liguori
Quoted 18 lines · Uri Lublin

Currently the live-part (section QEMU_VM_SECTION_PART) of ram_save_live has only one convergence rule, which is when the number of dirty pages is smaller than a threshold.

When the guest uses more memory pages than the threshold (e.g. playing a movie, copying files, sending/receiving many packets), it may take a very long time before convergence according to this rule.

This patch (re)introduces a no-progress convergence rule, which limit the number of times the migration process is not progressing (and even regressing), with regards to the number of dirty pages. No-progress means that the number of pages that got dirty is larger than the number of pages that got transferred to the destination during the last transfer.
This rule applies only after the first round (in which most memory pages are being transferred).

Quoted 6 lines

Also this patch enlarges the number-dirty-pages threshold (of the first convergence rule) to 50 pages (was 10)

The right place to do this is in a management tool. An arbitrary convergence rule of 50 can do more damage than good.

For some set of users, it's better that live migration fail than it cause an arbitrarily long pause in the guest which can result in dropped TCP connections, soft lock ups, and other badness.

Quoted 7 lines

A management tool can force convergence by issuing a "stop" command in the monitor. I suspect a management tool cares more about wall-clock time than number of iterations too so a valid metric would be something along the lines of if not converged after N seconds, issue stop monitor command where N is calculated from available network bandwidth and guest memory size.

Another possibility is for the management tool to increase the bandwidth for
little periods if it perceives that no progress is being made.

Quoted 8 lines

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about
the internal state of migration, so the mgmt tool can take a more informed decision?

Quoted 9 lines

The problem is that if migration is not progressing since the guest is dirtying pages
faster than the migration protocol can send, than we just waist time and cpu.
The minimum is to notify the monitor interface in order to let mgmt daemon to trap it.
We can easily see this issue while running iperf in the guest or any other high load/dirty
pages scenario.

I know that, seen myself. What I believe and insist is only that qemu does not really have to possess the knowledge to deal with it. Providing migration stats in info migrate seems to me the better thing to do, rather than one single one size fits all notification. The mgmt tool then can take the appropriate action depending on the scenario it has in mind.

Quoted 3 lines

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

#5 lore raw
Quoted 10 lines · Glauber Costa
Quoted 3 lines · Dor Laor

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

Threshold is a bad metric. There's no way to choose a right number. If we were going to have a means to support metrics-based forced convergence (and I really think this belongs in libvirt) I'd rather see something based on bandwidth or wall clock time.

Let me put it this way, why 50? What were the guidelines for choosing that number and how would you explain what number a user should choose?

Regards,

Anthony Liguori

author
#6 lore raw
Quoted 11 lines · Anthony Liguori
Quoted 9 lines · Glauber Costa
Quoted 3 lines · Dor Laor

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

Quoted 7 lines

Threshold is a bad metric. There's no way to choose a right number. If we were going to have a means to support metrics-based forced convergence (and I really think this belongs in libvirt) I'd rather see something based on bandwidth or wall clock time.

Let me put it this way, why 50? What were the guidelines for choosing that number and how would you explain what number a user should choose?

I've changed the threshold of the first convergence rule, to 50 from 10. Why 10 ? For this rule the threshold (number of dirty pages) and the number of bytes to transfer are equivalent.

50 pages is about 200K, which can be still sent quickly.
I've added debug messages and noticed we never hit a number smaller than 10 (excluding 0). The truth is there were very little number of runs with less than 50 dirty pages too. I don't mind leaving it at 10 (should be configurable too).

For the second migration convergence rule I've set the limit to 10, as it seems much larger than what I've needed (all the runs I've made a number of 2-4 no-progress iterations was good enough, as it seems to have a repetitive behavior later), but I've enlarged it "just in case". No real research work was done here.

Note that a no-progress iteration depends on both network bandwidth and guest actions.

Regards,
Uri.

#7 lore raw
Quoted 36 lines · Uri Lublin
Quoted 15 lines · Anthony Liguori
Quoted 12 lines · Glauber Costa
Quoted 4 lines · Dor Laor

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

Quoted 8 lines

Threshold is a bad metric. There's no way to choose a right number. If we were going to have a means to support metrics-based forced convergence (and I really think this belongs in libvirt) I'd rather see something based on bandwidth or wall clock time.

Let me put it this way, why 50? What were the guidelines for choosing that number and how would you explain what number a user should choose?

I've changed the threshold of the first convergence rule, to 50 from 10.
Why 10 ? For this rule the threshold (number of dirty pages) and the number of bytes to transfer are equivalent.

50 pages is about 200K, which can be still sent quickly.
I've added debug messages and noticed we never hit a number smaller than 10
(excluding 0). The truth is there were very little number of runs with less than 50 dirty pages too. I don't mind leaving it at 10 (should be configurable too).

Quoted 8 lines

For the second migration convergence rule I've set the limit to 10, as it
seems much larger than what I've needed (all the runs I've made a number of 2-4 no-progress iterations was good enough, as it seems to have a repetitive behavior later), but I've enlarged it "just in case". No real research work was done here.

Note that a no-progress iteration depends on both network bandwidth and
guest actions.

Instead of freezing the guest or aborting the migration, the guest could be throttled a bit by giving it less CPU time relative to migration, or by incurring a small delay for each page dirtying write access. Maybe this method would find the balance faster.

author
#8 lore raw
Quoted 41 lines · Blue Swirl
Quoted 40 lines · Uri Lublin
Quoted 14 lines · Anthony Liguori
Quoted 12 lines · Glauber Costa
Quoted 4 lines · Dor Laor

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

Quoted 8 lines

Threshold is a bad metric. There's no way to choose a right number. If we were going to have a means to support metrics-based forced convergence (and I really think this belongs in libvirt) I'd rather see something based on bandwidth or wall clock time.

Let me put it this way, why 50? What were the guidelines for choosing that number and how would you explain what number a user should choose?

I've changed the threshold of the first convergence rule, to 50 from 10.
Why 10 ? For this rule the threshold (number of dirty pages) and the number of bytes to transfer are equivalent.

50 pages is about 200K, which can be still sent quickly.
I've added debug messages and noticed we never hit a number smaller than 10
(excluding 0). The truth is there were very little number of runs with less than 50 dirty pages too. I don't mind leaving it at 10 (should be configurable too).

For the second migration convergence rule I've set the limit to 10, as it
seems much larger than what I've needed (all the runs I've made a number of 2-4 no-progress iterations was good enough, as it seems to have a repetitive behavior later), but I've enlarged it "just in case". No real research work was done here.

Quoted 7 lines
Quoted 2 lines

Note that a no-progress iteration depends on both network bandwidth and
guest actions.

Instead of freezing the guest or aborting the migration, the guest could be throttled a bit by giving it less CPU time relative to migration, or by incurring a small delay for each page dirtying write access. Maybe this method would find the balance faster.

Enlarging the bandwidth limitation (while migration is active) is one way of doing that (give more cpu to migration code and less to guest). I think currently it's not possible though. I don't think I want to "play" with delaying write accesses.

#9 lore raw
In reply to Glauber Costa
Quoted 7 lines · Glauber Costa
Quoted 3 lines

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

it can be done, but it fits better as a different monitor command

anthony, do you have any strong opinions here, or is this scheme acceptable?

Oh, and to add to my last post, the action to take after a certain time period is failing the migration. Forced convergence is a bug.

Regards,

Anthony Liguori

#10 lore raw
In reply to Dor Laor
Quoted 5 lines · Dor Laor

The problem is that if migration is not progressing since the guest is dirtying pages faster than the migration protocol can send, than we just waist time and cpu. The minimum is to notify the monitor interface in order to let mgmt daemon to trap it. We can easily see this issue while running iperf in the guest or any other high load/dirty pages scenario.

The problem is, what's the metric for determining the guest isn't progressing? A raw iteration count is not a valid metric. It may be expected that the migration take 50 iterations.

The management tool knows the guest isn't progressing when it decides that a guest isn't progressing :-)

Quoted 2 lines

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

Theshold is really a bad metric to use. You have no idea how much data has been passed in each iteration. If you only needed one more iteration, then stopping the migration short was a really bad idea.

The only thing that this does is give a false sense of security. Management tools have to deal with forcing migration convergence based on policies. If a management tool isn't doing this today, it's broken IMHO.

Basically, threshold introduces a regression. If you run iperf and migrate a guest with a very large memory size, after migration, you'll get soft lockups because the guest hasn't been running for 10 seconds. This is bad.

Regards,

Anthony Liguori

author
#11 lore raw
Quoted 11 lines · Anthony Liguori
Quoted 9 lines · Dor Laor

The problem is that if migration is not progressing since the guest is dirtying pages
faster than the migration protocol can send, than we just waist time and cpu.
The minimum is to notify the monitor interface in order to let mgmt daemon to trap it.
We can easily see this issue while running iperf in the guest or any other high load/dirty
pages scenario.

Quoted 3 lines

The problem is, what's the metric for determining the guest isn't progressing? A raw iteration count is not a valid metric. It may be expected that the migration take 50 iterations.

We've defined "no-progress" as a memory transfer iteration where the number of pages that got dirty is larger than the number of pages transferred. For such iterations we have more data to transfer when the iteration completes. Note that we did not limit the number of iterations (yet), we want to limit the number of no-progress iterations. Migrations with many such iterations just waste resources (cpu, network, etc).

Quoted 3 lines

The management tool knows the guest isn't progressing when it decides that a guest isn't progressing :-)

Currently the management tool only knows the migration is still active.

Quoted 8 lines
Quoted 4 lines

We can also make it configurable using the monitor migrate command. For example:
migrate -d -no_progress -threshold=x tcp:....

Theshold is really a bad metric to use. You have no idea how much data has been passed in each iteration. If you only needed one more iteration, then stopping the migration short was a really bad idea.

You can never know there is only one more iteration needed, no matter what metric you use. Again this threshold limits the number of no-progress iterations.

We can extend this rule (or add another flag/command) to enlarge the bandwidth limitation upon a no-progress iteration.

Quoted 4 lines

The only thing that this does is give a false sense of security. Management tools have to deal with forcing migration convergence based on policies. If a management tool isn't doing this today, it's broken IMHO.

I agree migration convergence rules should be based on policies.

What Dor is suggesting is that the management tool do that by passing parameters to the migrate command (or using other migrate_X monitor commands).

I'm not sure management tools can have good such policies today. The only information they have is how much time passed since the migration started. The only actions they can take is stop the guest or cancel the migration.

Quoted 5 lines

Basically, threshold introduces a regression. If you run iperf and migrate a guest with a very large memory size, after migration, you'll get soft lockups because the guest hasn't been running for 10 seconds. This is bad.

Just keep resending pages that are constantly changing is bad too, probably worse.

Regards,
Uri.

#12 lore raw
In reply to Glauber Costa
Quoted 9 lines · Glauber Costa

Another possibility is for the management tool to increase the bandwidth for little periods if it perceives that no progress is being made.

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about the internal state of migration, so the mgmt tool can take a more informed decision?

Yes, I've also suggested this before. I'm willing to expose just about any metric that makes sense. We need to be careful about not exposing implementation details, but things like iteration count, last working set size, average working set size, etc. should all be relatively stable metrics even if the implementation changes.

Regards,

Anthony Liguori

author
#13 lore raw
Quoted 12 lines · Anthony Liguori
Quoted 11 lines · Glauber Costa

Another possibility is for the management tool to increase the bandwidth for
little periods if it perceives that no progress is being made.

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about
the internal state of migration, so the mgmt tool can take a more informed
decision?

Quoted 6 lines

Yes, I've also suggested this before. I'm willing to expose just about any metric that makes sense. We need to be careful about not exposing implementation details, but things like iteration count, last working set size, average working set size, etc. should all be relatively stable metrics even if the implementation changes.

I agree we need to provide more information via "info migration". That's not enough though.

In addition of augmenting "info migration" we need to add more monitor commands to set/change migration parameters (e.g. current bandwidth limit), and change the migration code to act according to such parameters. These commands should affect the migration when used before and during migration.

Note that as management tool, most likely, call "info migration" periodically, it may miss information about some (current/last) statistics. Also the first iteration may cause some averages to get biased.

How would you recognize "stuck" migrations ? By comparing Average Working Set Size and Average Iteration Transfer Size ? Counting the number of no-progress iterations ? The average "regression" of no-progress iterations ?

The no-progress convergence rule was fairly easy to implement and gave a pretty good heuristic to recognizing the migration is stuck.

Regards,
Uri.

#14 lore raw
In reply to Glauber Costa
Quoted 16 lines · Glauber Costa
Quoted 9 lines · Anthony Liguori

Uri Lublin wrote:

The right place to do this is in a management tool. An arbitrary convergence rule of 50 can do more damage than good.

For some set of users, it's better that live migration fail than it cause an arbitrarily long pause in the guest which can result in dropped TCP connections, soft lock ups, and other badness.

Quoted 6 lines

A management tool can force convergence by issuing a "stop" command in the monitor. I suspect a management tool cares more about wall-clock time than number of iterations too so a valid metric would be something along the lines of if not converged after N seconds, issue stop monitor command where N is calculated from available network bandwidth and guest memory size.

Quoted 8 lines

Another possibility is for the management tool to increase the bandwidth for little periods if it perceives that no progress is being made.

Anyhow, I completely agree that we should not introduce this in qemu.

However, maybe we could augment our "info migrate" to provide more info about the internal state of migration, so the mgmt tool can take a more informed decision?

Yes, I think a 'info migration' command is neccessary regardless of whether conversion is progressing successfully or not. If a migration takes 60 seconds total in normal case, it is useful for mgmt tool can get some indication of how far it has progressed. eg, report pages sent, pages remaining, and total pages. NB, sent + remaining != total

The mgmt tool knows how much wall clock time has elapsed, and can either present progress info the admin. So after x seconds has elapsed the admin can see if it has nearly completed, or is stuck, and make a decision, or let the mgmt app apply policies of its own, cancelling the migration, or pausing the guest to let it complete non-live.

If you want to put a policy into QEMU too go ahead, as long as its optional so a mgmt app can have full control if desired.

Regards,
Daniel

Signature · 4 lines
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|