Skip to contentSkip to conversation

kvmclock: Ensure time in migration never goes backward (v3)

PATCH Applied Marcelo Tosatti 4 participants 16 messages Sep 5, 2014 – Sep 16, 2014

lore
Share & tools
lore
#0 lore raw

cover kvmclock: Ensure time in migration never goes backward (v3)

Regression fix tested with fio and ping-pong migration.

#1 lore raw

PATCH 1/3 kvmclock: Ensure time in migration never goes backward

From: Alexander Graf <[email protected]>

When we migrate we ask the kernel about its current belief on what the guest time would be. However, I've seen cases where the kvmclock guest structure indicates a time more recent than the kvm returned time.

To make sure we never go backwards, calculate what the guest would have seen as time at the point of migration and use that value instead of the kernel returned one when it's more recent. This bases the view of the kvmclock after migration on the
same foundation in host as well as guest.

---
 hw/i386/kvm/clock.c |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
hw/i386/kvm/clock.c +48 -0
Index: qemu/hw/i386/kvm/clock.c
===================================================================
@@ -14,6 +14,7 @@
  */
 
 #include "qemu-common.h"
+#include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
 #include "hw/sysbus.h"
@@ -34,6 +35,47 @@
     bool clock_valid;
 } KVMClockState;
 
+struct pvclock_vcpu_time_info {
+    uint32_t   version;
+    uint32_t   pad0;
+    uint64_t   tsc_timestamp;
+    uint64_t   system_time;
+    uint32_t   tsc_to_system_mul;
+    int8_t     tsc_shift;
+    uint8_t    flags;
+    uint8_t    pad[2];
+} __attribute__((__packed__)); /* 32 bytes */
+
+static uint64_t kvmclock_current_nsec(KVMClockState *s)
+{
+    CPUState *cpu = first_cpu;
+    CPUX86State *env = cpu->env_ptr;
+    hwaddr kvmclock_struct_pa = env->system_time_msr & ~1ULL;
+    uint64_t migration_tsc = env->tsc;
+    struct pvclock_vcpu_time_info time;
+    uint64_t delta;
+    uint64_t nsec_lo;
+    uint64_t nsec_hi;
+    uint64_t nsec;
+
+    if (!(env->system_time_msr & 1ULL)) {
+        /* KVM clock not active */
+        return 0;
+    }
+
+    cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
+
+    delta = migration_tsc - time.tsc_timestamp;
+    if (time.tsc_shift < 0) {
+        delta >>= -time.tsc_shift;
+    } else {
+        delta <<= time.tsc_shift;
+    }
+
+    mulu64(&nsec_lo, &nsec_hi, delta, time.tsc_to_system_mul);
+    nsec = (nsec_lo >> 32) | (nsec_hi << 32);
+    return nsec + time.system_time;
+}
 
 static void kvmclock_vm_state_change(void *opaque, int running,
                                      RunState state)
@@ -45,9 +87,15 @@
 
     if (running) {
         struct kvm_clock_data data;
+        uint64_t time_at_migration = kvmclock_current_nsec(s);
 
         s->clock_valid = false;
 
+	/* We can't rely on the migrated clock value, just discard it */
+	if (time_at_migration) {
+	        s->clock = time_at_migration;
+	}
+
         data.clock = s->clock;
         data.flags = 0;
         ret = kvm_vm_ioctl(kvm_state, KVM_SET_CLOCK, &data);
#2 lore raw

Re: [Qemu-devel] [patch 1/3] kvmclock: Ensure time in migration never goes backward

Ping?

Quoted 14 lines · Marcelo Tosatti

From: Alexander Graf <[email protected]>

When we migrate we ask the kernel about its current belief on what the guest time would be. However, I've seen cases where the kvmclock guest structure indicates a time more recent than the kvm returned time.

To make sure we never go backwards, calculate what the guest would have seen as time at the point of migration and use that value instead of the kernel returned one when it's more recent. This bases the view of the kvmclock after migration on the
same foundation in host as well as guest.

#3 lore raw

Re: [Qemu-devel] [patch 1/3] kvmclock: Ensure time in migration never goes backward

Who are you pinging?

Alex

Quoted 19 lines · Marcelo Tosatti

Ping?

Quoted 14 lines · Marcelo Tosatti

From: Alexander Graf <[email protected]>

When we migrate we ask the kernel about its current belief on what the guest time would be. However, I've seen cases where the kvmclock guest structure indicates a time more recent than the kvm returned time.

To make sure we never go backwards, calculate what the guest would have seen as time at the point of migration and use that value instead of the kernel returned one when it's more recent. This bases the view of the kvmclock after migration on the
same foundation in host as well as guest.

#4 lore raw

Re: [Qemu-devel] [patch 1/3] kvmclock: Ensure time in migration never goes backward

Quoted 1 line · Alexander Graf

Who are you pinging?

Paolo.

#5 lore raw

PATCH 2/3 Introduce cpu_clean_all_dirty

In reply to Marcelo Tosatti

Introduce cpu_clean_all_dirty, to force subsequent cpu_synchronize_all_states to read in-kernel register state.

---
 cpus.c                |    9 +++++++++
 include/sysemu/cpus.h |    1 +
 include/sysemu/kvm.h  |    8 ++++++++
 kvm-all.c             |    5 +++++
 4 files changed, 23 insertions(+)
cpus.c +9 -0
Index: qemu/cpus.c
===================================================================
@@ -589,6 +589,15 @@
     }
 }
 
+void cpu_clean_all_dirty(void)
+{
+    CPUState *cpu;
+
+    CPU_FOREACH(cpu) {
+        cpu_clean_state(cpu);
+    }
+}
+
 static int do_vm_stop(RunState state)
 {
     int ret = 0;
include/sysemu/cpus.h +1 -0
Index: qemu/include/sysemu/cpus.h
===================================================================
@@ -10,6 +10,7 @@
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
 void cpu_synchronize_all_post_init(void);
+void cpu_clean_all_dirty(void);
 
 void qtest_clock_warp(int64_t dest);
 
include/sysemu/kvm.h +8 -0
Index: qemu/include/sysemu/kvm.h
===================================================================
@@ -348,6 +348,7 @@
 void kvm_cpu_synchronize_state(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
+void kvm_cpu_clean_state(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -372,6 +373,13 @@
     }
 }
 
+static inline void cpu_clean_state(CPUState *cpu)
+{
+    if (kvm_enabled()) {
+        kvm_cpu_clean_state(cpu);
+    }
+}
+
 int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg);
 int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg);
 void kvm_irqchip_release_virq(KVMState *s, int virq);
kvm-all.c +5 -0
Index: qemu/kvm-all.c
===================================================================
@@ -1695,6 +1695,11 @@
     run_on_cpu(cpu, do_kvm_cpu_synchronize_post_init, cpu);
 }
 
+void kvm_cpu_clean_state(CPUState *cpu)
+{
+    cpu->kvm_vcpu_dirty = false;
+}
+
 int kvm_cpu_exec(CPUState *cpu)
 {
     struct kvm_run *run = cpu->kvm_run;
#6 lore raw

PATCH 3/3 kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec calculation

In reply to Marcelo Tosatti

Ensure proper env->tsc value for kvmclock_current_nsec calculation.

---
 hw/i386/kvm/clock.c |    5 +++++
 1 file changed, 5 insertions(+)
hw/i386/kvm/clock.c +5 -0
Index: qemu/hw/i386/kvm/clock.c
===================================================================
@@ -17,6 +17,7 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
+#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -65,6 +66,7 @@
 
     cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
 
+    assert(time.tsc_timestamp <= migration_tsc);
     delta = migration_tsc - time.tsc_timestamp;
     if (time.tsc_shift < 0) {
         delta >>= -time.tsc_shift;
@@ -123,6 +125,9 @@
         if (s->clock_valid) {
             return;
         }
+
+        cpu_synchronize_all_states();
+        cpu_clean_all_dirty();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
#7 lore raw
In reply to Marcelo Tosatti
Quoted 3 lines · Marcelo Tosatti

Regression fix tested with fio and ping-pong migration.

Thanks, actually this series also introduces regression with average live migration time - 42s avg vs 7s avg for the reference VM on ping-pong test. Can you please confirm this?

#8 lore raw
Quoted 8 lines · Andrey Korolyov
Quoted 3 lines · Marcelo Tosatti

Regression fix tested with fio and ping-pong migration.

Thanks, actually this series also introduces regression with average live migration time - 42s avg vs 7s avg for the reference VM on ping-pong test. Can you please confirm this?

Can't confirm - migration time is similar for me.

What workload are you using?

#9 lore raw

Il 05/09/2014 20:26, Marcelo Tosatti ha scritto:

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

Thanks, actually this series also introduces regression with average live migration time - 42s avg vs 7s avg for the reference VM on ping-pong test. Can you please confirm this?

Can't confirm - migration time is similar for me.

It's also fairly unlikely.

Paolo

#10 lore raw
In reply to Marcelo Tosatti
Quoted 6 lines · Marcelo Tosatti
Quoted 5 lines · Andrey Korolyov
Quoted 3 lines · Marcelo Tosatti

Regression fix tested with fio and ping-pong migration.

Quoted 8 lines
Quoted 3 lines

Thanks, actually this series also introduces regression with average live migration time - 42s avg vs 7s avg for the reference VM on ping-pong test. Can you please confirm this?

Can't confirm - migration time is similar for me.

What workload are you using?

Just an 'empty' VM on reference test and fio tests at the patch-specific test suite. Actual presence of workload brings difference for about 2s, effectively unmeasurable on a small series of 40+s migration times.

#11 lore raw
Quoted 13 lines · Andrey Korolyov
Quoted 12 lines · Marcelo Tosatti
Quoted 8 lines · Andrey Korolyov
Quoted 3 lines · Marcelo Tosatti

Regression fix tested with fio and ping-pong migration.

Thanks, actually this series also introduces regression with average live migration time - 42s avg vs 7s avg for the reference VM on ping-pong test. Can you please confirm this?

Can't confirm - migration time is similar for me.

Quoted 7 lines
Quoted 2 lines

What workload are you using?

Just an 'empty' VM on reference test and fio tests at the patch-specific test suite. Actual presence of workload brings difference for about 2s, effectively unmeasurable on a small series of 40+s migration times.

Heh... hit independent bug due to different pre-test conditions, the patch itself is ok.

Bug follows:
- boot up a VM, make sure to feed it heavy long task, for example kernel compilation,
- neither migrate once in the middle of process (with fallback to non-live migration which is available in libvirt, else it may take forever) or just wait up to the end,
- live migration time degrades from sub-10s value to larger values forever, not regarding if workload is still presented or not.

#12 lore raw

Il 05/09/2014 21:35, Andrey Korolyov ha scritto:

Quoted 7 lines

- boot up a VM, make sure to feed it heavy long task, for example kernel compilation,
- neither migrate once in the middle of process (with fallback to non-live migration which is available in libvirt, else it may take forever) or just wait up to the end,
- live migration time degrades from sub-10s value to larger values forever, not regarding if workload is still presented or not.

Are you sure you aren't simply doing migration with a low bandwidth limit (the default is 32 MB/s), and moving over a gigabyte worth of page cache data to the new machine?

Paolo

#13 lore raw
Quoted 9 lines · Paolo Bonzini

Il 05/09/2014 21:35, Andrey Korolyov ha scritto:

Quoted 7 lines

- boot up a VM, make sure to feed it heavy long task, for example kernel compilation,
- neither migrate once in the middle of process (with fallback to non-live migration which is available in libvirt, else it may take forever) or just wait up to the end,
- live migration time degrades from sub-10s value to larger values forever, not regarding if workload is still presented or not.

Quoted 5 lines

Are you sure you aren't simply doing migration with a low bandwidth limit (the default is 32 MB/s), and moving over a gigabyte worth of page cache data to the new machine?

Paolo

As far as I can see setspeed does not change anything over stdev variation for gigabit, but rss size does. Will check if this part is actually broken, because we are overriding default speed value every time and if it started being constant, it may explain such a difference.

#14 lore raw
Quoted 14 lines · Andrey Korolyov
Quoted 13 lines · Paolo Bonzini

Il 05/09/2014 21:35, Andrey Korolyov ha scritto:

Quoted 7 lines

- boot up a VM, make sure to feed it heavy long task, for example kernel compilation,
- neither migrate once in the middle of process (with fallback to non-live migration which is available in libvirt, else it may take forever) or just wait up to the end,
- live migration time degrades from sub-10s value to larger values forever, not regarding if workload is still presented or not.

Are you sure you aren't simply doing migration with a low bandwidth limit (the default is 32 MB/s), and moving over a gigabyte worth of page cache data to the new machine?

Quoted 7 lines
Quoted 1 line

Paolo

As far as I can see setspeed does not change anything over stdev variation for gigabit, but rss size does. Will check if this part is actually broken, because we are overriding default speed value every time and if it started being constant, it may explain such a difference.

The mess was caused by libvirt which sets migration bandwidth to 8 EB/s, so process took shorter times when I expect from it to utilize some percent of server interconnection bandwidth.

#15 lore raw
In reply to Marcelo Tosatti

Il 05/09/2014 15:52, Marcelo Tosatti ha scritto:

Quoted 1 line

Regression fix tested with fio and ping-pong migration.

I applied the patches, but in a different order (2-3-1) to keep them bisectable.

Can you send a follow-up patch, adding a comment before the call to cpu_clean_all_dirty that explains what's going on?

Paolo