Skip to contentSkip to conversation

savevm odness related to kvmclock

PATCH In review Glauber Costa 3 participants 9 messages Dec 6, 2010 – Dec 13, 2010 KVM

lore
Share & tools
lore
#0 lore raw

v2 cover savevm odness related to kvmclock

Some users told me that savevm path is behaving oddly wrt kvmclock. The first oddness is that a guarantee we never made (AFAIK) is being broken:
two consecutive "savevm" operations, with the machine stopped in between produces different results, due to the call to KVM_GET_CLOCK ioctl. I believe the assumption that if the vm does not run, its saveable state won't change is fairly reasonable. Maybe we should formally guarantee that?

Also, this patch deals with the fact that this happens even if kvmclock is disabled in cpuid: its savevm section is registered nevertheless. Here, I try to register it only if it's enabled at machine start.

v2: improvements suggested by Paolo, and patch reordering.

Glauber Costa (2):
Do not register kvmclock savevm section if kvmclock is disabled. make kvmclock value idempotent for stopped machine

 cpus.c            |    3 +++
 qemu-kvm-x86.c    |   23 +++++++++++++++--------
 qemu-kvm.h        |    3 +++
 target-i386/kvm.c |    7 +++++++
 4 files changed, 28 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.2.3
#1 lore raw

PATCH v2 1/2 Do not register kvmclock savevm section if kvmclock is disabled.

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

---
 cpus.c            |    3 +++
 qemu-kvm-x86.c    |   19 +++++++++++++------
 qemu-kvm.h        |    3 +++
 target-i386/kvm.c |    7 +++++++
 4 files changed, 26 insertions(+), 6 deletions(-)
cpus.c +3 -0
index a55c330..a24098e 100644
@@ -97,6 +97,9 @@ void cpu_synchronize_all_post_init(void)
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
         cpu_synchronize_post_init(cpu);
     }
+    if (kvm_enabled()) {
+        kvmclock_register_savevm();
+    }
 }
 
 int cpu_is_stopped(CPUState *env)
qemu-kvm-x86.c +13 -6
index 20b7d6d..14a52ce 100644
@@ -504,6 +504,9 @@ static void kvmclock_pre_save(void *opaque)
 {
     struct kvm_clock_data *cl = opaque;
 
+    if (!kvmclock_enabled)
+        return;
+
     kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, cl);
 }
 
@@ -528,6 +531,16 @@ static const VMStateDescription vmstate_kvmclock= {
 };
 #endif
 
+/* This has to happen after vcpu setup*/
+void kvmclock_register_savevm(void)
+{
+#ifdef KVM_CAP_ADJUST_CLOCK
+    if (kvmclock_enabled && kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
+        vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
+    }
+#endif
+}
+
 int kvm_arch_qemu_create_context(void)
 {
     int r;
@@ -545,12 +558,6 @@ int kvm_arch_qemu_create_context(void)
         return -1;
     }
 
-#ifdef KVM_CAP_ADJUST_CLOCK
-    if (kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
-        vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
-    }
-#endif
-
     r = kvm_set_boot_cpu_id(0);
     if (r < 0 && r != -ENOSYS) {
         return r;
qemu-kvm.h +3 -0
index 0f3fb50..0a104ef 100644
@@ -752,6 +752,9 @@ int handle_tpr_access(void *opaque, CPUState *env, uint64_t rip,
 #define qemu_kvm_cpu_stop(env) do {} while(0)
 #endif
 
+extern int kvmclock_enabled;
+void kvmclock_register_savevm(void);
+
 #ifdef CONFIG_KVM
 
 typedef struct KVMSlot {
target-i386/kvm.c +7 -0
index 95e5d02..5443765 100644
@@ -293,6 +293,7 @@ void kvm_inject_x86_mce(CPUState *cenv, int bank, uint64_t status,
 }
 
 static int _kvm_arch_init_vcpu(CPUState *env);
+int kvmclock_enabled = 1;
 
 int kvm_arch_init_vcpu(CPUState *env)
 {
@@ -350,6 +351,12 @@ int kvm_arch_init_vcpu(CPUState *env)
     memset(c, 0, sizeof(*c));
     c->function = KVM_CPUID_FEATURES;
     c->eax = env->cpuid_kvm_features & get_para_features(env);
+
+    if (!(c->eax & (1 << KVM_FEATURE_CLOCKSOURCE))) {
+        /* In theory cpuid is per-cpu, and this is a global variable,
+         * but we don't expect kvmclock enabled in some cpus only */
+        kvmclock_enabled = 0;
+    }
 #endif
 
     cpu_x86_cpuid(env, 0, 0, &limit, &unused, &unused, &unused);
Signature · 1 line
1.7.2.3
#2 lore raw

Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

Quoted 7 lines · Glauber Costa

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

Quoted 6 lines

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

What is the problem of registering the section? Restoring the value if the host does not support it returns an error?

Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?

Also, please generate against uq/master.

#3 lore raw

Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

Quoted 15 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

Quoted 6 lines

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

Quoted 7 lines

What is the problem of registering the section? Restoring the value if the host does not support it returns an error?

Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?

Also, please generate against uq/master.

Sorry, this is not upstream yet, so ignore the uq/master part.

#4 lore raw

Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

In reply to Marcelo Tosatti
Quoted 12 lines · Marcelo Tosatti
Quoted 11 lines · Glauber Costa

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

Quoted 8 lines
Quoted 2 lines

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

What is the problem of registering the section? Restoring the value if the host does not support it returns an error?

Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?

We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.

#5 lore raw

Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

Quoted 19 lines · Glauber Costa
Quoted 18 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

Quoted 6 lines

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

What is the problem of registering the section? Restoring the value if the host does not support it returns an error?

Quoted 8 lines
Quoted 2 lines

Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?

We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.

OK. Can't you register conditionally on kvmclock cpuid bit at the end of kvm_arch_init_vcpu, in target-i386/kvm.c?

#6 lore raw

Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

Quoted 23 lines · Marcelo Tosatti
Quoted 22 lines · Glauber Costa
Quoted 20 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa

Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.

It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.

Quoted 6 lines

In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.

What is the problem of registering the section? Restoring the value if the host does not support it returns an error?

Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?

Quoted 8 lines
Quoted 5 lines

We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.

OK. Can't you register conditionally on kvmclock cpuid bit at the end of kvm_arch_init_vcpu, in target-i386/kvm.c?

Haven't looked at it, but will today. Actually, tsc has (obviously) the same problem and I plan to respin the patch today including a fix for it as well.

Thanks!

#7 lore raw

PATCH v2 2/2 make kvmclock value idempotent for stopped machine

In reply to Glauber Costa

Although we never made such commitment clear (well, to the best of my knowledge), some people expect that two savevm issued in sequence in a stopped machine will yield the same results. This is not a crazy requirement, since we don't expect a stopped machine to be updating its state, for any device.

With kvmclock, this is not the case, since the .pre_save hook will issue an ioctl to the host to acquire a timestamp, which is always changing.

This patch moves the value acquisition to vm state change handlers, conditional on not being run. This could mean mean our get clock ioctl is issued more times, but this should be fine since vm_stop is not a hot path.

When we do migrate, we'll transfer that value along.

---
 qemu-kvm-x86.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
qemu-kvm-x86.c +3 -3
index 14a52ce..0e357ac 100644
@@ -500,11 +500,11 @@ static int kvm_enable_tpr_access_reporting(CPUState *env)
 #ifdef KVM_CAP_ADJUST_CLOCK
 static struct kvm_clock_data kvmclock_data;
 
-static void kvmclock_pre_save(void *opaque)
+static void kvmclock_update_clock(void *opaque, int running, int reason)
 {
     struct kvm_clock_data *cl = opaque;
 
-    if (!kvmclock_enabled)
+    if ((!kvmclock_enabled) || running)
         return;
 
     kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, cl);
@@ -522,7 +522,6 @@ static const VMStateDescription vmstate_kvmclock= {
     .version_id = 1,
     .minimum_version_id = 1,
     .minimum_version_id_old = 1,
-    .pre_save = kvmclock_pre_save,
     .post_load = kvmclock_post_load,
     .fields      = (VMStateField []) {
         VMSTATE_U64(clock, struct kvm_clock_data),
@@ -537,6 +536,7 @@ void kvmclock_register_savevm(void)
 #ifdef KVM_CAP_ADJUST_CLOCK
     if (kvmclock_enabled && kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
         vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
+        qemu_add_vm_change_state_handler(kvmclock_update_clock, &kvmclock_data);
     }
 #endif
 }
Signature · 1 line
1.7.2.3
maintainer
#8 lore raw
In reply to Glauber Costa
Quoted 19 lines · Glauber Costa

Some users told me that savevm path is behaving oddly wrt kvmclock. The first oddness is that a guarantee we never made (AFAIK) is being broken:
two consecutive "savevm" operations, with the machine stopped in between produces different results, due to the call to KVM_GET_CLOCK ioctl. I believe the assumption that if the vm does not run, its saveable state won't change is fairly reasonable. Maybe we should formally guarantee that?

Also, this patch deals with the fact that this happens even if kvmclock is disabled in cpuid: its savevm section is registered nevertheless. Here, I try to register it only if it's enabled at machine start.

v2: improvements suggested by Paolo, and patch reordering.

Glauber Costa (2):
Do not register kvmclock savevm section if kvmclock is disabled. make kvmclock value idempotent for stopped machine

Quoted 5 lines
  cpus.c            |    3 +++
  qemu-kvm-x86.c    |   23 +++++++++++++++--------
  qemu-kvm.h        |    3 +++
  target-i386/kvm.c |    7 +++++++
  4 files changed, 28 insertions(+), 8 deletions(-)

ACK

Paolo