savevm odness related to kvmclock
v2 cover savevm odness related to kvmclock
Some users told me that savevm path is behaving oddly wrt kvmclock. The first oddness is that a guarantee we never made (AFAIK) is being broken:
two consecutive "savevm" operations, with the machine stopped in between produces different results, due to the call to KVM_GET_CLOCK ioctl. I believe the assumption that if the vm does not run, its saveable state won't change is fairly reasonable. Maybe we should formally guarantee that?
Also, this patch deals with the fact that this happens even if kvmclock is disabled in cpuid: its savevm section is registered nevertheless. Here, I try to register it only if it's enabled at machine start.
v2: improvements suggested by Paolo, and patch reordering.
Glauber Costa (2):
Do not register kvmclock savevm section if kvmclock is disabled. make kvmclock value idempotent for stopped machine
cpus.c | 3 +++ qemu-kvm-x86.c | 23 +++++++++++++++-------- qemu-kvm.h | 3 +++ target-i386/kvm.c | 7 +++++++ 4 files changed, 28 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.2.3
PATCH v2 1/2 Do not register kvmclock savevm section if kvmclock is disabled.
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.
In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
--- cpus.c | 3 +++ qemu-kvm-x86.c | 19 +++++++++++++------ qemu-kvm.h | 3 +++ target-i386/kvm.c | 7 +++++++ 4 files changed, 26 insertions(+), 6 deletions(-)
cpus.c +3 -0
@@ -97,6 +97,9 @@ void cpu_synchronize_all_post_init(void)for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {cpu_synchronize_post_init (cpu); } +if (kvm_enabled ()) { +kvmclock_register_savevm (); + } }int cpu_is_stopped (CPUState *env)
qemu-kvm-x86.c +13 -6
@@ -504,6 +504,9 @@ static void kvmclock_pre_save(void *opaque) {struct kvm_clock_data *cl = opaque; +if (!kvmclock_enabled) +return ; +kvm_vm_ioctl (kvm_state,KVM_GET_CLOCK , cl); }
@@ -528,6 +531,16 @@ static const VMStateDescription vmstate_kvmclock= { };#endif +/* This has to happen after vcpu setup*/ +void kvmclock_register_savevm (void ) +{ +#ifdef KVM_CAP_ADJUST_CLOCK +if (kvmclock_enabled &&kvm_check_extension (kvm_state,KVM_CAP_ADJUST_CLOCK )) { +vmstate_register (NULL ,0 , &vmstate_kvmclock, &kvmclock_data); + } +#endif +} +int kvm_arch_qemu_create_context (void ) {int r;
@@ -545,12 +558,6 @@ int kvm_arch_qemu_create_context(void)return -1 ; } -#ifdef KVM_CAP_ADJUST_CLOCK -if (kvm_check_extension (kvm_state,KVM_CAP_ADJUST_CLOCK )) { -vmstate_register (NULL ,0 , &vmstate_kvmclock, &kvmclock_data); - } -#endif - r =kvm_set_boot_cpu_id (0 );if (r <0 && r != -ENOSYS ) {return r;
qemu-kvm.h +3 -0
@@ -752,6 +752,9 @@ int handle_tpr_access(void *opaque, CPUState *env, uint64_t rip,#define qemu_kvm_cpu_stop (env) do {} while(0)#endif +extern int kvmclock_enabled; +void kvmclock_register_savevm (void ); + #ifdef CONFIG_KVM typedef struct KVMSlot {
target-i386/kvm.c +7 -0
@@ -293,6 +293,7 @@ void kvm_inject_x86_mce(CPUState *cenv, int bank, uint64_t status, }static int _kvm_arch_init_vcpu (CPUState *env); +int kvmclock_enabled =1 ;int kvm_arch_init_vcpu (CPUState *env) {
@@ -350,6 +351,12 @@ int kvm_arch_init_vcpu(CPUState *env)memset (c,0 ,sizeof (*c)); c->function =KVM_CPUID_FEATURES ; c->eax = env->cpuid_kvm_features &get_para_features (env); + +if (!(c->eax & (1 <<KVM_FEATURE_CLOCKSOURCE ))) { +/* In theory cpuid is per-cpu, and this is a global variable, +* but we don't expect kvmclock enabled in some cpus only */ + kvmclock_enabled =0 ; + }#endif cpu_x86_cpuid (env,0 ,0 , &limit, &unused, &unused, &unused);
Signature · 1 line
1.7.2.3
Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.
Quoted 7 lines · Glauber Costa
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.
Quoted 6 lines
In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
What is the problem of registering the section? Restoring the value if the host does not support it returns an error?
Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?
Also, please generate against uq/master.
Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.
Quoted 15 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.Quoted 6 lines
In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
Quoted 7 lines
What is the problem of registering the section? Restoring the value if the host does not support it returns an error?
Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?
Also, please generate against uq/master.
Sorry, this is not upstream yet, so ignore the uq/master part.
Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.
In reply to Marcelo TosattiQuoted 12 lines · Marcelo Tosatti
Quoted 11 lines · Glauber Costa
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
Quoted 8 lines
Quoted 2 lines
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
What is the problem of registering the section? Restoring the value if the host does not support it returns an error?
Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?
We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.
Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.
Quoted 19 lines · Glauber Costa
Quoted 18 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.Quoted 6 lines
In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
What is the problem of registering the section? Restoring the value if the host does not support it returns an error?
Quoted 8 lines
Quoted 2 lines
Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?
We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.
OK. Can't you register conditionally on kvmclock cpuid bit at the end of kvm_arch_init_vcpu, in target-i386/kvm.c?
Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.
Quoted 23 lines · Marcelo Tosatti
Quoted 22 lines · Glauber Costa
Quoted 20 lines · Marcelo Tosatti
Quoted 7 lines · Glauber Costa
Usually nobody usually thinks about that scenario (me included and specially), but kvmclock can be actually disabled in the host.
It happens in two scenarios:
1. host too old.
2. we passed -kvmclock to our -cpu parameter.Quoted 6 lines
In both cases, we should not register kvmclock savevm section. This patch achives that by registering this section only if kvmclock is actually currently enabled in cpuid.
The only caveat is that we have to register the savevm section a little bit later, since we won't know the final kvmclock state before cpuid gets parsed.
What is the problem of registering the section? Restoring the value if the host does not support it returns an error?
Can't you ignore the error if kvmclock is not reported in cpuid, in the restore handler?
Quoted 8 lines
Quoted 5 lines
We can change the restore handler, but not the restore handler of binaries that are already out there. The motivation here is precisely to address migration to hosts without kvmclock, so it's better to have a way to disable, than to count on the fact that the other side will be able to ignore it.
OK. Can't you register conditionally on kvmclock cpuid bit at the end of
kvm_arch_init_vcpu, intarget-i386/kvm.c?
Haven't looked at it, but will today. Actually, tsc has (obviously) the same problem and I plan to respin the patch today including a fix for it as well.
Thanks!
PATCH v2 2/2 make kvmclock value idempotent for stopped machine
In reply to Glauber CostaAlthough we never made such commitment clear (well, to the best of my knowledge), some people expect that two savevm issued in sequence in a stopped machine will yield the same results. This is not a crazy requirement, since we don't expect a stopped machine to be updating its state, for any device.
With kvmclock, this is not the case, since the .pre_save hook will issue an ioctl to the host to acquire a timestamp, which is always changing.
This patch moves the value acquisition to vm state change handlers, conditional on not being run. This could mean mean our get clock ioctl is issued more times, but this should be fine since vm_stop is not a hot path.
When we do migrate, we'll transfer that value along.
--- qemu-kvm-x86.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
qemu-kvm-x86.c +3 -3
@@ -500,11 +500,11 @@ static int kvm_enable_tpr_access_reporting(CPUState *env)#ifdef KVM_CAP_ADJUST_CLOCK static struct kvm_clock_data kvmclock_data; -static void kvmclock_pre_save (void *opaque) +static void kvmclock_update_clock (void *opaque,int running,int reason) {struct kvm_clock_data *cl = opaque; -if (!kvmclock_enabled) +if ((!kvmclock_enabled) || running)return ;kvm_vm_ioctl (kvm_state,KVM_GET_CLOCK , cl);
@@ -522,7 +522,6 @@ static const VMStateDescription vmstate_kvmclock= { .version_id = 1, .minimum_version_id = 1, .minimum_version_id_old = 1, - .pre_save = kvmclock_pre_save, .post_load = kvmclock_post_load, .fields = (VMStateField []) { VMSTATE_U64(clock, struct kvm_clock_data),
@@ -537,6 +536,7 @@ void kvmclock_register_savevm(void)#ifdef KVM_CAP_ADJUST_CLOCK if (kvmclock_enabled &&kvm_check_extension (kvm_state,KVM_CAP_ADJUST_CLOCK )) {vmstate_register (NULL ,0 , &vmstate_kvmclock, &kvmclock_data); +qemu_add_vm_change_state_handler (kvmclock_update_clock, &kvmclock_data); }#endif }
Signature · 1 line
1.7.2.3
Paolo Bonzini
maintainerQuoted 19 lines · Glauber Costa
Some users told me that savevm path is behaving oddly wrt kvmclock. The first oddness is that a guarantee we never made (AFAIK) is being broken:
two consecutive "savevm" operations, with the machine stopped in between produces different results, due to the call toKVM_GET_CLOCKioctl. I believe the assumption that if the vm does not run, its saveable state won't change is fairly reasonable. Maybe we should formally guarantee that?Also, this patch deals with the fact that this happens even if kvmclock is disabled in cpuid: its savevm section is registered nevertheless. Here, I try to register it only if it's enabled at machine start.
v2: improvements suggested by Paolo, and patch reordering.
Glauber Costa (2):
Do not register kvmclock savevm section if kvmclock is disabled. make kvmclock value idempotent for stopped machine
Quoted 5 lines
cpus.c | 3 +++ qemu-kvm-x86.c | 23 +++++++++++++++-------- qemu-kvm.h | 3 +++ target-i386/kvm.c | 7 +++++++ 4 files changed, 28 insertions(+), 8 deletions(-)
ACK
Paolo