Skip to contentSkip to conversation

kvmclock: clarify usage of cpu_clean_all_dirty

PATCH In review Marcelo Tosatti 2 participants 10 messages Sep 16, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvmclock: clarify usage of cpu_clean_all_dirty

Explain the cpu_clean_all_dirty call.

---
 hw/i386/kvm/clock.c |    4 ++++
 1 file changed, 4 insertions(+)
hw/i386/kvm/clock.c +4 -0
Index: qemu/hw/i386/kvm/clock.c
===================================================================
@@ -127,6 +127,10 @@
         }
 
         cpu_synchronize_all_states();
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.
+ 	 */
         cpu_clean_all_dirty();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
#1 lore raw

Il 16/09/2014 17:14, Marcelo Tosatti ha scritto:

hw/i386/kvm/clock.c line 132

Quoted 3 lines
hw/i386/kvm/clock.c +3 -0
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.

Which other callback could affect the in-kernel state, and should that call cpu_clean_all_dirty instead?

Paolo

Quoted 2 lines
hw/i386/kvm/clock.c +1 -0
+ 	 */
         cpu_clean_all_dirty();
#2 lore raw
Quoted 6 lines · Paolo Bonzini

Il 16/09/2014 17:14, Marcelo Tosatti ha scritto:

hw/i386/kvm/clock.c line 132

Quoted 3 lines
hw/i386/kvm/clock.c +3 -0
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.

Which other callback could affect the in-kernel state,

Marcin mentioned that APIC state was the culprit.

Perhaps

     bdrv_drain_all(); 
     ret = bdrv_flush_all();

Can change the interrupt state ?

Then that should read "once VM stop has finished".

Quoted 6 lines

call cpu_clean_all_dirty instead?

Paolo

Quoted 2 lines
hw/i386/kvm/clock.c +1 -0
+ 	 */
         cpu_clean_all_dirty();
#3 lore raw

Il 16/09/2014 17:55, Marcelo Tosatti ha scritto:

Quoted 8 lines · Paolo Bonzini
Quoted 6 lines

Il 16/09/2014 17:14, Marcelo Tosatti ha scritto:

hw/i386/kvm/clock.c line 132

Quoted 3 lines
hw/i386/kvm/clock.c +3 -0
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.

Which other callback could affect the in-kernel state,

Quoted 8 lines

Marcin mentioned that APIC state was the culprit.

Perhaps

     bdrv_drain_all(); 
     ret = bdrv_flush_all();

Can change the interrupt state ?

Ah, I thought Marcin was checking on the destination, not the source.

Quoted 1 line

Then that should read "once VM stop has finished".

But I still do not understand.

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

But if the synchronize_all_states+clean_all_dirty pair runs on the source, why is the cpu_synchronize_all_states() call in qemu_savevm_state_complete() not enough? It runs even later than kvmclock_vm_state_change.

I don't understand even the original patch without cpu_clean_all_dirty()...

Paolo

#4 lore raw
Quoted 18 lines · Paolo Bonzini

Il 16/09/2014 17:55, Marcelo Tosatti ha scritto:

Quoted 8 lines · Paolo Bonzini
Quoted 6 lines

Il 16/09/2014 17:14, Marcelo Tosatti ha scritto:

hw/i386/kvm/clock.c line 132

Quoted 3 lines
hw/i386/kvm/clock.c +3 -0
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.

Which other callback could affect the in-kernel state,

Quoted 8 lines

Marcin mentioned that APIC state was the culprit.

Perhaps

     bdrv_drain_all(); 
     ret = bdrv_flush_all();

Can change the interrupt state ?

Quoted 8 lines

Ah, I thought Marcin was checking on the destination, not the source.

Quoted 1 line

Then that should read "once VM stop has finished".

But I still do not understand.

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

Its there to make sure the pair

env->tsc, s->clock = data.clock

are relative to point close in time.

Quoted 4 lines

But if the synchronize_all_states+clean_all_dirty pair runs on the source, why is the cpu_synchronize_all_states() call in qemu_savevm_state_complete() not enough? It runs even later than kvmclock_vm_state_change.

Because of the "pair of time values" explanation above.

Quoted 3 lines

I don't understand even the original patch without cpu_clean_all_dirty()...

Paolo

#5 lore raw

Il 16/09/2014 18:07, Marcelo Tosatti ha scritto:

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

Its there to make sure the pair

env->tsc, s->clock = data.clock

are relative to point close in time.

Ok. But why are they not close in time?

Could we have the opposite situation where env->tsc is loaded a long time _after_ s->clock, and something breaks?

Also, there is no reason to do kvmclock_current_nsec() during normal execution of the VM. Is the s->clock sent by the source ever good across migration, and could the source send kvmclock_current_nsec() instead of whatever KVM_GET_CLOCK returns?

I don't understand this code very well, but it seems to me that the migration handling and VM state change handler are mixed up...

Paolo

Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

But if the synchronize_all_states+clean_all_dirty pair runs on the source, why is the cpu_synchronize_all_states() call in qemu_savevm_state_complete() not enough? It runs even later than kvmclock_vm_state_change.

Because of the "pair of time values" explanation above.

#6 lore raw
Quoted 10 lines · Paolo Bonzini

Il 16/09/2014 18:07, Marcelo Tosatti ha scritto:

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

Its there to make sure the pair

env->tsc, s->clock = data.clock

are relative to point close in time.

Ok. But why are they not close in time?

Scenario 1

A. s->clock = get_clock()
B. env->tsc = rdtsc()

Scenario 2

A. s->clock = get_clock()
C. VM callbacks, bdrv_flush, ...
B. env->tsc = rdtsc()

They are not "close in time" because of C.

Quoted 2 lines

Could we have the opposite situation where env->tsc is loaded a long time _after_ s->clock, and something breaks?

This particular read avoids an overflow. See

+ assert(time.tsc_timestamp <= migration_tsc);

About your question, perhaps, would have to make up an env->tsc, s->clock pair which breaks the code or a guest application.

Quoted 4 lines

Also, there is no reason to do kvmclock_current_nsec() during normal execution of the VM. Is the s->clock sent by the source ever good across migration, and could the source send kvmclock_current_nsec() instead of whatever KVM_GET_CLOCK returns?

Yes it could. What difference does it make?

Quoted 2 lines

I don't understand this code very well, but it seems to me that the migration handling and VM state change handler are mixed up...

I don't see what the problem is. I am sure you can understand the code.

#7 lore raw
Quoted 42 lines · Marcelo Tosatti
Quoted 10 lines · Paolo Bonzini

Il 16/09/2014 18:07, Marcelo Tosatti ha scritto:

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

Its there to make sure the pair

env->tsc, s->clock = data.clock

are relative to point close in time.

Ok. But why are they not close in time?

Scenario 1

A. s->clock = get_clock()
B. env->tsc = rdtsc()

Scenario 2

A. s->clock = get_clock()
C. VM callbacks, bdrv_flush, ...
B. env->tsc = rdtsc()

They are not "close in time" because of C.

Quoted 2 lines

Could we have the opposite situation where env->tsc is loaded a long time _after_ s->clock, and something breaks?

This particular read avoids an overflow. See

+ assert(time.tsc_timestamp <= migration_tsc);

About your question, perhaps, would have to make up an env->tsc, s->clock pair which breaks the code or a guest application.

Quoted 4 lines

Also, there is no reason to do kvmclock_current_nsec() during normal execution of the VM. Is the s->clock sent by the source ever good across migration, and could the source send kvmclock_current_nsec() instead of whatever KVM_GET_CLOCK returns?

Quoted 6 lines

Yes it could. What difference does it make?

Quoted 2 lines

I don't understand this code very well, but it seems to me that the migration handling and VM state change handler are mixed up...

I don't see what the problem is. I am sure you can understand the code.

If you have a suggestion to improve the code, i am all ears. But IMHO, its not terrible (migration and VM state change handling are mixed in other drivers as well).

#8 lore raw
In reply to Paolo Bonzini
Quoted 11 lines · Paolo Bonzini

Il 16/09/2014 18:07, Marcelo Tosatti ha scritto:

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

The cpu_synchronize_all_states() call in kvmclock_vm_state_change() is needed to make env->tsc up to date with the value on the source, right?

Its there to make sure the pair

env->tsc, s->clock = data.clock

are relative to point close in time.

Ok. But why are they not close in time?

Quoted 7 lines

Could we have the opposite situation where env->tsc is loaded a long time _after_ s->clock, and something breaks?

Also, there is no reason to do kvmclock_current_nsec() during normal execution of the VM. Is the s->clock sent by the source ever good across migration, and could the source send kvmclock_current_nsec() instead of whatever KVM_GET_CLOCK returns?

guest clock read = pvclock.system_timestamp + (rdtsc() - pvclock.tsc)

Host kernel updates pvclock.system_timestamp in certain situations, such as guest initialization. With master clock scheme, pvclock.system_timestamp is only updated on guest initialization.

In case TSC runs faster than the host system clock, you cannot do the following on destination:

pvclock.system_timestamp = ioctl(KVM_GET_CLOCK)
povclock.tsc = rdtsc

guest clock read = pvclock.system_timestampOLD + (rdtsc() - pvclock.tsc)

Because the effective clock was not pvclock.system_timestamp but the TSC, which is running at a higher frequency. If you do that, the time goes backward.

Q: could the source send kvmclock_current_nsec()
instead of whatever KVM_GET_CLOCK returns?

Well no because there are other users of KVM_GET_CLOCK such as Hyper-V clock.

Quoted 2 lines

I don't understand this code very well, but it seems to me that the migration handling and VM state change handler are mixed up...

Again, suggestions are welcome.

Quoted 9 lines

Paolo

Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

But if the synchronize_all_states+clean_all_dirty pair runs on the source, why is the cpu_synchronize_all_states() call in qemu_savevm_state_complete() not enough? It runs even later than kvmclock_vm_state_change.

Because of the "pair of time values" explanation above.

#9 lore raw
In reply to Paolo Bonzini
Quoted 6 lines · Paolo Bonzini

Il 16/09/2014 17:14, Marcelo Tosatti ha scritto:

hw/i386/kvm/clock.c line 132

Quoted 3 lines
hw/i386/kvm/clock.c +3 -0
+	/*
+	 * Make sure that CPU state is synchronized from KVM
+ 	 * once every VM state change callback has finished.

Which other callback could affect the in-kernel state,

Do you want me to find out exactly which callback is changing the in-kernel interrupt state?
To be honest: i don't know.

Quoted 1 line

and should that call cpu_clean_all_dirty instead?

Does not make sense: any callback which modifies any in-kernel register state should clean dirty state because its possibly set.