Skip to contentSkip to conversation

KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

BUG REPORT Chenzhendong (alex) 3 participants 3 messages Nov 1 – Nov 3

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

While testing storage snapshots that suspend and resume a VM on the same host (vm_stop -> snapshot -> vm_start), we observed that the guest CLOCK_MONOTONIC jumps *backward* by the freeze duration after every resume.

The problem
-----------

On a same-host pause/resume cycle, QEMU saves the kvmclock value at the freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:

kvmclock_offset = data.clock - now_raw_ns

with now_raw_ns sampled at T2. Since

data.clock = now_raw_ns(T1) + old_kvmclock_offset

this gives

new_offset = old_kvmclock_offset - freeze_duration

i.e. the offset is pulled back by the freeze duration.

The guest computes CLOCK_MONOTONIC from the pvclock page as:

CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

where system_time = kernel_ns + kvmclock_offset is filled by the host at each vCPU entry. After resume the host re-samples tsc_timestamp, so the (rdtsc - tsc_timestamp) term starts growing normally again and never goes backward. The only term that can shrink is system_time: because kvmclock_offset was pulled back by freeze_duration, the new system_time is freeze_duration smaller than the value the guest last observed before the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore reads a value freeze_duration smaller than before the freeze -- a backward jump, violating the invariant that CLOCK_MONOTONIC never goes backward.

The fix
-------

Only allow kvmclock_offset to move forward (or stay the same). If the newly computed offset would be smaller than the current one, keep the current value, so the guest clock keeps running forward and the freeze duration is simply skipped from the guest's perspective -- which is the expected behavior for a same-host pause/resume. This is a pure monotonic protection with no time threshold; the normal forward path (guest boot, KVM_CLOCK_REALTIME, cross-host migration toward a later destination host clock) is unaffected.

    old_off = ka->kvmclock_offset;
    new_off = data.clock - now_raw_ns;
    if (new_off >= old_off)
            ka->kvmclock_offset = new_off;

Test
----

Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC) twice in a row, flagging any backward step. Over 50 snapshot suspend/resume cycles the backward-step counter stays at 0 with this patch, while the unpatched kernel reports a backward jump on every cycle (measured freeze duration 27-36 ms).

To the best of our knowledge this scenario is not handled upstream. Any feedback on whether monotonic protection is the right approach (vs. e.g. teaching QEMU to feed a post-resume clock) would be much appreciated.

---
 arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +0 -0
index 7a0d0e8e5c1f..000000000000 100644
@@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 	struct kvm_arch *ka = &kvm->arch;
 	struct kvm_clock_data data;
 	u64 now_raw_ns;
+	s64 old_off, new_off;
 	...
 	if (ka->use_master_clock)
 		now_raw_ns = ka->master_kernel_ns;
 	else
 		now_raw_ns = get_kvmclock_base_ns();
-	ka->kvmclock_offset = data.clock - now_raw_ns;
+
+	/*
+	 * Monotonic protection: data.clock is the kvmclock value userspace
+	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
+	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
+	 * offset back to the freeze instant, which makes pvti.system_time =
+	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
+	 * guest vDSO CLOCK_MONOTONIC goes backward.
+	 *
+	 * Only allow kvmclock_offset to move forward (or stay). If the new
+	 * offset would shrink, keep the old one so the guest clock keeps
+	 * running forward (the freeze duration is simply skipped from the
+	 * guest's perspective, which is the desired behavior for a same-host
+	 * pause/resume). Pure monotonic protection, no time threshold.
+	 */
+	old_off = ka->kvmclock_offset;
+	new_off = data.clock - now_raw_ns;
+	if (new_off >= old_off)
+		ka->kvmclock_offset = new_off;
 	kvm_end_pvclock_update(kvm);
#1 lore raw

+ David.

FYI there were some prior discussions related to this topic.

David has a patch at KVM to enhance the KVM API to resolve the issue. It is still not in mainline.

[PATCH v7 31/36] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
https://lore.kernel.org/all/[email protected]

I used to try to enhance from QEMU side to avoid clock drift while accounting for downtime.

[PATCH 1/1] target/i386/kvm: account blackout downtime for kvm-clock and guest TSC
https://lore.kernel.org/qemu-devel/[email protected]

Here is my RCA about forward drift and backward drift from another perspective.

https://lore.kernel.org/all/[email protected]

Quoted 24 lines · Chenzhendong (alex)

Hi,

While testing storage snapshots that suspend and resume a VM on the same host (vm_stop -> snapshot -> vm_start), we observed that the guest CLOCK_MONOTONIC jumps *backward* by the freeze duration after every resume.

The problem
-----------

On a same-host pause/resume cycle, QEMU saves the kvmclock value at the freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:

kvmclock_offset = data.clock - now_raw_ns

with now_raw_ns sampled at T2. Since

data.clock = now_raw_ns(T1) + old_kvmclock_offset

this gives

Quoted 7 lines

new_offset = old_kvmclock_offset - freeze_duration

i.e. the offset is pulled back by the freeze duration.

The guest computes CLOCK_MONOTONIC from the pvclock page as:

CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

CLOCK_MONOTONIC and kvm-clock are different topics.

I would suggest here use kvm-clock or PV clock.

Quoted 12 lines

where system_time = kernel_ns + kvmclock_offset is filled by the host at each vCPU entry. After resume the host re-samples tsc_timestamp, so the (rdtsc - tsc_timestamp) term starts growing normally again and never goes backward. The only term that can shrink is system_time: because kvmclock_offset was pulled back by freeze_duration, the new system_time is freeze_duration smaller than the value the guest last observed before the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore reads a value freeze_duration smaller than before the freeze -- a backward jump, violating the invariant that CLOCK_MONOTONIC never goes backward.

Quoted 4 lines

The fix
-------

Only allow kvmclock_offset to move forward (or stay the same). If the

Ideally, neither forward nor backward drift is appreciated.

Thank you very much!

Dongli Zhang

Quoted 68 lines

newly computed offset would be smaller than the current one, keep the current value, so the guest clock keeps running forward and the freeze duration is simply skipped from the guest's perspective -- which is the expected behavior for a same-host pause/resume. This is a pure monotonic protection with no time threshold; the normal forward path (guest boot, KVM_CLOCK_REALTIME, cross-host migration toward a later destination host clock) is unaffected.

    old_off = ka->kvmclock_offset;
    new_off = data.clock - now_raw_ns;
    if (new_off >= old_off)
            ka->kvmclock_offset = new_off;

Test
----

Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC) twice in a row, flagging any backward step. Over 50 snapshot suspend/resume cycles the backward-step counter stays at 0 with this patch, while the unpatched kernel reports a backward jump on every cycle (measured freeze duration 27-36 ms).

To the best of our knowledge this scenario is not handled upstream. Any feedback on whether monotonic protection is the right approach (vs. e.g. teaching QEMU to feed a post-resume clock) would be much appreciated.

---
 arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +0 -0
index 7a0d0e8e5c1f..000000000000 100644

@@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)

arch/x86/kvm/x86.c +20 -1
 	struct kvm_arch *ka = &kvm->arch;
 	struct kvm_clock_data data;
 	u64 now_raw_ns;
+	s64 old_off, new_off;
...
 	if (ka->use_master_clock)
 		now_raw_ns = ka->master_kernel_ns;
 	else
 		now_raw_ns = get_kvmclock_base_ns();
-	ka->kvmclock_offset = data.clock - now_raw_ns;
+
+	/*
+	 * Monotonic protection: data.clock is the kvmclock value userspace
+	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
+	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
+	 * offset back to the freeze instant, which makes pvti.system_time =
+	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
+	 * guest vDSO CLOCK_MONOTONIC goes backward.
+	 *
+	 * Only allow kvmclock_offset to move forward (or stay). If the new
+	 * offset would shrink, keep the old one so the guest clock keeps
+	 * running forward (the freeze duration is simply skipped from the
+	 * guest's perspective, which is the desired behavior for a same-host
+	 * pause/resume). Pure monotonic protection, no time threshold.
+	 */
+	old_off = ka->kvmclock_offset;
+	new_off = data.clock - now_raw_ns;
+	if (new_off >= old_off)
+		ka->kvmclock_offset = new_off;
 	kvm_end_pvclock_update(kvm);
#2 lore raw
Quoted 1 line · Dongli Zhang

+ David.

Thanks.

Quoted 15 lines

FYI there were some prior discussions related to this topic.

David has a patch at KVM to enhance the KVM API to resolve the issue. It is still not in mainline.

[PATCH v7 31/36] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
https://lore.kernel.org/all/[email protected]

I used to try to enhance from QEMU side to avoid clock drift while accounting for downtime.

[PATCH 1/1] target/i386/kvm: account blackout downtime for kvm-clock and guest TSC
https://lore.kernel.org/qemu-devel/[email protected]

Quoted 19 lines

Here is my RCA about forward drift and backward drift from another perspective.

https://lore.kernel.org/all/[email protected]

Quoted 13 lines · Chenzhendong (alex)

Hi,

While testing storage snapshots that suspend and resume a VM on the same host (vm_stop -> snapshot -> vm_start), we observed that the guest CLOCK_MONOTONIC jumps *backward* by the freeze duration after every resume.

The problem
-----------

On a same-host pause/resume cycle, QEMU saves the kvmclock value at the freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

The whole of this in KVM upstream today is completely insane.

Note that those are completely different clock domains. Your T1 and T2 are CLOCK_REALTIME, which doesn't even run at the same *rate* as the host's CLOCK_MONOTONIC_RAW which the kvmclock is theoretically bound to. And if you do it during a leap second, it's doubly hosed because it's UTC not TAI.

Ignoring the leap seconds, it's almost tolerable to use KVM_SET_CLOCK with the KVM_CLOCK_REALTIME flag for live *migration*, but it makes zero sense for live update on the *same* host.

The source of timekeeping is the TSC. The guest's TSC is just an *offset* (and optional scaling) from the host's TSC. So if you're running on the same host, the guest *TSC* can be reconstituted with *exactly* the same relationship to the host TSC as before. Not a single cycle of difference.

The kvmclock is then advertised to the guest as a mathematical function of the guest's TSC. There's not much excuse for even a single nanosecond of difference in that *either*. In fact, even with KVM_[GS]ET_CLOCK_GUEST in my patch series, there's still a ±1ns variance across get/set. Fixing that would require a lot *more* refactoring than I've already done, so probably isn't worth it.

Guests should never be using kvmclock on modern CPUs where the TSC is reliable, *anyway*.

Quoted 6 lines
Quoted 6 lines

kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:

    kvmclock_offset = data.clock - now_raw_ns

with now_raw_ns sampled at T2. Since

Quoted 7 lines
Quoted 7 lines

    data.clock  = now_raw_ns(T1) + old_kvmclock_offset

this gives

    new_offset  = old_kvmclock_offset - freeze_duration

i.e. the offset is pulled back by the freeze duration.

Hm? There's no freeze_duration in kvm_vm_ioctl_set_clock(). It looks like this:

	/*
	 * This pairs with kvm_guest_time_update(): when masterclock

is
* in use, we use master_kernel_ns + kvmclock_offset to set
* unsigned 'system_time' so if we use get_kvmclock_ns()
(which

	 * is slightly ahead) here we risk going negative on unsigned
	 * 'system_time' when 'data.clock' is very small.
	 */
	if (data.flags & KVM_CLOCK_REALTIME) {
		u64 now_real_ns = ktime_get_real_ns();

		/*
		 * Avoid stepping the kvmclock backwards.
		 */
		if (now_real_ns > data.realtime)
			data.clock += now_real_ns - data.realtime;
	}

Does that not already *have* the monotonicity protection that you are trying to add?

Quoted 7 lines
Quoted 3 lines

The guest computes CLOCK_MONOTONIC from the pvclock page as:

    CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

CLOCK_MONOTONIC and kvm-clock are different topics.

I would suggest here use kvm-clock or PV clock.

Right, don't use CLOCK_MONOTONIC in the guest; use CLOCK_MONOTONIC_RAW.

You can do this with KVM selftests though; perform the save/restore from the host and then just run some code in the guest which checks for monotonicity. Didn't my series already add a test which is very similar to that?