Skip to contentSkip to conversation

KVM: x86: snapshotted TSC frequency causing time drifts in vms

DISCUSSION Jayaramappa, Srilakshmi 2 participants 6 messages Oct 31, 2022 – Dec 15, 2022 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

We were recently notified of significant time drift on some of our virtual machines. Upon investigation it was found that the jumps in time were larger than ntp was able to gracefully correct. After further probing we discovered that the affected vms booted with tsc frequency equal to the early tsc frequency of the host and not the calibrated frequency.

There were two variables that cached tsc_khz - cpu_tsc_khz and max_tsc_khz.
Caching max_tsc_khz would cause further scaling of the user_tsc_khz when the vcpu is created after the host tsc calibrabration and kvm is loaded before calibration. But it appears that Sean's commit "KVM: x86: Don't snapshot "max" TSC if host TSC is constant" would fix that issue. [1]

The cached cpu_tsc_khz is used in
1. get_kvmclock_ns() which incorrectly sets the factors hv_clock.tsc_to_system_mul and hv_clock.shift that estimate passage of time.
2. kvm_guest_time_update()

We came across Anton Romanov's patch "KVM: x86: Use current rather than snapshotted TSC frequency if it is constant" [2] that seems to address the cached cpu_tsc_khz case. The patch description says "the race can be hit if and only if userspace is able to create a VM before TSC refinement completes". We think as long as the kvm module is loaded before the host tsc calibration happens the vms can be created anytime and they will have the problem (confirmed this by shutting down an affected vm and relaunching it - it continued to experience time issues). VMs need not be created before tsc refinement.

Even if kvm module loads and vcpu is created before the host tsc refinement and have incorrect time estimation on the vm until the tsc refinement, the patches referenced here would subsequently provide the correct factors to determine time. And any error in time in that small interval can be corrected by ntp if it is running on the guest. If there was no ntp, the error would probably be negligible and would not accumulate.

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged ?

I'd appreciate you taking the time with this query.

Thanks
-Sri

[1] commit id: 741e511b42086a100c05dbe8fd1baeec42e7c584
[2] https://lore.kernel.org/all/[email protected]/

#1 lore raw
Quoted 35 lines · Srilakshmi

Hi,

We were recently notified of significant time drift on some of our virtual machines. Upon investigation it was found that the jumps in time were larger than ntp was able to gracefully correct. After further probing we discovered that the affected vms booted with tsc frequency equal to the early tsc frequency of the host and not the calibrated frequency.

There were two variables that cached tsc_khz - cpu_tsc_khz and max_tsc_khz. Caching max_tsc_khz would cause further scaling of the user_tsc_khz when the vcpu is created after the host tsc calibrabration and kvm is loaded before calibration. But it appears that Sean's commit "KVM: x86: Don't snapshot "max" TSC if host TSC is constant" would fix that issue. [1]

The cached cpu_tsc_khz is used in 1. get_kvmclock_ns() which incorrectly sets the factors hv_clock.tsc_to_system_mul and hv_clock.shift that estimate passage of time. 2. kvm_guest_time_update()

We came across Anton Romanov's patch "KVM: x86: Use current rather than snapshotted TSC frequency if it is constant" [2] that seems to address the cached cpu_tsc_khz case. The patch description says "the race can be hit if and only if userspace is able to create a VM before TSC refinement completes". We think as long as the kvm module is loaded before the host tsc calibration happens the vms can be created anytime and they will have the problem (confirmed this by shutting down an affected vm and relaunching it - it continued to experience time issues). VMs need not be created before tsc refinement.

Even if kvm module loads and vcpu is created before the host tsc refinement and have incorrect time estimation on the vm until the tsc refinement, the patches referenced here would subsequently provide the correct factors to determine time. And any error in time in that small interval can be corrected by ntp if it is running on the guest. If there was no ntp, the error would probably be negligible and would not accumulate.

Quoted 2 lines

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged?

Drat, it slipped through the cracks.

Paolo, can you pick up the below patch? Oobviously assuming you don't spy any problems.

It has a superficial conflict with commit 938c8745bcf2 ("KVM: x86: Introduce "struct kvm_caps" to track misc caps/settings"), but otherwise applies cleanly.

Quoted 1 line

[2] https://lore.kernel.org/all/[email protected]/

#2 lore raw

From: Sean Christopherson <[email protected]>
Sent: Monday, October 31, 2022 5:59 PM
To: Jayaramappa, Srilakshmi
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hunt, Joshua
Subject: Re: KVM: x86: snapshotted TSC frequency causing time drifts in vms

Quoted 35 lines · Srilakshmi

Hi,

We were recently notified of significant time drift on some of our virtual machines. Upon investigation it was found that the jumps in time were larger than ntp was able to gracefully correct. After further probing we discovered that the affected vms booted with tsc frequency equal to the early tsc frequency of the host and not the calibrated frequency.

There were two variables that cached tsc_khz - cpu_tsc_khz and max_tsc_khz. Caching max_tsc_khz would cause further scaling of the user_tsc_khz when the vcpu is created after the host tsc calibrabration and kvm is loaded before calibration. But it appears that Sean's commit "KVM: x86: Don't snapshot "max" TSC if host TSC is constant" would fix that issue. [1]

The cached cpu_tsc_khz is used in 1. get_kvmclock_ns() which incorrectly sets the factors hv_clock.tsc_to_system_mul and hv_clock.shift that estimate passage of time.  2. kvm_guest_time_update()

We came across Anton Romanov's patch "KVM: x86: Use current rather than snapshotted TSC frequency if it is constant" [2] that seems to address the cached cpu_tsc_khz  case. The patch description says "the race can be hit if and only if userspace is able to create a VM before TSC refinement completes". We think as long as the kvm module is loaded before the host tsc calibration happens the vms can be created anytime and they will have the problem (confirmed this by shutting down an affected vm and relaunching it - it continued to experience time issues). VMs need not be created before tsc refinement.

Even if kvm module loads and vcpu is created before the host tsc refinement and have incorrect time estimation on the vm until the tsc refinement, the patches referenced here would subsequently provide the correct factors to determine time. And any error in time in that small interval can be corrected by ntp if it is running on the guest. If there was no ntp, the error would probably be negligible and would not accumulate.

Quoted 2 lines

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged?

Drat, it slipped through the cracks.

Paolo, can you pick up the below patch?  Oobviously assuming you don't spy any problems.

It has a superficial conflict with commit 938c8745bcf2 ("KVM: x86: Introduce "struct kvm_caps" to track misc caps/settings"), but otherwise applies cleanly.

Quoted 1 line

[2] https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!GjvTz_vk!QH6DrxJkEWcYdjwasd9zcBVokREj7lO9qb6tynY5SpQoRRXRxi959dCvoy_sbU9oRcrSbNCxXwA_dw$

Thanks, Sean! Appreciate it.

-Sri

#3 lore raw

From: Jayaramappa, Srilakshmi
Sent: Monday, October 31, 2022 7:00 PM
To: Sean Christopherson
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hunt, Joshua
Subject: Re: KVM: x86: snapshotted TSC frequency causing time drifts in vms

From: Sean Christopherson <[email protected]>
Sent: Monday, October 31, 2022 5:59 PM
To: Jayaramappa, Srilakshmi
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hunt, Joshua
Subject: Re: KVM: x86: snapshotted TSC frequency causing time drifts in vms

Quoted 35 lines · Srilakshmi

Hi,

We were recently notified of significant time drift on some of our virtual machines. Upon investigation it was found that the jumps in time were larger than ntp was able to gracefully correct. After further probing we discovered that the affected vms booted with tsc frequency equal to the early tsc frequency of the host and not the calibrated frequency.

There were two variables that cached tsc_khz - cpu_tsc_khz and max_tsc_khz. Caching max_tsc_khz would cause further scaling of the user_tsc_khz when the vcpu is created after the host tsc calibrabration and kvm is loaded before calibration. But it appears that Sean's commit "KVM: x86: Don't snapshot "max" TSC if host TSC is constant" would fix that issue. [1]

The cached cpu_tsc_khz is used in 1. get_kvmclock_ns() which incorrectly sets the factors hv_clock.tsc_to_system_mul and hv_clock.shift that estimate passage of time.  2. kvm_guest_time_update()

We came across Anton Romanov's patch "KVM: x86: Use current rather than snapshotted TSC frequency if it is constant" [2] that seems to address the cached cpu_tsc_khz  case. The patch description says "the race can be hit if and only if userspace is able to create a VM before TSC refinement completes". We think as long as the kvm module is loaded before the host tsc calibration happens the vms can be created anytime and they will have the problem (confirmed this by shutting down an affected vm and relaunching it - it continued to experience time issues). VMs need not be created before tsc refinement.

Even if kvm module loads and vcpu is created before the host tsc refinement and have incorrect time estimation on the vm until the tsc refinement, the patches referenced here would subsequently provide the correct factors to determine time. And any error in time in that small interval can be corrected by ntp if it is running on the guest. If there was no ntp, the error would probably be negligible and would not accumulate.

Quoted 2 lines

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged?

Drat, it slipped through the cracks.

Paolo, can you pick up the below patch?  Oobviously assuming you don't spy any problems.

It has a superficial conflict with commit 938c8745bcf2 ("KVM: x86: Introduce "struct kvm_caps" to track misc caps/settings"), but otherwise applies cleanly.

Quoted 1 line

[2]  https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!GjvTz_vk!QH6DrxJkEWcYdjwasd9zcBVokREj7lO9qb6tynY5SpQoRRXRxi959dCvoy_sbU9oRcrSbNCxXwA_dw$

Thanks, Sean! Appreciate it.

-Sri

Hi Paolo,

Could I trouble you to take a look at this patch please?

Thanks
-Sri

#4 lore raw
Quoted 9 lines · Srilakshmi
Quoted 2 lines

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged?

Drat, it slipped through the cracks.

Paolo, can you pick up the below patch?  Oobviously assuming you don't spy any problems.

It has a superficial conflict with commit 938c8745bcf2 ("KVM: x86: Introduce

...

Quoted 1 line

Could I trouble you to take a look at this patch please?

It's already in kvm/next

3ebcbd2244f5 ("KVM: x86: Use current rather than snapshotted TSC frequency if it is constant")

but there was a hiccup with the KVM pull request for 6.2[*], which is why it hasn't made it's way to Linus yet.

[*] https://lore.kernel.org/all/[email protected]

#5 lore raw

From: Sean Christopherson <[email protected]>
Sent: Wednesday, December 14, 2022 6:58 PM
To: Jayaramappa, Srilakshmi
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hunt, Joshua
Subject: Re: KVM: x86: snapshotted TSC frequency causing time drifts in vms

Quoted 9 lines · Srilakshmi
Quoted 2 lines

There doesn't seem to be any response on the v6 of Anton's patch. I wanted to ask if there is further changes in progress or if it is all set to be merged?

Drat, it slipped through the cracks.

Paolo, can you pick up the below patch?  Oobviously assuming you don't spy any problems.

It has a superficial conflict with commit 938c8745bcf2 ("KVM: x86: Introduce

...

Quoted 1 line

Could I trouble you to take a look at this patch please?

It's already in kvm/next

  3ebcbd2244f5 ("KVM: x86: Use current rather than snapshotted TSC frequency if it is constant")

but there was a hiccup with the KVM pull request for 6.2[*], which is why it hasn't made it's way to Linus yet.

[*] https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]__;!!GjvTz_vk!VYIJqzFNCr9fQP6gLlryCKVNhGb-OrtosOocQzLpVk0aIEGoGFKL5OD0Zw8rmEN3QhhX9BmLkMpH7A$

Oh that's great, thanks, Sean! Appreciate the help.

-Sri