KVM: x86: Cleaning up the KVM clock mess, part 1
Sean Christopherson
maintainerv10 cover KVM: x86: Cleaning up the KVM clock mess, part 1
Part 1 of David's series to clean up the host-side mess of kvmclock. The cut point in this version is still quite arbitrary, but what's left for "part 2" fits nicely into 2-3 buckets:
1: Masterclock overhaul
2. New uAPI and tests
v10:
- Drop the stale reference to using div_u64() for the now-removed CPUID leaf. [Sashiko]
- Don't bother with a enabling the pvclock_gtod notifier on 32-bit, and go straight to ktime_mono_to_any(). [Sashiko]
v9:
- https://lore.kernel.org/all/[email protected]
- Pull in "Compute kvmclock base without pvclock_gtod_data" before "Avoid NTP frequency skew for KVM clock on 32-bit host" to avoid having to provide what is effectively an open-coded version of ktime_mono_to_any() to avoid data races. [Sashiko].
- Pull in use of kernel timekeeping snapshots, split in three patches purely to aid review and bisection.
- Check and clear KVM_REQ_MASTERCLOCK_UPDATE in kvm_arch_vcpu_postcreate(). [me, Sashiko]
- Add the missing return value in "Move "no master clock" fallback from...". [Sashiko]
v8:
- Obviously cut partway through the full series.
- Update commit references for 633d7652f80f ("KVM: x86/xen: Do not corrupt KVM clock in kvm_xen_shared_info_init()").
- Chunk the "Restructure ..." patches into individual logical changes ("Restructure kvm_guest_time_update() for TSC upscaling" in particular was quite gnarly).
- Ensure the TSC+freq pair during guest time updates happens on a single CPU. [Sashiko]
- Keep __get_kvmclock() as a masterclock-specific helper to avoid deep indentation and gotos.
- Do waaaaay too much spelunking to try piece together the history of some of the many warts.
- Ensure preemption is disabled in get_kvmclock() when getting the TSC freq.
v7: https://lore.kernel.org/all/[email protected]
David Woodhouse (14):
KVM: x86: Improve accuracy of KVM clock when TSC scaling is in force
KVM: x86: Explicitly disable TSC scaling without CONSTANT_TSC
KVM: x86: Activate master clock immediately on vCPU creation
KVM: x86: Compute kvmclock base without pvclock_gtod_data
KVM: x86: Avoid NTP frequency skew for KVM clock on 32-bit host
KVM: x86: Wrap all of __get_kvmclock_master_clock() with
CONFIG_X86_64=y
KVM: x86: Fall back to non-master-clock if clockread fails in
get_kvmclock()
KVM: x86: Fix KVM clock precision in get_kvmclock() with TSC scaling
KVM: x86: Use get_kvmclock() in kvm_get_wall_clock_epoch()
KVM: x86: Fix compute_guest_tsc() to handle negative time deltas
KVM: x86: Upscale TSC to "now", not master clock when updating PV
clocks
KVM: x86: Simplify and comment kvm_get_time_scale()
KVM: x86: Remove implicit rdtsc() from kvm_compute_l1_tsc_offset()
KVM: x86: Use kernel timekeeping snapshots for getting kvmclock time
since boot
Sean Christopherson (7):
KVM: x86: Update "last guest TSC" snapshot prior to enabling
IRQs/preemption
KVM: x86: Drop unnecessary CPU pinning when computing/getting kvmclock
KVM: x86: Move "no master clock" fallback from __get_kvmclock() to
get_kvmclock()
KVM: x86: Disable preemption, not IRQs, when getting TSC+freq pair
KVM: x86: Make master clock logic in guest PV clock updates 64-bit
only
KVM: x86: Use kernel timekeeping snapshot for monotonic clock
KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
arch/x86/include/asm/kvm_host.h | 6 +- arch/x86/kvm/cpuid.c | 1 + arch/x86/kvm/msrs.c | 3 +- arch/x86/kvm/svm/svm.c | 3 +- arch/x86/kvm/vmx/vmx.c | 10 + arch/x86/kvm/x86.c | 480 +++++++++++++++----------------- arch/x86/kvm/x86.h | 3 +- 7 files changed, 240 insertions(+), 266 deletions(-)
base-commit: 76671054f9a1ff6abb976583cd8da37650acdc97
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 1/21 KVM: x86: Update "last guest TSC" snapshot prior to enabling IRQs/preemption
When refreshing the last observed guest TSC during a guest time update, write the snapshot before enabling IRQs, i.e. before enabling preemption. If the task is migrated between updating the local tsc_timestamp, e.g. to account for catch-up mode, and setting last_guest_tsc, kvm_arch_vcpu_load() would set the vCPU's TSC offset using the old last_guest_tsc.
In practice, the bug is largely benign as it's not even strictly necessary for KVM to refresh last_guest_tsc when updating guest time, as KVM's goal is purely to prevent the guest from observing time jump backwards, i.e. super duper strictly speaking, KVM only *needs* to update last_guest_tsc in the VM-Exit path.
In fact, the update kvm_guest_time_update() in wasn't even added to play nice with kvm_arch_vcpu_load(), it was added by commit 28e4639adf0c ("KVM:
x86: Fix kvmclock bug") to fix code that no longer exists. As of commit 28e4639adf0c, kvm_guest_time_update() also consumed last_guest_tsc, to try and prevent guest time from jumping backwards. That code was eventually removed by commit f25e656d31ad ("KVM: x86: fix tsc catchup issue with tsc scaling"), but the last_guest_tsc update hung around.
Keep the update even though it's technically ok to drop the update, e.g. so that the tsc_catchup updates aren't lost, and so that the guest won't see a PV clock timestamp that appears to be in the future.
--- arch/x86/kvm/x86.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +6 -1
@@ -1820,6 +1820,12 @@ int kvm_guest_time_update(struct kvm_vcpu *v) } } +/* +* Refresh L1's last "observed" TSC to match the PV clock's timestamp, +* e.g. so that the guest can't see a TSC that's behind the reference. +*/ + vcpu->last_guest_tsc = tsc_timestamp; +local_irq_restore (flags);/* With all the info we got, fill in the values */
@@ -1841,7 +1847,6 @@ int kvm_guest_time_update(struct kvm_vcpu *v) hv_clock.tsc_to_system_mul = vcpu->pvclock_tsc_mul; hv_clock.tsc_timestamp = tsc_timestamp; hv_clock.system_time = kernel_ns + v->kvm->arch.kvmclock_offset; - vcpu->last_guest_tsc = tsc_timestamp;/* If the host uses TSC clocksource, then it is stable */ hv_clock.flags =0 ;
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 2/21 KVM: x86: Improve accuracy of KVM clock when TSC scaling is in force
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
The kvm_guest_time_update() function scales the host TSC frequency to the guest's using kvm_scale_tsc() and the v->arch.l1_tsc_scaling_ratio scaling ratio previously calculated for that vCPU. Then calculates the scaling factors for the KVM clock itself based on that guest TSC frequency.
However, it uses kHz as the unit when scaling, and then multiplies by 1000 only at the end.
With a host TSC frequency of 3000MHz and a guest set to 2500MHz, the result of kvm_scale_tsc() will actually come out at 2,499,999kHz. So the KVM clock advertised to the guest is based on a frequency of 2,499,999,000 Hz.
By using Hz as the unit from the beginning, the KVM clock would be based on a more accurate frequency of 2,499,999,999 Hz in this example.
Use u64 for the hw_tsc_hz field since an unsigned int would overflow for TSC frequencies above 4GHz.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/cpuid.c | 1 + arch/x86/kvm/x86.c | 18 ++++++++++-------- 3 files changed, 12 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -864,7 +864,7 @@ struct kvm_vcpu_arch {gpa_t time;s8 pvclock_tsc_shift;u32 pvclock_tsc_mul; -unsigned int hw_tsc_khz; +u64 hw_tsc_hz;struct gfn_to_pfn_cache pv_time;/* set guest stopped flag in pvclock flags field */ bool pvclock_set_guest_stopped_request;
arch/x86/kvm/cpuid.c +1 -0
@@ -16,6 +16,7 @@#include <linux/vmalloc.h> #include <linux/uaccess.h> #include <linux/sched/stat.h> +#include <linux/units.h> #include <asm/processor.h> #include <asm/user.h>
arch/x86/kvm/x86.c +10 -8
@@ -61,6 +61,7 @@#include <linux/mem_encrypt.h> #include <linux/suspend.h> #include <linux/smp.h> +#include <linux/units.h> #include <trace/events/ipi.h> #include <trace/events/kvm.h>
@@ -1763,7 +1764,8 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,int kvm_guest_time_update (struct kvm_vcpu *v) {struct pvclock_vcpu_time_info hv_clock = {}; -unsigned long flags, tgt_tsc_khz; +unsigned long flags; +u64 tgt_tsc_hz;unsigned seq;struct kvm_vcpu_arch *vcpu = &v->arch;struct kvm_arch *ka = &v->kvm->arch;
@@ -1789,8 +1791,8 @@ int kvm_guest_time_update(struct kvm_vcpu *v)/* Keep irq disabled to prevent changes to the clock */ local_irq_save (flags); - tgt_tsc_khz =get_cpu_tsc_khz (); -if (unlikely (tgt_tsc_khz ==0 )) { + tgt_tsc_hz = (u64 )get_cpu_tsc_khz () *HZ_PER_KHZ ; +if (unlikely (tgt_tsc_hz ==0 )) {local_irq_restore (flags);kvm_make_request (KVM_REQ_CLOCK_UPDATE , v);return 1 ;
@@ -1831,16 +1833,16 @@ int kvm_guest_time_update(struct kvm_vcpu *v)/* With all the info we got, fill in the values */ if (kvm_caps.has_tsc_control) { - tgt_tsc_khz =kvm_scale_tsc (tgt_tsc_khz, + tgt_tsc_hz =kvm_scale_tsc (tgt_tsc_hz, v->arch.l1_tsc_scaling_ratio); - tgt_tsc_khz = tgt_tsc_khz ? :1 ; + tgt_tsc_hz = tgt_tsc_hz ? :1 ; } -if (unlikely (vcpu->hw_tsc_khz != tgt_tsc_khz)) { -kvm_get_time_scale (NSEC_PER_SEC , tgt_tsc_khz *1000LL , +if (unlikely (vcpu->hw_tsc_hz != tgt_tsc_hz)) { +kvm_get_time_scale (NSEC_PER_SEC , tgt_tsc_hz, &vcpu->pvclock_tsc_shift, &vcpu->pvclock_tsc_mul); - vcpu->hw_tsc_khz = tgt_tsc_khz; + vcpu->hw_tsc_hz = tgt_tsc_hz; } hv_clock.tsc_shift = vcpu->pvclock_tsc_shift;
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 3/21 KVM: x86: Explicitly disable TSC scaling without CONSTANT_TSC
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
KVM does make an attempt to cope with non-constant TSC, and has notifiers to handle host TSC frequency changes. However, it *only* adjusts the KVM clock, and doesn't adjust TSC frequency scaling when the host changes.
This is presumably because non-constant TSCs were fixed in hardware long before TSC scaling was implemented, so there should never be real CPUs which have TSC scaling but *not* CONSTANT_TSC.
Such a combination could potentially happen in some odd L1 nesting environment, but it isn't worth trying to support it. Just make the dependency explicit.
--- arch/x86/kvm/svm/svm.c | 3 ++- arch/x86/kvm/vmx/vmx.c | 10 ++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-)
arch/x86/kvm/svm/svm.c +2 -1
@@ -5620,7 +5620,8 @@ static __init int svm_hardware_setup(void) XFEATURE_MASK_BNDCSR);if (tsc_scaling) { -if (!boot_cpu_has (X86_FEATURE_TSCRATEMSR )) { +if (!boot_cpu_has (X86_FEATURE_TSCRATEMSR ) || + !boot_cpu_has (X86_FEATURE_CONSTANT_TSC )) { tsc_scaling = false; }else {pr_info ("TSC scaling supported\n" );
arch/x86/kvm/vmx/vmx.c +10 -0
@@ -2841,6 +2841,16 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf,if (!cpu_has_sgx ()) _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_ENCLS_EXITING ; +/* +* KVM doesn't re-derive the TSC scaling ratio when the host TSC +* frequency changes, so TSC scaling is only usable with a constant +* TSC. Clear the control here rather than in vmx_hardware_setup() so +* that the per-CPU configs recomputed by vmx_check_processor_compat() +* stay consistent with the golden vmcs_config. +*/ +if (!boot_cpu_has (X86_FEATURE_CONSTANT_TSC )) + _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_TSC_SCALING ; +if (_cpu_based_exec_control &CPU_BASED_ACTIVATE_TERTIARY_CONTROLS ) _cpu_based_3rd_exec_control = adjust_vmx_controls64(KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL,
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 4/21 KVM: x86: Activate master clock immediately on vCPU creation
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Previously, the master clock was only activated when the first vCPU processed KVM_REQ_MASTERCLOCK_UPDATE during KVM_RUN. This meant that KVM_GET_CLOCK could not return the host_tsc field until after the first KVM_RUN, making it impossible for userspace to follow the documented TSC migration procedure without a dummy vCPU run.
Fix this by calling kvm_update_masterclock() directly from kvm_arch_vcpu_postcreate(), after kvm_synchronize_tsc() has already set all_vcpus_matched_freq. This ensures the master clock is active immediately, and KVM_GET_CLOCK returns a valid {host_tsc, realtime} pair as soon as a vCPU exists.
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -9447,6 +9447,8 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)return ;vcpu_load (vcpu);kvm_synchronize_tsc (vcpu,NULL ); +if (kvm_check_request (KVM_REQ_MASTERCLOCK_UPDATE , vcpu)) +kvm_update_masterclock (vcpu->kvm);vcpu_put (vcpu);/* poll control enabled by default */
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 5/21 KVM: x86: Compute kvmclock base without pvclock_gtod_data
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
get_kvmclock_base_ns() needs CLOCK_MONOTONIC_RAW + offs_boot. Compute this directly rather than reading offs_boot from the pvclock_gtod_data private copy. offs_boot only changes at suspend/resume so does not need to be atomically paired with the raw clock read.
--- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/x86.c +1 -1
@@ -930,7 +930,7 @@ static void update_pvclock_gtod(struct timekeeper *tk)static s64 get_kvmclock_base_ns (void ) {/* Count up from boot time, but with the frequency of the raw clock. */ -return ktime_to_ns (ktime_add (ktime_get_raw (), pvclock_gtod_data.offs_boot)); +return ktime_to_ns (ktime_mono_to_any (ktime_get_raw (),TK_OFFS_BOOT )); }#else static s64 get_kvmclock_base_ns (void )
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 6/21 KVM: x86: Avoid NTP frequency skew for KVM clock on 32-bit host
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Now that 64-bit KVM doesn't consume pvclock_gtod_data when getting the kvmclock base, use the ktime_mono_to_any() version on 32-bit hosts so that they too are unaffected by NPT sync.
Note, even if KVM still used pvclock_gtod_data for getting kvmclock base, it would still be very doable and reasonable to avoid skew on 32-bit hosts, it's just much simpler now that KVM uses ktime_mono_to_any().
Specifically, commit 53fafdbb8b21 ("KVM: x86: switch KVMCLOCK base to monotonic raw clock") did so only for 64-bit hosts, by capturing the boot offset from within the existing clocksource notifier update_pvclock_gtod().
That notifier was added in commit 16e8d74d2da9 ("KVM: x86: notifier for clocksource changes") but only on x86_64, because its original purpose was just to disable the "master clock" mode which is only supported on x86_64.
Now that the notifier is used for more than disabling master clock mode, enabling it would be trivial for 32-bit builds as well, but as above, that's a moot point since get_kvmclock_base_ns() no longer relies on the notifier.
--- arch/x86/kvm/x86.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/kvm/x86.c +1 -7
@@ -926,19 +926,13 @@ static void update_pvclock_gtod(struct timekeeper *tk) write_seqcount_end(&vdata->seq); } +#endif static s64 get_kvmclock_base_ns (void ) {/* Count up from boot time, but with the frequency of the raw clock. */ return ktime_to_ns (ktime_mono_to_any (ktime_get_raw (),TK_OFFS_BOOT )); } -#else -static s64 get_kvmclock_base_ns (void ) -{ -/* Master clock not used, so we can just use CLOCK_BOOTTIME. */ -return ktime_get_boottime_ns (); -} -#endif static uint32_t div_frac (uint32_t dividend,uint32_t divisor) {
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 7/21 KVM: x86: Drop unnecessary CPU pinning when computing/getting kvmclock
In reply to Sean ChristophersonWhen computing the current kvmclock value, don't pin the task to the current CPU for the entire duration of the master clock path, as the CPU pinning was never about ensuring rdtsc() and cpu_tsc_khz would agree. As pointed out by David, ka->use_master_clock can only be true when the host clocksource is TSC based, which in turn requires a stable, constant and synchronised TSC across all CPUs.
The CPU pinning was added in commit e2c2206a1899 ("KVM: x86: Fix potential preemption when get the current kvmclock timestamp") purely in response to a CONFIG_DEBUG_PREEMPT=y bug due to accessing a per-CPU variable with preemption enabled. Despite what the comment would suggest, including rdtsc() in the {get,put}_cpu() section was opportunistic. In fact, Paolo even said exactly that when suggesting that KVM guarantee the rdtsc() would execute on the same CPU[*]:
: Also, rdtsc() should really be on the same CPU as __this_cpu_read. We : know it's not really really necessary because the master clock is : active, but since we need a get_cpu/put_cpu pair, better be clean.
Nothing has changed in the last ~9 years, i.e. the rdtsc() still *should* be on the same CPU, but super strictly speaking, all will be fine if the task is migrated between grabbing the frequency and doing rdtsc(). Dropping the CPU pinning will allow dropping the rdtsc() entirely without having to resort to a large "rewrite get_kvmclock()" patch.
Opportunistically add a comment to explain why KVM needs to snapshot the frequency, because that _is_ a hard requirement to avoid reintroducing the bug fixed by commit e70b57a6ce4e ("KVM: X86: Fix softlockup when get the current kvmclock")
--- arch/x86/kvm/x86.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/kvm/x86.c +9 -6
@@ -1657,13 +1657,18 @@ static void __get_kvmclock(struct kvm *kvm, struct kvm_clock_data *data) {struct kvm_arch *ka = &kvm->arch;struct pvclock_vcpu_time_info hv_clock; +u64 tsc_hz; -/* both __this_cpu_read() and rdtsc() should be on the same cpu */ +/* +* Snapshot and validate the TSC frequency as kvmclock_cpu_down_prep() +* zeros the per-CPU value when a CPU is going offline. +*/ get_cpu (); + tsc_hz = (u64 )get_cpu_tsc_khz () *HZ_PER_KHZ ; +put_cpu (); data->flags =0 ; -if (ka->use_master_clock && - (static_cpu_has (X86_FEATURE_CONSTANT_TSC ) ||__this_cpu_read (cpu_tsc_khz))) { +if (ka->use_master_clock && tsc_hz) {#ifdef CONFIG_X86_64 struct timespec64 ts;
@@ -1677,15 +1682,13 @@ static void __get_kvmclock(struct kvm *kvm, struct kvm_clock_data *data) data->flags |=KVM_CLOCK_TSC_STABLE ; hv_clock.tsc_timestamp = ka->master_cycle_now; hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset; -kvm_get_time_scale (NSEC_PER_SEC ,get_cpu_tsc_khz () *1000LL , +kvm_get_time_scale (NSEC_PER_SEC , tsc_hz, &hv_clock.tsc_shift, &hv_clock.tsc_to_system_mul); data->clock =__pvclock_read_cycles (&hv_clock, data->host_tsc); }else { data->clock =get_kvmclock_base_ns () + ka->kvmclock_offset; } - -put_cpu (); }static void get_kvmclock (struct kvm *kvm,struct kvm_clock_data *data)
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 8/21 KVM: x86: Move "no master clock" fallback from __get_kvmclock() to get_kvmclock()
In reply to Sean ChristophersonMove the fallback logic for getting the current kvmclock when not in master clock mode out of __get_kvmclock() and into its sole caller, get_kvmclock(). This will allow using early-return logic in the master clock code, without having to resort to a do-while() loop and/or gotos.
No functional change intended.
--- arch/x86/kvm/x86.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
arch/x86/kvm/x86.c +15 -6
@@ -1653,12 +1653,16 @@ static unsigned long get_cpu_tsc_khz(void) }/* Called within read_seqcount_begin/retry for kvm->pvclock_sc. */ -static void __get_kvmclock (struct kvm *kvm,struct kvm_clock_data *data) +static bool __get_kvmclock_master_clock (struct kvm *kvm, +struct kvm_clock_data *data) {struct kvm_arch *ka = &kvm->arch;struct pvclock_vcpu_time_info hv_clock;u64 tsc_hz; +if (!ka->use_master_clock) +return false; +/* * Snapshot and validate the TSC frequency as kvmclock_cpu_down_prep() * zeros the per-CPU value when a CPU is going offline.
@@ -1667,8 +1671,10 @@ static void __get_kvmclock(struct kvm *kvm, struct kvm_clock_data *data) tsc_hz = (u64 )get_cpu_tsc_khz () *HZ_PER_KHZ ;put_cpu (); - data->flags =0 ; -if (ka->use_master_clock && tsc_hz) { +if (!tsc_hz) +return false; + + {#ifdef CONFIG_X86_64 struct timespec64 ts;
@@ -1686,9 +1692,9 @@ static void __get_kvmclock(struct kvm *kvm, struct kvm_clock_data *data) &hv_clock.tsc_shift, &hv_clock.tsc_to_system_mul); data->clock =__pvclock_read_cycles (&hv_clock, data->host_tsc); - }else { - data->clock =get_kvmclock_base_ns () + ka->kvmclock_offset; } + +return true; }static void get_kvmclock (struct kvm *kvm,struct kvm_clock_data *data)
@@ -1697,8 +1703,11 @@ static void get_kvmclock(struct kvm *kvm, struct kvm_clock_data *data)unsigned seq;do { + data->flags =0 ; + seq =read_seqcount_begin (&ka->pvclock_sc); -__get_kvmclock (kvm, data); +if (!__get_kvmclock_master_clock (kvm, data)) + data->clock =get_kvmclock_base_ns () + ka->kvmclock_offset; }while (read_seqcount_retry (&ka->pvclock_sc, seq)); }
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 9/21 KVM: x86: Wrap all of __get_kvmclock_master_clock() with CONFIG_X86_64=y
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Wrap the entire use_master_clock block in #ifdef CONFIG_X86_64, since use_master_clock is never true on 32-bit (host_tsc_clocksource is only set under CONFIG_X86_64), and declare hv_clock inside the block so it is not left as an unused variable on 32-bit.
--- arch/x86/kvm/x86.c | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-)
arch/x86/kvm/x86.c +16 -17
@@ -1656,8 +1656,10 @@ static unsigned long get_cpu_tsc_khz(void)static bool __get_kvmclock_master_clock (struct kvm *kvm,struct kvm_clock_data *data) { +#ifdef CONFIG_X86_64 struct kvm_arch *ka = &kvm->arch;struct pvclock_vcpu_time_info hv_clock; +struct timespec64 ts;u64 tsc_hz;if (!ka->use_master_clock)
@@ -1674,27 +1676,24 @@ static bool __get_kvmclock_master_clock(struct kvm *kvm,if (!tsc_hz)return false; - { -#ifdef CONFIG_X86_64 -struct timespec64 ts; - -if (kvm_get_walltime_and_clockread (&ts, &data->host_tsc)) { - data->realtime = ts.tv_nsec +NSEC_PER_SEC * ts.tv_sec; - data->flags |=KVM_CLOCK_REALTIME |KVM_CLOCK_HOST_TSC ; - } else -#endif +if (kvm_get_walltime_and_clockread (&ts, &data->host_tsc)) { + data->realtime = ts.tv_nsec +NSEC_PER_SEC * ts.tv_sec; + data->flags |=KVM_CLOCK_REALTIME |KVM_CLOCK_HOST_TSC ; + }else { data->host_tsc =rdtsc (); - - data->flags |=KVM_CLOCK_TSC_STABLE ; - hv_clock.tsc_timestamp = ka->master_cycle_now; - hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset; -kvm_get_time_scale (NSEC_PER_SEC , tsc_hz, - &hv_clock.tsc_shift, - &hv_clock.tsc_to_system_mul); - data->clock =__pvclock_read_cycles (&hv_clock, data->host_tsc); } + data->flags |=KVM_CLOCK_TSC_STABLE ; + hv_clock.tsc_timestamp = ka->master_cycle_now; + hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset; +kvm_get_time_scale (NSEC_PER_SEC , tsc_hz, + &hv_clock.tsc_shift, + &hv_clock.tsc_to_system_mul); + data->clock =__pvclock_read_cycles (&hv_clock, data->host_tsc);return true; +#else +return false; +#endif }static void get_kvmclock (struct kvm *kvm,struct kvm_clock_data *data)
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 10/21 KVM: x86: Fall back to non-master-clock if clockread fails in get_kvmclock()
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
When computing kvmclock and it's currently in master-clock mode, fall back to the non-master-clock path if the clock read fails, e.g. if the kernel's clocksource is transitioning away from TSC but ka->use_master_clock hasn't been updated yet.
The rdtsc() fallback was added (well, kept) in commit c68dc1b577ea ("KVM:
x86: Report host tsc and realtime values in KVM_GET_CLOCK") purely to avoid uninitialized variables and compilation problems on 32-bit kernels (already addressed). In hindsight, keeping the rdtsc() was a hack and a mistake.
--- arch/x86/kvm/x86.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
arch/x86/kvm/x86.c +6 -7
@@ -1676,14 +1676,13 @@ static bool __get_kvmclock_master_clock(struct kvm *kvm,if (!tsc_hz)return false; -if (kvm_get_walltime_and_clockread (&ts, &data->host_tsc)) { - data->realtime = ts.tv_nsec +NSEC_PER_SEC * ts.tv_sec; - data->flags |=KVM_CLOCK_REALTIME |KVM_CLOCK_HOST_TSC ; - }else { - data->host_tsc =rdtsc (); - } +if (!kvm_get_walltime_and_clockread (&ts, &data->host_tsc)) +return false; + + data->realtime = ts.tv_nsec +NSEC_PER_SEC * ts.tv_sec; + data->flags |=KVM_CLOCK_REALTIME |KVM_CLOCK_HOST_TSC | +KVM_CLOCK_TSC_STABLE ; - data->flags |=KVM_CLOCK_TSC_STABLE ; hv_clock.tsc_timestamp = ka->master_cycle_now; hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset; kvm_get_time_scale(NSEC_PER_SEC, tsc_hz,
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 11/21 KVM: x86: Fix KVM clock precision in get_kvmclock() with TSC scaling
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
When in master clock mode, the KVM clock is defined in terms of the guest TSC. But get_kvmclock() was computing it from the host TSC without applying TSC scaling, leading to a systemic drift from the values the guest computes from its own TSC.
Store the VM's TSC scaling ratio in kvm_arch and precompute the guest-TSC-based mul/shift in pvclock_update_vm_gtod_copy(). Use these in get_kvmclock() to scale the host TSC delta to guest TSC before converting to nanoseconds.
This avoids "definition C" of the KVM clock described in commit 633d7652f80f ("KVM: x86/xen: Do not corrupt KVM clock in kvm_xen_shared_info_init()").
--- arch/x86/include/asm/kvm_host.h | 4 +++ arch/x86/kvm/x86.c | 61 ++++++++++++++++++++++++--------- 2 files changed, 48 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -1236,6 +1236,7 @@ struct kvm_arch {u64 last_tsc_write;u32 last_tsc_khz;u64 last_tsc_offset; +u64 last_tsc_scaling_ratio;u64 cur_tsc_nsec;u64 cur_tsc_write;u64 cur_tsc_offset;
@@ -1251,6 +1252,9 @@ struct kvm_arch {u64 master_kernel_ns;u64 master_cycle_now;struct ratelimit_state kvmclock_update_rs; +u64 master_tsc_scaling_ratio; +s8 master_tsc_shift; +u32 master_tsc_mul;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;
arch/x86/kvm/x86.c +44 -17
97 lines not on this page · raw message
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 12/21 KVM: x86: Use get_kvmclock() in kvm_get_wall_clock_epoch()
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Now that get_kvmclock() correctly handles TSC scaling and captures both wallclock and kvmclock from the same TSC reading, kvm_get_wall_clock_epoch() can simply call it instead of duplicating the pvclock computation.
This eliminates the last instance of the "definition C" kvmclock calculation — as described in commit 633d7652f80f ("KVM: x86/xen: Do not corrupt KVM clock in kvm_xen_shared_info_init()") — which computed nanoseconds directly from the host TSC without accounting for guest TSC scaling.
--- arch/x86/kvm/x86.c | 59 +++++++--------------------------------------- 1 file changed, 9 insertions(+), 50 deletions(-)
arch/x86/kvm/x86.c +9 -50
@@ -1941,63 +1941,22 @@ int kvm_guest_time_update(struct kvm_vcpu *v)* wallclock and kvmclock times, and subtracting one from the other. * * Fall back to using their values at slightly different moments by -* calling ktime_get_real_ns() and get_kvmclock_ns() separately. +* calling ktime_get_real_ns() and get_kvmclock() separately. */ uint64_t kvm_get_wall_clock_epoch (struct kvm *kvm) { -#ifdef CONFIG_X86_64 -struct pvclock_vcpu_time_info hv_clock; -struct kvm_arch *ka = &kvm->arch; -unsigned long seq, local_tsc_khz; -struct timespec64 ts; -uint64_t host_tsc; +struct kvm_clock_data data; -do { - seq =read_seqcount_begin (&ka->pvclock_sc); - - local_tsc_khz =0 ; -if (!ka->use_master_clock) -break ; - -/* -* The TSC read and the call to get_cpu_tsc_khz() must happen -* on the same CPU. -*/ -get_cpu (); - - local_tsc_khz =get_cpu_tsc_khz (); - -if (local_tsc_khz && - !kvm_get_walltime_and_clockread (&ts, &host_tsc)) - local_tsc_khz =0 ;/* Fall back to old method */ - -put_cpu (); - -/* -* These values must be snapshotted within the seqcount loop. -* After that, it's just mathematics which can happen on any -* CPU at any time. -*/ - hv_clock.tsc_timestamp = ka->master_cycle_now; - hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset; - - }while (read_seqcount_retry (&ka->pvclock_sc, seq)); +get_kvmclock (kvm, &data);/* -* If the conditions were right, and obtaining the wallclock+TSC was -* successful, calculate the KVM clock at the corresponding time and -* subtract one from the other to get the guest's epoch in nanoseconds -* since 1970-01-01. +* If get_kvmclock() captured both wallclock and kvmclock from the +* same TSC reading, use them for a precise epoch calculation. */ -if (local_tsc_khz) { -kvm_get_time_scale (NSEC_PER_SEC , local_tsc_khz *NSEC_PER_USEC , - &hv_clock.tsc_shift, - &hv_clock.tsc_to_system_mul); -return ts.tv_nsec +NSEC_PER_SEC * ts.tv_sec - -__pvclock_read_cycles (&hv_clock, host_tsc); - } -#endif -return ktime_get_real_ns () -get_kvmclock_ns (kvm); +if (data.flags &KVM_CLOCK_REALTIME ) +return data.realtime - data.clock; + +return ktime_get_real_ns () - data.clock; }/*
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 13/21 KVM: x86: Fix compute_guest_tsc() to handle negative time deltas
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
The compute_guest_tsc() function computes the guest TSC at a given kernel_ns timestamp. When the master clock reference point (master_kernel_ns) is earlier than vcpu->arch.this_tsc_nsec, the delta is negative. Since pvclock_scale_delta() takes a u64, the negative value wraps to a huge positive number, producing a wildly wrong result.
Handle negative deltas explicitly by scaling the absolute value of the delta and applying it to this_tsc_write with the appropriate sign.
This is believed to be unreachable in practice; no path has been identified which invokes compute_guest_tsc() with a timestamp from before the vCPU's TSC generation was established. Fix it for robustness, in the spirit of defence in depth.
--- arch/x86/kvm/x86.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +9 -5
@@ -1056,11 +1056,15 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)static u64 compute_guest_tsc (struct kvm_vcpu *vcpu,s64 kernel_ns) { -u64 tsc =pvclock_scale_delta (kernel_ns-vcpu->arch.this_tsc_nsec, - vcpu->arch.virtual_tsc_mult, - vcpu->arch.virtual_tsc_shift); - tsc += vcpu->arch.this_tsc_write; -return tsc; +s64 delta_ns = kernel_ns - vcpu->arch.this_tsc_nsec; +u64 tsc; + +/* Handle negative deltas gracefully (master clock ref may be earlier) */ + tsc =pvclock_scale_delta (abs (delta_ns), + vcpu->arch.virtual_tsc_mult, + vcpu->arch.virtual_tsc_shift); + +return vcpu->arch.this_tsc_write + (delta_ns >=0 ? tsc : -tsc); }#ifdef CONFIG_X86_64
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 14/21 KVM: x86: Disable preemption, not IRQs, when getting TSC+freq pair
In reply to Sean ChristophersonDisable "just" preemption, not IRQs, when reading the TSC+frequency pair to update guest time, as disabling IRQs to protect against task migration is overkill (though it's *extremely* hard to see that it's overkill).
Disabling IRQs was added by commit 18068523d3a0 ("KVM: paravirtualized
clocksource: host part") before there was any coordination with timekeeping (presumably disabling IRQs prevented the kernel from completing a software- induced frequency change).
After the coordination and locking was added, commit c09664bb4418 ("KVM:
x86: fix deadlock in clock-in-progress request handling") moved the locking and coordination out of IRQ protection, and thus made disabling IRQs pointless, except for protecting get_cpu_tsc_khz().
And while cpu_tsc_khz is written only from IRQ context, and the *extremely* confusing double IPIs sent by __kvmclock_cpufreq_notifier() to update the per-CPU frequency make it seem like they would require readers to disable IRQs, it is safe to read and consume cpu_tsc_khz (via get_cpu_tsc_khz()) with IRQs enabled. The per-CPU variable is specifically written only in IRQ context to ensure hotplugging a CPU wouldn't write cpu_tsc_khz with a stale value (because apparently disabling IRQs would be too simple?!?).
As for the double IPIs in the frequency notifier, both IPIs are red herrings. The actual sequence that ensures KVM updates guest time with the new frequency is that the first write is completed *before* the notifier sets KVM_REQ_CLOCK_UPDATE for all vCPUs that last ran on the target pCPU. The first write is done via IPI to adhere to the above rules, and the second IPI is sent purely to kick any vCPU that happens to be running on the target CPU out of the guest. I.e. the second IPI writes cpu_tsc_khz out of pure KVM laziness: it saves having to define another IPI callback. In fact prior to commit 8cfdc0008542 ("KVM: x86: Make cpu_tsc_khz updates use local CPU"), KVM did indeed use an empty callback to ack the IPI. As for why it was deemed cleaner to abuse tsc_khz_changed()...
--- arch/x86/kvm/x86.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +7 -5
@@ -1799,7 +1799,6 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,int kvm_guest_time_update (struct kvm_vcpu *v) {struct pvclock_vcpu_time_info hv_clock = {}; -unsigned long flags;u64 tgt_tsc_hz;unsigned seq;struct kvm_vcpu_arch *vcpu = &v->arch;
@@ -1824,11 +1823,14 @@ int kvm_guest_time_update(struct kvm_vcpu *v) } }while (read_seqcount_retry (&ka->pvclock_sc, seq)); -/* Keep irq disabled to prevent changes to the clock */ -local_irq_save (flags); +/* +* Ensure reading the TSC+frequency pair is done on the same CPU. When +* NOT using the master clock, the TSC frequency may vary between CPUs. +*/ +preempt_disable (); tgt_tsc_hz = (u64 )get_cpu_tsc_khz () *HZ_PER_KHZ ;if (unlikely (tgt_tsc_hz ==0 )) { -local_irq_restore (flags); +preempt_enable ();kvm_make_request (KVM_REQ_CLOCK_UPDATE , v);return 1 ; }
@@ -1863,7 +1865,7 @@ int kvm_guest_time_update(struct kvm_vcpu *v)*/ vcpu->last_guest_tsc = tsc_timestamp; -local_irq_restore (flags); +preempt_enable ();/* With all the info we got, fill in the values */
Signature · 1 line
2.55.0.887.g758fc8c411-goog
[email protected]
AI reviewRe: [PATCH v10 14/21] KVM: x86: Disable preemption, not IRQs, when getting TSC+freq pair
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Replacing `local_irq_save()` with `preempt_disable()` allows interrupts to occur between reading…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Replacing local_irq_save() with preempt_disable() allows interrupts to occur between reading host_tsc and kernel_ns, causing the guest clock to abruptly jump forward. --
commit 905bf0112d20298ccaf2fcc14bf1871e4d77ff84
Author: Sean Christopherson <[email protected]>
Subject: KVM: x86: Disable preemption, not IRQs, when getting TSC+freq pair
This commit replaces local_irq_save() with preempt_disable() when reading the TSC and frequency pair to update guest time. The commit message notes that disabling IRQs is overkill for task migration protection.
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0
[ ... ]
Quoted 8 lines
arch/x86/kvm/x86.c +2
-2@@ -1824,11 +1823,14 @@ int kvm_guest_time_update(struct kvm_vcpu *v) } }while (read_seqcount_retry (&ka->pvclock_sc, seq)); -/* Keep irq disabled to prevent changes to the clock */ -local_irq_save (flags); +/* +* Ensure reading the TSC+frequency pair is done on the same CPU. When
arch/x86/kvm/x86.c int kvm_guest_time_update(struct kvm_vcpu *v) · line 1836
Quoted 10 lines
arch/x86/kvm/x86.c +4
-1+* NOT using the master clock, the TSC frequency may vary between CPUs. +*/ +preempt_disable (); tgt_tsc_hz = (u64 )get_cpu_tsc_khz () *HZ_PER_KHZ ;if (unlikely (tgt_tsc_hz ==0 )) { -local_irq_restore (flags); +preempt_enable ();kvm_make_request (KVM_REQ_CLOCK_UPDATE , v);return 1 ; }
[Severity: High]
Does this replacement of local_irq_save() with preempt_disable() expose the fallback clock path to an interrupt latency race?
Because preempt_disable() does not disable hardware IRQs, it appears an interrupt can now fire between reading host_tsc and kernel_ns:
kvm_guest_time_update() {
...
if (!use_master_clock) {
host_tsc = rdtsc();
<-- hardware interrupt fires here
kernel_ns = get_kvmclock_base_ns();
}
...
}If an interrupt occurs between these two reads, kernel_ns will be captured after the interrupt completes. Would this artificially inflate the kernel_ns value relative to the exact moment host_tsc was captured?
It looks like this could cause the elapsed interrupt time to be double-counted (once in kernel_ns and again when the guest calculates elapsed time since host_tsc), causing the guest clock to abruptly jump forward by milliseconds.
While later patches in this series update the use_master_clock path to use the atomic ktime_get_snapshot_id(), it seems this !use_master_clock fallback path remains exposed to this race.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Sean Christopherson
maintainerPATCH v10 15/21 KVM: x86: Make master clock logic in guest PV clock updates 64-bit only
In reply to Sean ChristophersonWrap kvm_guest_time_update()'s entire use_master_clock block in #ifdef CONFIG_X86_64 and provide a hardcoded-to-false variable for the 32-bit path, as use_master_clock is never true on 32-bit (host_tsc_clocksource is only set under CONFIG_X86_64)
Keep the local "ka" variable outside of the 64-bit-only code as it's also used for Xen code, but tag it as potentially unused since it's not used on 32-bit kernels without Xen emulation.
For all intents and purposes, no functional change intended.
--- arch/x86/kvm/x86.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
arch/x86/kvm/x86.c +11 -10
@@ -1798,22 +1798,21 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,int kvm_guest_time_update (struct kvm_vcpu *v) { +struct kvm_arch *ka__maybe_unused = &v->kvm->arch;struct pvclock_vcpu_time_info hv_clock = {};u64 tgt_tsc_hz; -unsigned seq;struct kvm_vcpu_arch *vcpu = &v->arch; -struct kvm_arch *ka = &v->kvm->arch;s64 kernel_ns;u64 tsc_timestamp, host_tsc; + +/* +* If the host uses TSC clock, then passthrough TSC as stable +* to the guest. +*/ +#ifdef CONFIG_X86_64 bool use_master_clock; + unsigned int seq; - kernel_ns =0 ; - host_tsc =0 ; - -/* -* If the host uses TSC clock, then passthrough TSC as stable -* to the guest. -*/ do { seq = read_seqcount_begin(&ka->pvclock_sc); use_master_clock = ka->use_master_clock;
@@ -1822,7 +1821,9 @@ int kvm_guest_time_update(struct kvm_vcpu *v) kernel_ns = ka->master_kernel_ns; } }while (read_seqcount_retry (&ka->pvclock_sc, seq)); - +#else +const bool use_master_clock = false; +#endif /* * Ensure reading the TSC+frequency pair is done on the same CPU. When * NOT using the master clock, the TSC frequency may vary between CPUs.
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 16/21 KVM: x86: Upscale TSC to "now", not master clock when updating PV clocks
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
When doing TSC upscaling in software, e.g. when the guest TSC frequency is configured to run faster than the host without hardware support, always advance guest TSC to "now", not to the master clock's reference values, which represents some time in the past.
In master clock mode, there are two points of time which need to be taken into account. First there is the master clock reference point, stored in kvm->arch.master_kernel_ns (and associated host TSC ->master_cycle_now). Secondly, there is the time *now*, at the point kvm_update_guest_time() is being called.
With software TSC upscaling, the guest TSC is getting further and further ahead of the host TSC as time elapses. So at time "now", the guest TSC should be further ahead of the host, than it was at master_kernel_ns. Not taking that into account means KVM was advancing the guest TSC only by the amount appropriate for the snapshot taken in the past, *not* to the current time (which is what RDTSC is supposed to return).
--- arch/x86/kvm/x86.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-)
arch/x86/kvm/x86.c +22 -7
@@ -1798,12 +1798,11 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,int kvm_guest_time_update (struct kvm_vcpu *v) { +u64 tgt_tsc_hz, tsc_timestamp, host_tsc, master_tsc, master_ns;struct kvm_arch *ka__maybe_unused = &v->kvm->arch;struct pvclock_vcpu_time_info hv_clock = {}; -u64 tgt_tsc_hz;struct kvm_vcpu_arch *vcpu = &v->arch;s64 kernel_ns; -u64 tsc_timestamp, host_tsc;/* * If the host uses TSC clock, then passthrough TSC as stable
@@ -1816,10 +1815,16 @@ int kvm_guest_time_update(struct kvm_vcpu *v)do { seq =read_seqcount_begin (&ka->pvclock_sc); use_master_clock = ka->use_master_clock; -if (use_master_clock) { - host_tsc = ka->master_cycle_now; - kernel_ns = ka->master_kernel_ns; +if (!use_master_clock) +continue ; + +if (!kvm_get_time_and_clockread (&kernel_ns, &host_tsc)) { + use_master_clock = false; +continue ; } + + master_tsc = ka->master_cycle_now; + master_ns = ka->master_kernel_ns; }while (read_seqcount_retry (&ka->pvclock_sc, seq));#else const bool use_master_clock = false;
@@ -1885,8 +1890,18 @@ int kvm_guest_time_update(struct kvm_vcpu *v) hv_clock.tsc_shift = vcpu->pvclock_tsc_shift; hv_clock.tsc_to_system_mul = vcpu->pvclock_tsc_mul; - hv_clock.tsc_timestamp = tsc_timestamp; - hv_clock.system_time = kernel_ns + v->kvm->arch.kvmclock_offset; +/* +* If the master clock is NOT in use, the reference time placed in the +* hv_clock is "now". If master clock is in use, the reference time is +* the master clock's snapshot from some time in the past, not "now". +*/ +if (use_master_clock) { + hv_clock.tsc_timestamp =kvm_read_l1_tsc (v, master_tsc); + hv_clock.system_time = master_ns + v->kvm->arch.kvmclock_offset; + }else { + hv_clock.tsc_timestamp = tsc_timestamp; + hv_clock.system_time = kernel_ns + v->kvm->arch.kvmclock_offset; + }/* If the host uses TSC clocksource, then it is stable */ hv_clock.flags =0 ;
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 17/21 KVM: x86: Simplify and comment kvm_get_time_scale()
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
The kvm_get_time_scale() function was entirely opaque. Add comments explaining what it does: compute a fixed-point multiplier and shift for converting TSC ticks to nanoseconds via pvclock_scale_delta().
Rename the local variables from the cryptic tps64/tps32/scaled64 to base_hz_u64/base32/scaled_hz_u64 to make the code self-documenting. The "tps32" name stood for "Ticks Per Second" but was misleading since it held the shifted base frequency, not a tick count.
No functional change.
--- arch/x86/kvm/x86.c | 55 +++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 15 deletions(-)
arch/x86/kvm/x86.c +40 -15
@@ -940,32 +940,57 @@ static uint32_t div_frac(uint32_t dividend, uint32_t divisor)return dividend; } -static void kvm_get_time_scale (uint64_t scaled_hz,uint64_t base_hz, +static void kvm_get_time_scale (u64 scaled_hz,u64 base_hz,s8 *pshift,u32 *pmultiplier) { -uint64_t scaled64; -int32_t shift =0 ; -uint64_t tps64; -uint32_t tps32; +u64 scaled_hz_u64 = scaled_hz; +s32 shift =0 ; +u64 base_hz_u64; +u32 base32; - tps64 = base_hz; - scaled64 = scaled_hz; -while (tps64 > scaled64*2 || tps64 &0xffffffff00000000ULL ) { - tps64 >>=1 ; +/* +* This function calculates a fixed-point multiplier and shift such +* that: +* time_ns = (tsc_cycles << shift) * multiplier >> 32 +* +* Where tsc_cycles tick at base_hz, and time_ns should count at +* scaled_hz (typically NSEC_PER_SEC for a TSC→nanoseconds conversion). +* +* The multiplier is: (scaled_hz << 32) / base_hz, adjusted by shift +* to keep everything in range. +*/ + + base_hz_u64 = base_hz; + +/* +* Start by shifting base_hz right until it fits in 32 bits, and +* is lower than double the target rate. This introduces a negative +* shift value which would result in pvclock_scale_delta() shifting +* the actual tick count right before performing the multiplication. +*/ +while (base_hz_u64 > scaled_hz_u64 *2 || base_hz_u64 >>32 ) { + base_hz_u64 >>=1 ; shift--; } - tps32 = (uint32_t )tps64; -while (tps32 <= scaled64 || scaled64 &0xffffffff00000000ULL ) { -if (scaled64 &0xffffffff00000000ULL || tps32 &0x80000000 ) - scaled64 >>=1 ; +/* Now the shifted base_hz fits in 32 bits. */ + base32 = (u32 )base_hz_u64; + +/* +* Next, shift scaled_hz right until it fits in 32 bits, and ensure +* that the shifted base_hz is strictly larger (so that the result of the +* final division also fits in 32 bits). +*/ +while (base32 <= scaled_hz_u64 || scaled_hz_u64 >>32 ) { +if (scaled_hz_u64 >>32 || base32 &BIT (31 )) + scaled_hz_u64 >>=1 ;else - tps32 <<=1 ; + base32 <<=1 ; shift++; } *pshift = shift; - *pmultiplier =div_frac (scaled64, tps32); + *pmultiplier =div_frac (scaled_hz_u64, base32); }#ifdef CONFIG_X86_64
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 18/21 KVM: x86: Remove implicit rdtsc() from kvm_compute_l1_tsc_offset()
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Let the callers pass the host TSC value in as an explicit parameter. This leaves some fairly obviously stupid code, which is using this function to compare the guest TSC at some *other* time, with the newly-minted TSC value from rdtsc(). Unless it's being used to measure *elapsed* time, that isn't very sensible.
In this case, "obviously stupid" is an improvement over being non-obviously so.
No functional change intended.
--- arch/x86/kvm/msrs.c | 3 ++- arch/x86/kvm/x86.c | 11 ++++++----- arch/x86/kvm/x86.h | 3 ++- 3 files changed, 10 insertions(+), 7 deletions(-)
arch/x86/kvm/msrs.c +2 -1
@@ -1631,7 +1631,8 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (msr_info->host_initiated) {kvm_synchronize_tsc (vcpu, &data); }else if (!vcpu->arch.guest_tsc_protected) { -u64 adj =kvm_compute_l1_tsc_offset (vcpu, data) - vcpu->arch.l1_tsc_offset; +u64 adj =kvm_compute_l1_tsc_offset (vcpu,rdtsc (), data) - + vcpu->arch.l1_tsc_offset;adjust_tsc_offset_guest (vcpu, adj); vcpu->arch.ia32_tsc_adjust_msr += adj; }
arch/x86/kvm/x86.c +6 -5
@@ -1155,11 +1155,12 @@ u64 kvm_scale_tsc(u64 tsc, u64 ratio)return _tsc; } -u64 kvm_compute_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 target_tsc) +u64 kvm_compute_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 host_tsc, +u64 target_tsc) {u64 tsc; - tsc =kvm_scale_tsc (rdtsc (), vcpu->arch.l1_tsc_scaling_ratio); + tsc =kvm_scale_tsc (host_tsc, vcpu->arch.l1_tsc_scaling_ratio);return target_tsc - tsc; }
@@ -1321,7 +1322,7 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)bool synchronizing = false;raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags); - offset =kvm_compute_l1_tsc_offset (vcpu, data); + offset =kvm_compute_l1_tsc_offset (vcpu,rdtsc (), data); ns =get_kvmclock_base_ns (); elapsed = ns - kvm->arch.last_tsc_nsec;
@@ -1370,7 +1371,7 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) }else {u64 delta =nsec_to_cycles (vcpu, elapsed); data += delta; - offset =kvm_compute_l1_tsc_offset (vcpu, data); + offset =kvm_compute_l1_tsc_offset (vcpu,rdtsc (), data); } matched = true; }
@@ -2627,7 +2628,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)mark_tsc_unstable ("KVM discovered backwards TSC" );if (kvm_check_tsc_unstable ()) { -u64 offset =kvm_compute_l1_tsc_offset (vcpu, +u64 offset =kvm_compute_l1_tsc_offset (vcpu,rdtsc (), vcpu->arch.last_guest_tsc);kvm_vcpu_write_tsc_offset (vcpu, offset);if (!vcpu->arch.guest_tsc_protected)
arch/x86/kvm/x86.h +2 -1
@@ -335,7 +335,8 @@ u64 kvm_scale_tsc(u64 tsc, u64 ratio);u64 kvm_read_l1_tsc (struct kvm_vcpu *vcpu,u64 host_tsc);u64 kvm_calc_nested_tsc_offset (u64 l1_offset,u64 l2_offset,u64 l2_multiplier);u64 kvm_calc_nested_tsc_multiplier (u64 l1_multiplier,u64 l2_multiplier); -u64 kvm_compute_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 target_tsc); +u64 kvm_compute_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 host_tsc, +u64 target_tsc);void kvm_vcpu_write_tsc_offset (struct kvm_vcpu *vcpu,u64 l1_offset); static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu,
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 19/21 KVM: x86: Use kernel timekeeping snapshots for getting kvmclock time since boot
In reply to Sean ChristophersonFrom: David Woodhouse <[email protected]>
Replace the KVM-private vgettsc()+do_kvmclock_base() timekeeping reimplementation with calls to the recently crafted, generic ktime_get_snapshot_id() interface. This is the first step towards dropping KVM's homebrewed implementation entirely (do_monotonic() and do_realtime() will be converted in the near future).
As with KVM's implementation, the snapshot provides both the system time and the raw_cycles (TSC), atomically paired using a sequence counter. The equivalents to vgettsc()'s TSC and HVCLOCK modes respectively are if the clocksource itself is TSC (cs_id == CSID_X86_TSC) and if the underlying hardware clocksource is TSC (hw_csid == CSID_X86_TSC). In the Hyper-V case, i.e. hw_csid == CSID_X86_TSC, if the clocksource couldn't provide a raw hardware counter value, treat the clock not being based on TSC, which which is equivalent to vgettsc() returning VDSO_CLOCKMODE_NONE.
Unlike KVM's current implementation, don't include offs_boot in the atomically-acquired tuple as there's simply no need to do so: the time since boot only changes at boot (duh) and at suspend/resume boundaries. Unless processes aren't being frozen/thawed before/after suspend/resume, which would completely break suspend/resume, TK_OFFS_BOOT can't change while kvm_get_time_and_clockread() is running. And if KVM does somehow try to take a snapshot during suspend, timekeeping core will WARN and refuse to provide the snapshot.
This is a step towards eliminating the pvclock_gtod_data private copy of timekeeping state and the associated notifier callback.
--- arch/x86/kvm/x86.c | 57 ++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 27 deletions(-)
arch/x86/kvm/x86.c +30 -27
@@ -35,6 +35,7 @@#include "smm.h" #include <linux/clocksource.h> +#include <linux/timekeeping.h> #include <linux/interrupt.h> #include <linux/kvm.h> #include <linux/fs.h>
@@ -1437,29 +1438,6 @@ static inline u64 vgettsc(struct pvclock_clock *clock, u64 *tsc_timestamp,return v * clock->mult; } -/* -* As with get_kvmclock_base_ns(), this counts from boot time, at the -* frequency of CLOCK_MONOTONIC_RAW (hence adding gtos->offs_boot). -*/ -static int do_kvmclock_base (s64 *t,u64 *tsc_timestamp) -{ -struct pvclock_gtod_data *gtod = &pvclock_gtod_data; -unsigned long seq; -int mode; -u64 ns; - -do { - seq =read_seqcount_begin (>od->seq); - ns = gtod->raw_clock.base_cycles; - ns +=vgettsc (>od->raw_clock, tsc_timestamp, &mode); - ns >>= gtod->raw_clock.shift; - ns +=ktime_to_ns (ktime_add (gtod->raw_clock.offset, gtod->offs_boot)); - }while (unlikely (read_seqcount_retry (>od->seq, seq))); - *t = ns; - -return mode; -} -/* * This calculates CLOCK_MONOTONIC at the time of the TSC snapshot, with * no boot time offset.
@@ -1504,6 +1482,29 @@ static int do_realtime(struct timespec64 *ts, u64 *tsc_timestamp)return mode; } +static bool kvm_snapshot_has_tsc (struct system_time_snapshot *snap, +u64 *tsc_timestamp) +{ +/* +* ktime_get_snapshot_id() cannot fail for standard clock IDs +* (only for invalid/aux clocks or during suspend, with a WARN). +*/ +if (!snap->valid) +return false; + +if (snap->cs_id ==CSID_X86_TSC ) { + *tsc_timestamp = snap->cycles; +return true; + } + +if (snap->hw_csid ==CSID_X86_TSC && snap->hw_cycles) { + *tsc_timestamp = snap->hw_cycles; +return true; + } + +return false; +} +/* * Calculates the kvmclock_base_ns (CLOCK_MONOTONIC_RAW + boot time) and * reports the TSC value from which it do so. Returns true if host is
@@ -1511,12 +1512,14 @@ static int do_realtime(struct timespec64 *ts, u64 *tsc_timestamp)*/ static bool kvm_get_time_and_clockread (s64 *kernel_ns,u64 *tsc_timestamp) { -/* checked again under seqlock below */ -if (!gtod_is_based_on_tsc (pvclock_gtod_data.clock.vclock_mode)) +struct system_time_snapshot snap = {}; + +ktime_get_snapshot_id (CLOCK_MONOTONIC_RAW , &snap); +if (!kvm_snapshot_has_tsc (&snap, tsc_timestamp))return false; -return gtod_is_based_on_tsc (do_kvmclock_base (kernel_ns, - tsc_timestamp)); + *kernel_ns =ktime_to_ns (ktime_mono_to_any (snap.systime,TK_OFFS_BOOT )); +return true; }/*
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 20/21 KVM: x86: Use kernel timekeeping snapshot for monotonic clock
In reply to Sean ChristophersonReplace the KVM-private vgettsc()+/do_monotonic() timekeeping reimplementation with calls to the recently crafted, generic ktime_get_snapshot_id() interface. As noted previously, the snapshot provides both the system time and the raw_cycles (TSC), atomically paired using a sequence counter.
--- arch/x86/kvm/x86.c | 33 ++++++--------------------------- 1 file changed, 6 insertions(+), 27 deletions(-)
arch/x86/kvm/x86.c +6 -27
@@ -1438,29 +1438,6 @@ static inline u64 vgettsc(struct pvclock_clock *clock, u64 *tsc_timestamp,return v * clock->mult; } -/* -* This calculates CLOCK_MONOTONIC at the time of the TSC snapshot, with -* no boot time offset. -*/ -static int do_monotonic (s64 *t,u64 *tsc_timestamp) -{ -struct pvclock_gtod_data *gtod = &pvclock_gtod_data; -unsigned long seq; -int mode; -u64 ns; - -do { - seq =read_seqcount_begin (>od->seq); - ns = gtod->clock.base_cycles; - ns +=vgettsc (>od->clock, tsc_timestamp, &mode); - ns >>= gtod->clock.shift; - ns +=ktime_to_ns (gtod->clock.offset); - }while (unlikely (read_seqcount_retry (>od->seq, seq))); - *t = ns; - -return mode; -} -static int do_realtime (struct timespec64 *ts,u64 *tsc_timestamp) {struct pvclock_gtod_data *gtod = &pvclock_gtod_data;
@@ -1528,12 +1505,14 @@ static bool kvm_get_time_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp)*/ bool kvm_get_monotonic_and_clockread (s64 *kernel_ns,u64 *tsc_timestamp) { -/* checked again under seqlock below */ -if (!gtod_is_based_on_tsc (pvclock_gtod_data.clock.vclock_mode)) +struct system_time_snapshot snap = {}; + +ktime_get_snapshot_id (CLOCK_MONOTONIC , &snap); +if (!kvm_snapshot_has_tsc (&snap, tsc_timestamp))return false; -return gtod_is_based_on_tsc (do_monotonic (kernel_ns, - tsc_timestamp)); + *kernel_ns =ktime_to_ns (snap.systime); +return true; }/*
Signature · 1 line
2.55.0.887.g758fc8c411-goog
Sean Christopherson
maintainerPATCH v10 21/21 KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
In reply to Sean ChristophersonReplace the KVM-private vgettsc()+/do_do_realtime() timekeeping reimplementation with calls to the recently crafted, generic ktime_get_snapshot_id() interface. As noted previously, the snapshot provides both the system time and the raw_cycles (TSC), atomically paired using a sequence counter.
With great pleasure, delete the now unused read_tsc() and vgettsc()
--- arch/x86/kvm/x86.c | 85 ++++------------------------------------------ 1 file changed, 6 insertions(+), 79 deletions(-)
arch/x86/kvm/x86.c +6 -79
@@ -1383,82 +1383,6 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)#ifdef CONFIG_X86_64 -static u64 read_tsc (void ) -{ -u64 ret = (u64 )rdtsc_ordered (); -u64 last = pvclock_gtod_data.clock.cycle_last; - -if (likely (ret >= last)) -return ret; - -/* -* GCC likes to generate cmov here, but this branch is extremely -* predictable (it's just a function of time and the likely is -* very likely) and there's a data dependence, so force GCC -* to generate a branch instead. I don't barrier() because -* we don't actually need a barrier, and if this function -* ever gets inlined it will generate worse code. -*/ - asmvolatile ("" ); -return last; -} - -static inline u64 vgettsc (struct pvclock_clock *clock,u64 *tsc_timestamp, -int *mode) -{ -u64 tsc_pg_val; -long v; - -switch (clock->vclock_mode) { -case VDSO_CLOCKMODE_HVCLOCK : -if (hv_read_tsc_page_tsc (hv_get_tsc_page (), - tsc_timestamp, &tsc_pg_val)) { -/* TSC page valid */ - *mode =VDSO_CLOCKMODE_HVCLOCK ; - v = (tsc_pg_val - clock->cycle_last) & - clock->mask; - }else { -/* TSC page invalid */ - *mode =VDSO_CLOCKMODE_NONE ; - } -break ; -case VDSO_CLOCKMODE_TSC : - *mode =VDSO_CLOCKMODE_TSC ; - *tsc_timestamp =read_tsc (); - v = (*tsc_timestamp - clock->cycle_last) & - clock->mask; -break ; -default : - *mode =VDSO_CLOCKMODE_NONE ; - } - -if (*mode ==VDSO_CLOCKMODE_NONE ) - *tsc_timestamp = v =0 ; - -return v * clock->mult; -} - -static int do_realtime (struct timespec64 *ts,u64 *tsc_timestamp) -{ -struct pvclock_gtod_data *gtod = &pvclock_gtod_data; -unsigned long seq; -int mode; -u64 ns; - -do { - seq =read_seqcount_begin (>od->seq); - ts->tv_sec = gtod->wall_time_sec; - ns = gtod->clock.base_cycles; - ns +=vgettsc (>od->clock, tsc_timestamp, &mode); - ns >>= gtod->clock.shift; - }while (unlikely (read_seqcount_retry (>od->seq, seq))); - - ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); - ts->tv_nsec = ns; - -return mode; -} -static bool kvm_snapshot_has_tsc (struct system_time_snapshot *snap,u64 *tsc_timestamp) {
@@ -1525,11 +1449,14 @@ bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp)static bool kvm_get_walltime_and_clockread (struct timespec64 *ts,u64 *tsc_timestamp) { -/* checked again under seqlock below */ -if (!gtod_is_based_on_tsc (pvclock_gtod_data.clock.vclock_mode)) +struct system_time_snapshot snap = {}; + +ktime_get_snapshot_id (CLOCK_REALTIME , &snap); +if (!kvm_snapshot_has_tsc (&snap, tsc_timestamp))return false; -return gtod_is_based_on_tsc (do_realtime (ts, tsc_timestamp)); + *ts =ktime_to_timespec64 (snap.systime); +return true; }#endif
Signature · 1 line
2.55.0.887.g758fc8c411-goog
[email protected]
AI reviewRe: [PATCH v10 21/21] KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Removal of KVM's local TSC clamping exposes guests to mismatched (tsc, walltime) pairs when the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Removal of KVM's local TSC clamping exposes guests to mismatched (tsc, walltime) pairs when the host TSC goes backwards, causing guest time to jump forward. --
commit 538536e82164aaa6297ef0ce3bf878f455e79fc7
Author: Sean Christopherson <[email protected]>
KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
This commit replaces the KVM-private timekeeping reimplementation with calls to the generic ktime_get_snapshot_id() interface. It provides both the system time and raw cycles atomically paired using a sequence counter, allowing the deletion of the unused read_tsc() and vgettsc() functions.
Quoted 4 lines
arch/x86/kvm/x86.c +0
-0
[ ... ]
arch/x86/kvm/x86.c bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp) · line 1454
Quoted 9 lines
arch/x86/kvm/x86.c +3
-2@@ -1525,11 +1449,14 @@ bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp)static bool kvm_get_walltime_and_clockread (struct timespec64 *ts,u64 *tsc_timestamp) { -/* checked again under seqlock below */ -if (!gtod_is_based_on_tsc (pvclock_gtod_data.clock.vclock_mode)) +struct system_time_snapshot snap = {}; + +ktime_get_snapshot_id (CLOCK_REALTIME , &snap);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Severity: High]
Does removing KVM's local TSC clamping expose guests to mismatched time pairs?
When establishing the guest's PV clock, KVM provides a master clock snapshot pairing a host TSC with a system time. The previous read_tsc() logic explicitly clamped the TSC to cycle_last to maintain mathematical consistency.
Looking at ktime_get_snapshot_id() in kernel/time/timekeeping.c, the function does not clamp the returned raw TSC (cycles):
kernel/time/timekeeping.c:ktime_get_snapshot_id() {
...
systime_snapshot->cycles = now;
...
}However, the system time is clamped by timekeeping_cycles_to_ns(), which skips the negative delta if now < cycle_last.
If the host TSC read on the current CPU is slightly behind tkr->cycle_last (for example, due to cross-CPU TSC skew), could this result in KVM returning a mathematically mismatched pair?
An older unclamped TSC (now) paired with a clamped future walltime (at cycle_last) would cause the elapsed delta (guest_tsc - master_tsc) to be artificially inflated when the guest later reads the TSC.
Could this cause the guest's clock to jump forward by that offset?
Quoted 8 lines
arch/x86/kvm/x86.c +3
-1+if (!kvm_snapshot_has_tsc (&snap, tsc_timestamp))return false; - return gtod_is_based_on_tsc(do_realtime(ts, tsc_timestamp)); + *ts =ktime_to_timespec64 (snap.systime); + return true; }#endif
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
Re: [PATCH v10 21/21] KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
Quoted 7 lines · On Wed, 2026-08-26 at 21:49 +0000, [email protected] wrote:
[Severity: High]
Does removing KVM's local TSC clamping expose guests to mismatched time pairs?When establishing the guest's PV clock, KVM provides a master clock snapshot pairing a host TSC with a system time. The previous
read_tsc()logic explicitly clamped the TSC tocycle_lastto maintain mathematical consistency.
I think this is basically the same class of issue I brought up in
https://lore.kernel.org/all/87v7beb7s3.ffs@fw13/ and Thomas said the right thing is just to ignore it?
If the discrepancy was more than a few cycles, we shouldn't be in masterclock mode anyway, should we?
Sean Christopherson
maintainerRe: [PATCH v10 21/21] KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
Quoted 9 lines · David Woodhouse
Quoted 7 lines · On Wed, 2026-08-26 at 21:49 +0000, [email protected] wrote:
[Severity: High]
Does removing KVM's local TSC clamping expose guests to mismatched time pairs?When establishing the guest's PV clock, KVM provides a master clock snapshot pairing a host TSC with a system time. The previous
read_tsc()logic explicitly clamped the TSC tocycle_lastto maintain mathematical consistency.
Quoted 6 lines
I think this is basically the same class of issue I brought up in
https://lore.kernel.org/all/87v7beb7s3.ffs@fw13/ and Thomas said the right thing is just to ignore it?If the discrepancy was more than a few cycles, we shouldn't be in masterclock mode anyway, should we?
Agreed. If a host CPU really has a TSC that is observably behind others, KVM will mark the TSC unstable when loading a vCPU on that pCPU.
s64 tsc_delta = !vcpu->arch.last_host_tsc ? 0 :
rdtsc() - vcpu->arch.last_host_tsc;
if (tsc_delta < 0)
mark_tsc_unstable("KVM discovered backwards TSC");And as Thomas pointed out, the discrepancy will show up at some point. E.g. even if the kernel provided a perfect pair, RDTSC from the guest would read a too-low value since KVM has historically required identical offsets to use masterclock, i.e. the guest would still be able to observe time going backwards if a vCPU did __pvclock_clocksource_read() with a TSC that is behind the masterclock reference.
Sean Christopherson
maintainerRe: [PATCH v10 21/21] KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC
In reply to Sean ChristophersonQuoted 11 lines · Sean Christopherson
Replace the KVM-private vgettsc()+/do_do_realtime() timekeeping reimplementation with calls to the recently crafted, generic
ktime_get_snapshot_id()interface. As noted previously, the snapshot provides both the system time and theraw_cycles(TSC), atomically paired using a sequence counter.With great pleasure, delete the now unused
read_tsc()andvgettsc()Signed-off-by David Woodhouse <[email protected]>[sean: separate from other conversions, express joy at vgettsc()'s demise]Signed-off-by Sean Christopherson <[email protected]>
Note to self, this needs to be fixed up so that David is the Author. I'll check the other patches too, I suspect I forgot to specify --author when splitting the original patch into 3.
Quoted 8 lines · Sean Christopherson
Part 1 of David's series to clean up the host-side mess of kvmclock. The cut point in this version is still quite arbitrary, but what's left for "part 2" fits nicely into 2-3 buckets:
1: Masterclock overhaul
2. New uAPI and tests[...]
Applied to kvm-x86 clocks, thanks!
[01/21] KVM: x86: Update "last guest TSC" snapshot prior to enabling IRQs/preemption
https://github.com/kvm-x86/linux/commit/ba78382f34c1 [02/21] KVM: x86: Improve accuracy of KVM clock when TSC scaling is in force https://github.com/kvm-x86/linux/commit/d4f54ed48e00 [03/21] KVM: x86: Explicitly disable TSC scaling without CONSTANT_TSC https://github.com/kvm-x86/linux/commit/dd0a8675f4ba [04/21] KVM: x86: Activate master clock immediately on vCPU creation https://github.com/kvm-x86/linux/commit/7254e09bb5e9 [05/21] KVM: x86: Compute kvmclock base without pvclock_gtod_data https://github.com/kvm-x86/linux/commit/7ffdb9d111c3 [06/21] KVM: x86: Avoid NTP frequency skew for KVM clock on 32-bit host https://github.com/kvm-x86/linux/commit/4c710e8182f4 [07/21] KVM: x86: Drop unnecessary CPU pinning when computing/getting kvmclock https://github.com/kvm-x86/linux/commit/796b73bba577 [08/21] KVM: x86: Move "no master clock" fallback from __get_kvmclock() to get_kvmclock() https://github.com/kvm-x86/linux/commit/1a926fb39304 [09/21] KVM: x86: Wrap all of __get_kvmclock_master_clock() with CONFIG_X86_64=y https://github.com/kvm-x86/linux/commit/8e7e24b88e26 [10/21] KVM: x86: Fall back to non-master-clock if clockread fails in get_kvmclock() https://github.com/kvm-x86/linux/commit/25f75fb43fd5 [11/21] KVM: x86: Fix KVM clock precision in get_kvmclock() with TSC scaling https://github.com/kvm-x86/linux/commit/abdef42276d8 [12/21] KVM: x86: Use get_kvmclock() in kvm_get_wall_clock_epoch() https://github.com/kvm-x86/linux/commit/c56419275caa [13/21] KVM: x86: Fix compute_guest_tsc() to handle negative time deltas https://github.com/kvm-x86/linux/commit/d9173f59df3a [14/21] KVM: x86: Disable preemption, not IRQs, when getting TSC+freq pair https://github.com/kvm-x86/linux/commit/917a53a4b461 [15/21] KVM: x86: Make master clock logic in guest PV clock updates 64-bit only https://github.com/kvm-x86/linux/commit/f21b04302404 [16/21] KVM: x86: Upscale TSC to "now", not master clock when updating PV clocks https://github.com/kvm-x86/linux/commit/92fac63cc739 [17/21] KVM: x86: Simplify and comment kvm_get_time_scale() https://github.com/kvm-x86/linux/commit/2918af628d0d [18/21] KVM: x86: Remove implicit rdtsc() from kvm_compute_l1_tsc_offset() https://github.com/kvm-x86/linux/commit/d4bbf2a8eb82 [19/21] KVM: x86: Use kernel timekeeping snapshots for getting kvmclock time since boot https://github.com/kvm-x86/linux/commit/ef7d809987c9 [20/21] KVM: x86: Use kernel timekeeping snapshot for monotonic clock https://github.com/kvm-x86/linux/commit/0a0ab23b05f0 [21/21] KVM: x86: Use kernel timekeeping snapshot to get walltime+TSC https://github.com/kvm-x86/linux/commit/d7841275e1fb