KVM: x86: Fix nested TSC scaling edge cases
Sean Christopherson
maintainerv3 cover KVM: x86: Fix nested TSC scaling edge cases
Amirmohammad's fix to saturate L2's TSC frequency when stacking scale up/down ratios across L1+L2 yields a final scaling ratio that can't be plugged into hardware. The actual fix is functionally the same as v2, but I rewrote the comments and changelog to try and make them more approachable to KVM developers (AI has a habit of generating extremely verbose, "mathy" descriptions of bugs and fixes).
Patch 2 fixes a flaw pointed out by Sashiko.
Patch 3 fixes an off-by-one issue in KVM_SET_TSC_KHZ that is a complete non-issue in practice, but confused the heck out of me and made writing the testcases unnecessarily annoying.
The rest of the pile extends and cleans up the nested TSC scaling test to validate the saturation fixes, and to test all four combinations of scaling L1+L2 up+down (the existing code testing only L1 down, L2 up).
v2: https://lore.kernel.org/all/[email protected]
Amirmohammad Eftekhar (1):
KVM: x86: Saturate L2's TSC frequency if it exceeds hardware supports
Sean Christopherson (20):
KVM: SVM: Fallback to the default TSC ratio if KVM tries to use a bad
multiplier
KVM: x86: Allow userspace to set KVM's max supported guest TSC
frequency
KVM: selftests: Use KVM's pRNG to randomize L1's TSC ratio in TSC
scaling test
KVM: selftests: Drop redundant VMWRITE of TSC_MULTIPLIER_HIGH
KVM: selftests: Drop unnecessary use of PRIu64 in nested TSC scaling
test
KVM: selftests: Randomize L2's scale factor in nested TSC scaling test
KVM: selftests: Rename TSC freq checkers in nested TSC scaling test
KVM: selftests: Extract guts of nested TSC scaling test to helper
function
KVM: selftests: Track L2 multiplier, not scale-up factor, in nested
TSC test
KVM: selftests: Print out the failing L{0,1,2} level in nested TSC
scaling test
KVM: selftests: Allow +/- 1 tolerance if expected TSC frequency is
<100
KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity
checked)
KVM: selftests: Explicitly pass TSC frequencies to guts of TSC scaling
test
KVM: selftests: Sanity check KVM's default TSC freq in nested TSC
scaling test
KVM: selftests: Test L1 "up" and L2 "down" in nested TSC scaling test
KVM: selftests: Verify that KVM saturates L2 TSC freq on
{under,over}flow
KVM: selftests: Test non-zero TSC offset on SVM in nested TSC scaling
test
KVM: selftests: Randomize L2's TSC offset in the nested TSC scaling
test
KVM: selftests: Use GUEST_SYNC2() in nested TSC scaling test
KVM: selftests: Spell out UCALL in nested TSC scaling test's enums
arch/x86/kvm/svm/svm.c | 9 + arch/x86/kvm/x86.c | 62 ++++- .../kvm/x86/nested_tsc_scaling_test.c | 232 +++++++++++------- 3 files changed, 209 insertions(+), 94 deletions(-)
base-commit: b378201ccd5280d0fff89bbe55e1eb00620ec0d5
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 1/21 KVM: x86: Saturate L2's TSC frequency if it exceeds hardware supports
From: Amirmohammad Eftekhar <[email protected]>
When computing the TSC multiplier for L2, use the minimum/maximum frequency supported by hardware if the resulting multiplier can't be programmed into hardware, i.e. saturate L2's frequency on both sides. This can happen if userspace is running L1 at a low/high frequency relative to L0, and L1 is doing the same for L2 (relative to L1), because although each of the L1 and L2 inputs are constrained and validated, the end result can exceed SVM's and VMX's architectural limits.
Don't bother trying to log an error or do something more sophisticated, because in practice only a misbehaving L1 (and/or host userspace) will run afoul of the flaw. On SVM, which has the smallest "range" by far (8 bits for the integer multiplier, versus 16 bits on VMX), KVM can still run L2 with a frequency 255x that of L0. Even assuming a pessimistic L0 TSC frequency of 1GHz (modern CPUs run TSC at 2GHz+), L2 would need to be running at a whopping ~255GHz for the limitation to be a problem. Not to mention that if L2 is running at a legitimate frequency, then running that VM on existing hardware is already doomed irrespective of the nested angle.
Open code the mul_u64_u64_shr() if the compiler natively supports 128-bit integers, mostly to avoid having to do the multiply twice for what should be a very rare scenario, but also so that there's a slightly more scrutable sequence that can be read to understand what all is going on.
--- arch/x86/kvm/x86.c | 55 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +51 -4
@@ -1190,11 +1190,58 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_calc_nested_tsc_offset);u64 kvm_calc_nested_tsc_multiplier (u64 l1_multiplier,u64 l2_multiplier) { -if (l2_multiplier != kvm_caps.default_tsc_scaling_ratio) -return mul_u64_u64_shr (l1_multiplier, l2_multiplier, - kvm_caps.tsc_scaling_ratio_frac_bits); +u8 frac_bits = kvm_caps.tsc_scaling_ratio_frac_bits; +u64 nested_multiplier; -return l1_multiplier; +if (l2_multiplier == kvm_caps.default_tsc_scaling_ratio) +return l1_multiplier; + +/* +* The shift is fixed on both AMD and Intel, and operates on a 64-bit +* value. I.e. a shift greater than 63 is completely nonsensical. +*/ +if (WARN_ON_ONCE (frac_bits >63 )) +return l1_multiplier; + +/* +* If the resulting multiplier can't be programmed into hardware, run +* L2 at the minimum/maximum frequency supported by hardware, i.e. +* saturate L2's frequency on both sides. Because L2's frequency needs +* to be distilled down to a single multiplier to get from: +* +* L2 = (((L0 * L1_mult) >> frac) * L2_mult) >> frac) +* +* to: +* +* L2 = (L0 * mult) >> frac +* +* very small/large L1 and L2 multipliers can underflow/overflow the +* minimum/maximum multiplier supported by hardware when combined into +* a single value. +* +* Manually check for the case where the result would overflow a u64, +* i.e. if the multiplier would be silently truncated before the "too +* large" check. Avoid doing the multiply twice in the common case +* where the compiler natively supports 128-bit values. +*/ +#ifdef CONFIG_ARCH_SUPPORTS_INT128 +unsigned __int128 m = (unsigned __int128 )l1_multiplier * l2_multiplier; + +if (m >> (64 + frac_bits)) +return kvm_caps.max_tsc_scaling_ratio; + + nested_multiplier = m >> frac_bits; +#else +if (mul_u64_u64_shr (l1_multiplier, l2_multiplier,64 + frac_bits)) +return kvm_caps.max_tsc_scaling_ratio; + + nested_multiplier =mul_u64_u64_shr (l1_multiplier, l2_multiplier, frac_bits); +#endif +if (nested_multiplier > kvm_caps.max_tsc_scaling_ratio) +return kvm_caps.max_tsc_scaling_ratio; + +/* The minimum multiplier is '1' on both AMD and Intel. */ +return nested_multiplier ?:1 ; }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_calc_nested_tsc_multiplier);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 2/21 KVM: SVM: Fallback to the default TSC ratio if KVM tries to use a bad multiplier
In reply to Sean ChristophersonExplicitly WARN and fallback to the default TSC ratio, i.e. run the guest at L0's TSC frequency, if KVM tries to program a bad multiplier value. As pointed out by Sashiko, leaving the MSR as-is bleeds state from the vCPU that last ran on the pCPU into the likely-misbehaving current vCPU.
Leaking a vCPU's frequency isn't very interesting, and corrupting KVM's cache isn't a big deal either since KVM would only refuse to try to write the same bad value in the future, but falling back to the default value is trivial, and explicitly WARNing ensures a KVM bug won't slip by silently.
E.g. because ex_handler_msr() only WARNs once for *all* WRMSRs, it's possible for the potentially-fatal-to-the-guest issue to not exhibit any visible symptoms in the host.
--- arch/x86/kvm/svm/svm.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/kvm/svm/svm.c +9 -0
@@ -533,6 +533,15 @@ static int svm_check_processor_compat(void)static void __svm_write_tsc_multiplier (u64 multiplier) { +/* +* Fallback to the default ratio if KVM is buggy and tries to program +* an unsupported scaling ratio, e.g. so that the guest has a chance of +* surviving, so that the cache isn't stale/corrupted, and so that KVM +* doesn't leak state across VMs. +*/ +if (WARN_ON_ONCE (multiplier &SVM_TSC_RATIO_RSVD )) + multiplier =SVM_TSC_RATIO_DEFAULT ; +if (multiplier ==__this_cpu_read (current_tsc_ratio))return ;
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 3/21 KVM: x86: Allow userspace to set KVM's max supported guest TSC frequency
In reply to Sean ChristophersonReject KVM_SET_TSC_KHZ if the incoming frequency is strictly greater than KVM's max supported frequency, not if the frequency is greater than *or* equal to the max frequency. The effective off-by-one bug came about via (dubious) review feedback, which also subtly collided with a functional change in later versions of the original TSC scaling series.
In v2 of the original TSC scaling series[1], KVM set its absolute min/max to [1, UINT_MAX], i.e. allowed any value that would fit in the u32 passed to KVM_SET_TSC_KHZ[2].
min = max(1ULL, __scale_tsc(tsc_khz, TSC_RATIO_MIN)); max = min(0xffffffffULL, __scale_tsc(tsc_khz, TSC_RATIO_MAX));
That prompted Avi to suggest rejecting the "equals" case, presumably because the check would always succeed given an absolute max of UINT_MAX? But even that doesn't hold up to scrutiny, as allowing the actual min/max isn't inherently unsafe. Regardless, that feedback was taken and applied to future versions, but only for the maximum, not the minimum.
Review
Quoted 4 lines
(quoted diff) +4
-0+ + r = -EINVAL; + if (user_tsc_khz< kvm_min_guest_tsc_khz || + user_tsc_khz> kvm_max_guest_tsc_khz)
<= and >= are probably safer.
v3 of the series[3] then also realized that allowing UINT_MAX would lead to undesirable interactions with KVM_GET_TSC_KHZ, which returns a *signed* 32-bit integer that is implicitly converted into a signed 64-bit value on 64-bit kernels. I.e. allowing a value greater than INT_MAX would result in userspace observing a negative value when doing KVM_GET_TSC_KHZ after KVM_SET_TSC_KHZ.
/*
* Make sure the user can only configure tsc_khz values that
* fit into a signed integer.
* A min value is not calculated needed because it will always
* be 1 on all machines and a value of 0 is used to disable
* tsc-scaling for the vcpu.
*/
max = min(0x7fffffffULL, __scale_tsc(tsc_khz, TSC_RATIO_MAX));
kvm_max_guest_tsc_khz = max;
v3 also dropped the explicit minimum tracking, as both AMD and Intel support a minimum *fractional* ratio of 1, i.e. AMD and Intel support a minimum frequency of "host / 2^32" and "host / 2^48" respectively. And because KVM_GET_TSC_KHZ (and KVM itself) only supports frequencies that fit in a signed 32-bit integer, even AMD's more coarse-grained ratio can scale down any host frequency to '1', i.e. KVM can always support a minimum frequency of 1KHz. Rather than explicitly reject a frequency of 1KHz, v3 simply dropped the minimum check, i.e. ignored the "<=" suggestion, but kept the ">=" side of things.
As a result, KVM now has a bizarre uABI where KVM_SET_TSC_KHZ tops out at INT_MAX-1 for no discernible reason. Fix the off-by-one flaw to provide a less weird uABI, and so that KVM_SET_TSC_KHZ accepts the maximum possible value that can be returned by KVM_GET_TSC_KHZ (without running afoul of casting issues; KVM doesn't actually sanity check that tsc_khz fits in a signed 32-bit value, which is a non-issue in practice because the units are KHz, not Hz, i.e. KVM_GET_TSC_KHZ is fine until CPUs with a TSC frequency greater than ~2.147PHz come along).
Note, in practice, no real world VMM is likely to care. As above, running afoul of the off-by-one issue would mean trying to configure a virtual TSC frequency that is three orders of magnitude greater than what current CPUs support.
Opportunistically explain *why* KVM restricts KVM_SET_TSC_KHZ to values that fit in a signed integer, as it requires far too much spelunking to piece together the connection to KVM_GET_TSC_KHZ.
--- arch/x86/kvm/x86.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/kvm/x86.c +4 -3
@@ -3718,7 +3718,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, user_tsc_khz = (u32 )arg;if (kvm_caps.has_tsc_control && - user_tsc_khz >= kvm_caps.max_guest_tsc_khz) + user_tsc_khz > kvm_caps.max_guest_tsc_khz) goto out;if (user_tsc_khz ==0 )
@@ -4685,7 +4685,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) user_tsc_khz = (u32 )arg;if (kvm_caps.has_tsc_control && - user_tsc_khz >= kvm_caps.max_guest_tsc_khz) + user_tsc_khz > kvm_caps.max_guest_tsc_khz) goto out;if (user_tsc_khz ==0 )
@@ -7174,7 +7174,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)if (kvm_caps.has_tsc_control) {/* * Make sure the user can only configure tsc_khz values that -* fit into a signed integer. +* fit into a signed integer, otherwise KVM_GET_TSC_KHZ would +* return a negative value and confuse userspace. * A min value is not calculated because it will always * be 1 on all machines. */
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 4/21 KVM: selftests: Use KVM's pRNG to randomize L1's TSC ratio in TSC scaling test
In reply to Sean ChristophersonUse KVM's pRNG to randomize L1's TSC ratio in the nested TSC scaling test, so that reproducing a failure with a specific seed is slightly easier.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1 -2
@@ -166,8 +166,7 @@ int main(int argc, char *argv[])* referenced by both main() and l1_guest_code() and using a global * variable does not work. */ -srand (time (NULL )); - l1_scale_factor = (rand () %9 ) +2 ; + l1_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;printf ("L1's scale down factor is: %" PRIu64"\n" , l1_scale_factor);printf ("L2's scale up factor is: %llu\n" ,L2_SCALE_FACTOR );
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 5/21 KVM: selftests: Drop redundant VMWRITE of TSC_MULTIPLIER_HIGH
In reply to Sean ChristophersonDrop a redundant VMWRITE of TSC_MULTIPLIER_HIGH in the nested TSC scaling test. KVM selftests are 64-bit only, i.e. the full 64-bit multiplier is always written by setting TSC_MULTIPLIER.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 1 - 1 file changed, 1 deletion(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0 -1
@@ -122,7 +122,6 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)vmwrite (TSC_OFFSET ,TSC_OFFSET_L2 );vmwrite (TSC_MULTIPLIER ,TSC_MULTIPLIER_L2 ); -vmwrite (TSC_MULTIPLIER_HIGH ,TSC_MULTIPLIER_L2 >>32 );/* launch L2 */ vmlaunch ();
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 6/21 KVM: selftests: Drop unnecessary use of PRIu64 in nested TSC scaling test
In reply to Sean ChristophersonUse a plain "lu" instead of "PRIu64" to print u64s in the nested TSC scaling test, as KVM selftests are 64-bit only (and supporting 32-bit x86 selftest would be an absurdly large lift, for something absolutely no one cares about).
--- .../kvm/x86/nested_tsc_scaling_test.c | 21 +++++++------------ 1 file changed, 7 insertions(+), 14 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +7 -14
@@ -41,14 +41,9 @@ static void compare_tsc_freq(u64 actual, u64 expected) thresh_low = expected - tolerance; thresh_high = expected + tolerance; -TEST_ASSERT (thresh_low < actual, -"TSC freq is expected to be between %" PRIu64" and %" PRIu64 -" but it actually is %" PRIu64, - thresh_low, thresh_high, actual); -TEST_ASSERT (thresh_high > actual, -"TSC freq is expected to be between %" PRIu64" and %" PRIu64 -" but it actually is %" PRIu64, - thresh_low, thresh_high, actual); +TEST_ASSERT (thresh_low < actual && thresh_high > actual, +"TSC freq is '%lu', expected to be between %lu and %lu" , + actual, thresh_low, thresh_high); }static void check_tsc_freq (int level)
@@ -166,7 +161,7 @@ int main(int argc, char *argv[])* variable does not work. */ l1_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; -printf ("L1's scale down factor is: %" PRIu64"\n" , l1_scale_factor); +printf ("L1's scale down factor is: %lu\n" , l1_scale_factor);printf ("L2's scale up factor is: %llu\n" ,L2_SCALE_FACTOR ); tsc_start =rdtsc ();
@@ -174,7 +169,7 @@ int main(int argc, char *argv[]) tsc_end =rdtsc (); l0_tsc_freq = tsc_end - tsc_start; -printf ("real TSC frequency is around: %" PRIu64"\n" , l0_tsc_freq); +printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code);
@@ -207,16 +202,14 @@ int main(int argc, char *argv[])break ;case UCHECK_L1 : l1_tsc_freq = uc.args[1 ]; -printf ("L1's TSC frequency is around: %" PRIu64 -"\n" , l1_tsc_freq); +printf ("L1's TSC frequency is around: %lu\n" , l1_tsc_freq);compare_tsc_freq (l1_tsc_freq, l0_tsc_freq / l1_scale_factor);break ;case UCHECK_L2 : l2_tsc_freq = uc.args[1 ]; -printf ("L2's TSC frequency is around: %" PRIu64 -"\n" , l2_tsc_freq); +printf ("L2's TSC frequency is around: %lu\n" , l2_tsc_freq);compare_tsc_freq (l2_tsc_freq, l1_tsc_freq *L2_SCALE_FACTOR );
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 7/21 KVM: selftests: Randomize L2's scale factor in nested TSC scaling test
In reply to Sean ChristophersonRandomize L2's scale factor in the nested TSC scaling test; the test's comment that "using a global variable does not work" just means the author didn't know how to use sync_global_to_guest().
--- .../kvm/x86/nested_tsc_scaling_test.c | 21 ++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9 -12
@@ -17,10 +17,9 @@#include "kselftest.h" /* L2 is scaled up (from L1's perspective) by this factor */ -#define L2_SCALE_FACTOR 4ULL +static u64 l2_scale_factor;#define TSC_OFFSET_L2 ((u64)-33125236320908) -#define TSC_MULTIPLIER_L2 (L2_SCALE_FACTOR << 48)enum {USLEEP ,UCHECK_L1 ,UCHECK_L2 };#define GUEST_SLEEP (sec) ucall(UCALL_SYNC, 2, USLEEP, sec)
@@ -81,7 +80,7 @@ static void l1_svm_code(struct svm_test_data *svm)generic_svm_setup (svm, l2_guest_code);/* enable TSC scaling for L2 */ -wrmsr (MSR_AMD64_TSC_RATIO ,L2_SCALE_FACTOR <<32 ); +wrmsr (MSR_AMD64_TSC_RATIO , l2_scale_factor <<32 );/* launch L2 */ run_guest (svm->vmcb, svm->vmcb_gpa);
@@ -116,7 +115,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)vmwrite (SECONDARY_VM_EXEC_CONTROL , control);vmwrite (TSC_OFFSET ,TSC_OFFSET_L2 ); -vmwrite (TSC_MULTIPLIER ,TSC_MULTIPLIER_L2 ); +vmwrite (TSC_MULTIPLIER , l2_scale_factor <<48 );/* launch L2 */ vmlaunch ();
@@ -154,15 +153,12 @@ int main(int argc, char *argv[])TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ()); -/* -* We set L1's scale factor to be a random number from 2 to 10. -* Ideally we would do the same for L2's factor but that one is -* referenced by both main() and l1_guest_code() and using a global -* variable does not work. -*/ +/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ l1_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;printf ("L1's scale down factor is: %lu\n" , l1_scale_factor); -printf ("L2's scale up factor is: %llu\n" ,L2_SCALE_FACTOR ); + + l2_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; +printf ("L2's scale up factor is: %lu\n" , l2_scale_factor); tsc_start =rdtsc ();sleep (1 );
@@ -172,6 +168,7 @@ int main(int argc, char *argv[])printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code); +sync_global_to_guest (vm, l2_scale_factor);if (kvm_cpu_has (X86_FEATURE_VMX ))vcpu_alloc_vmx (vm, &guest_gva);
@@ -212,7 +209,7 @@ int main(int argc, char *argv[])printf ("L2's TSC frequency is around: %lu\n" , l2_tsc_freq);compare_tsc_freq (l2_tsc_freq, - l1_tsc_freq *L2_SCALE_FACTOR ); + l1_tsc_freq * l2_scale_factor);break ; }break ;
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 8/21 KVM: selftests: Rename TSC freq checkers in nested TSC scaling test
In reply to Sean ChristophersonRename the nested TSC scaling test's helpers to clarify which one is used by guest code and which one is used by host code, and to make it more obvious that the host version *checks* the result and asserts on errors. Contrary to what "compare" suggests, the helper doesn't return whether or not the actual vs. expected values match.
--- .../kvm/x86/nested_tsc_scaling_test.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9 -9
@@ -32,7 +32,7 @@ enum { USLEEP, UCHECK_L1, UCHECK_L2 };* measurements, a difference of 1% between the actual and the expected value * is tolerated. */ -static void compare_tsc_freq (u64 actual,u64 expected) +static void host_check_tsc_freq (u64 actual,u64 expected) {u64 tolerance, thresh_low, thresh_high;
@@ -45,7 +45,7 @@ static void compare_tsc_freq(u64 actual, u64 expected) actual, thresh_low, thresh_high); } -static void check_tsc_freq (int level) +static void guest_check_tsc_freq (int level) {u64 tsc_start, tsc_end, tsc_freq;
@@ -66,7 +66,7 @@ static void check_tsc_freq(int level)static void l2_guest_code (void ) { -check_tsc_freq (UCHECK_L2 ); +guest_check_tsc_freq (UCHECK_L2 );/* exit to L1 */ __asm__ __volatile__("vmcall" );
@@ -75,7 +75,7 @@ static void l2_guest_code(void)static void l1_svm_code (struct svm_test_data *svm) {/* check that L1's frequency looks alright before launching L2 */ -check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCHECK_L1 );generic_svm_setup (svm, l2_guest_code);
@@ -87,7 +87,7 @@ static void l1_svm_code(struct svm_test_data *svm)GUEST_ASSERT (svm->vmcb->control.exit_code ==SVM_EXIT_VMMCALL );/* check that L1's frequency still looks good */ -check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCHECK_L1 );GUEST_DONE (); }
@@ -97,7 +97,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)u32 control;/* check that L1's frequency looks alright before launching L2 */ -check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCHECK_L1 );prepare_for_vmx_operation (vmx_pages);load_vmcs (vmx_pages);
@@ -122,7 +122,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)GUEST_ASSERT (vmread (VM_EXIT_REASON ) ==EXIT_REASON_VMCALL );/* check that L1's frequency still looks good */ -check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCHECK_L1 );GUEST_DONE (); }
@@ -201,14 +201,14 @@ int main(int argc, char *argv[]) l1_tsc_freq = uc.args[1 ];printf ("L1's TSC frequency is around: %lu\n" , l1_tsc_freq); -compare_tsc_freq (l1_tsc_freq, +host_check_tsc_freq (l1_tsc_freq, l0_tsc_freq / l1_scale_factor);break ;case UCHECK_L2 : l2_tsc_freq = uc.args[1 ];printf ("L2's TSC frequency is around: %lu\n" , l2_tsc_freq); -compare_tsc_freq (l2_tsc_freq, +host_check_tsc_freq (l2_tsc_freq, l1_tsc_freq * l2_scale_factor);break ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 08/21] KVM: selftests: Rename TSC freq checkers in nested TSC scaling test
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] selftests: guest_check_tsc_freq name contradicts commit message semantics --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] selftests: guest_check_tsc_freq name contradicts commit message semantics --
commit 7c759b101cdd9e99d261e6f3022bcb5096eeadfb
Author: Sean Christopherson <[email protected]>
KVM: selftests: Rename TSC freq checkers in nested TSC scaling test
This commit renames the nested TSC scaling test helpers to clarify guest vs host usage. It aims to emphasize that the host version performs the actual assertion check on the result.
Quoted 4 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0
[ ... ]
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void compare_tsc_freq(u64 actual, u64 expected) · line 48
Quoted 6 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-1@@ -45,7 +45,7 @@ static void compare_tsc_freq(u64 actual, u64 expected) actual, thresh_low, thresh_high); } -static void check_tsc_freq (int level) +static void guest_check_tsc_freq (int level)
[Severity: Low]
The commit message states the goal is to make it more obvious that the host version *checks* the result and asserts on errors. However, renaming the guest helper to guest_check_tsc_freq implies it also performs validation, whereas it merely measures the frequency and invokes GUEST_CHECK() to pass the value back to the host for verification.
Would a name like guest_measure_tsc_freq be more semantically accurate given the commit's stated intent?
Quoted 3 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0{u64 tsc_start, tsc_end, tsc_freq;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Sean Christopherson
maintainerRe: [PATCH v3 08/21] KVM: selftests: Rename TSC freq checkers in nested TSC scaling test
Quoted 22 lines · On Wed, Sep 30, 2026, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Quoted 4 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0[ ... ]
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void compare_tsc_freq(u64 actual, u64 expected) · line 48
Quoted 6 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-1@@ -45,7 +45,7 @@ static void compare_tsc_freq(u64 actual, u64 expected) actual, thresh_low, thresh_high); } -static void check_tsc_freq (int level) +static void guest_check_tsc_freq (int level)[Severity: Low]
The commit message states the goal is to make it more obvious that the host version *checks* the result and asserts on errors. However, renaming the guest helper toguest_check_tsc_freqimplies it also performs validation, whereas it merely measures the frequency and invokesGUEST_CHECK()to pass the value back to the host for verification.
Quoted 2 lines
Would a name like
guest_measure_tsc_freqbe more semantically accurate given the commit's stated intent?
No, because hopefully humans will understand that even though the check is ultimately performed by the host, guest_check_tsc_freq() really does check the guest's observed frequency.
Sean Christopherson
maintainerPATCH v3 9/21 KVM: selftests: Extract guts of nested TSC scaling test to helper function
In reply to Sean ChristophersonExtract the guts of the nested TSC scaling test to a separate helper in anticipation of extending the test beyond its single "scale L1 up, L2 down" testcase.
No functional change intended.
--- .../kvm/x86/nested_tsc_scaling_test.c | 54 ++++++++++--------- 1 file changed, 30 insertions(+), 24 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +30 -24
@@ -135,38 +135,18 @@ static void l1_guest_code(void *data)l1_svm_code (data); } -int main (int argc,char *argv[]) +static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale) { +u64 tsc_khz, l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm; -gva_t guest_gva =0 ; +gva_t guest_gva; -u64 tsc_start, tsc_end; -u64 tsc_khz; -u64 l1_scale_factor; -u64 l0_tsc_freq =0 ; -u64 l1_tsc_freq =0 ; -u64 l2_tsc_freq =0 ; + l2_scale_factor = l2_scale; -TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) || -kvm_cpu_has (X86_FEATURE_SVM )); -TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL )); -TEST_REQUIRE (sys_clocksource_is_based_on_tsc ()); - -/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ - l1_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;printf ("L1's scale down factor is: %lu\n" , l1_scale_factor); - - l2_scale_factor = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;printf ("L2's scale up factor is: %lu\n" , l2_scale_factor); - tsc_start =rdtsc (); -sleep (1 ); - tsc_end =rdtsc (); - - l0_tsc_freq = tsc_end - tsc_start; -printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); - vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code);sync_global_to_guest (vm, l2_scale_factor);
@@ -183,6 +163,9 @@ int main(int argc, char *argv[])/* scale down L1's TSC frequency */ vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (tsc_khz / l1_scale_factor)); +/* L1 will communicate its frequency before the L2 check.*/ + l1_tsc_freq =0 ; +for (;;) {struct ucall uc;
@@ -222,5 +205,28 @@ int main(int argc, char *argv[]) done:kvm_vm_free (vm); +} + +int main (int argc,char *argv[]) +{ +u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; + +TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) || +kvm_cpu_has (X86_FEATURE_SVM )); +TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL )); +TEST_REQUIRE (sys_clocksource_is_based_on_tsc ()); + + tsc_start =rdtsc (); +sleep (1 ); + tsc_end =rdtsc (); + + l0_tsc_freq = tsc_end - tsc_start; +printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); + +/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ + l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; + l2_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; +test_tsc_scaling (l0_tsc_freq, l1_scale, l2_scale); +return 0 ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 10/21 KVM: selftests: Track L2 multiplier, not scale-up factor, in nested TSC test
In reply to Sean ChristophersonTrack L2's multiplier, not just the non-fractional scale-up factor, so that the nested TSC scaling test can also validate scaling L2's frequency *down*.
No functional change intended.
--- .../kvm/x86/nested_tsc_scaling_test.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +10 -8
@@ -16,8 +16,8 @@#include "svm_util.h" #include "kselftest.h" -/* L2 is scaled up (from L1's perspective) by this factor */ -static u64 l2_scale_factor; +/* L2's TSC multiplier, relative to L1. */ +static u64 l2_multiplier;#define TSC_OFFSET_L2 ((u64)-33125236320908)
@@ -80,7 +80,7 @@ static void l1_svm_code(struct svm_test_data *svm)generic_svm_setup (svm, l2_guest_code);/* enable TSC scaling for L2 */ -wrmsr (MSR_AMD64_TSC_RATIO , l2_scale_factor <<32 ); +wrmsr (MSR_AMD64_TSC_RATIO , l2_multiplier);/* launch L2 */ run_guest (svm->vmcb, svm->vmcb_gpa);
@@ -115,7 +115,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)vmwrite (SECONDARY_VM_EXEC_CONTROL , control);vmwrite (TSC_OFFSET ,TSC_OFFSET_L2 ); -vmwrite (TSC_MULTIPLIER , l2_scale_factor <<48 ); +vmwrite (TSC_MULTIPLIER , l2_multiplier);/* launch L2 */ vmlaunch ();
@@ -135,20 +135,22 @@ static void l1_guest_code(void *data)l1_svm_code (data); } -static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale) +static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) {u64 tsc_khz, l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva; - - l2_scale_factor = l2_scale; +u8 frac_bits;printf ("L1's scale down factor is: %lu\n" , l1_scale_factor);printf ("L2's scale up factor is: %lu\n" , l2_scale_factor); + frac_bits =kvm_cpu_has (X86_FEATURE_VMX ) ?48 :32 ; + l2_multiplier = l2_scale_factor << frac_bits; + vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code); -sync_global_to_guest (vm, l2_scale_factor); +sync_global_to_guest (vm, l2_multiplier);if (kvm_cpu_has (X86_FEATURE_VMX ))vcpu_alloc_vmx (vm, &guest_gva);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 11/21 KVM: selftests: Print out the failing L{0,1,2} level in nested TSC scaling test
In reply to Sean ChristophersonPrint out which one of L0, L1, or L2 TSC frequency checks failed if the nested TSC scaling test detects an out-of-bounds frequency. Triaging test failures without the level information is unnecessarily painful.
--- .../selftests/kvm/x86/nested_tsc_scaling_test.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +5 -5
@@ -32,7 +32,7 @@ enum { USLEEP, UCHECK_L1, UCHECK_L2 };* measurements, a difference of 1% between the actual and the expected value * is tolerated. */ -static void host_check_tsc_freq (u64 actual,u64 expected) +static void host_check_tsc_freq (int level,u64 actual,u64 expected) {u64 tolerance, thresh_low, thresh_high;
@@ -41,8 +41,8 @@ static void host_check_tsc_freq(u64 actual, u64 expected) thresh_high = expected + tolerance;TEST_ASSERT (thresh_low < actual && thresh_high > actual, -"TSC freq is '%lu', expected to be between %lu and %lu" , - actual, thresh_low, thresh_high); +"L%u TSC freq is '%lu', expected to be between %lu and %lu" , + level, actual, thresh_low, thresh_high); }static void guest_check_tsc_freq (int level)
@@ -186,14 +186,14 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ l1_tsc_freq = uc.args[1 ];printf ("L1's TSC frequency is around: %lu\n" , l1_tsc_freq); -host_check_tsc_freq (l1_tsc_freq, +host_check_tsc_freq (1 , l1_tsc_freq, l0_tsc_freq / l1_scale_factor);break ;case UCHECK_L2 : l2_tsc_freq = uc.args[1 ];printf ("L2's TSC frequency is around: %lu\n" , l2_tsc_freq); -host_check_tsc_freq (l2_tsc_freq, +host_check_tsc_freq (2 , l2_tsc_freq, l1_tsc_freq * l2_scale_factor);break ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 12/21 KVM: selftests: Allow +/- 1 tolerance if expected TSC frequency is <100
In reply to Sean ChristophersonMassage the nested TSC scaling test's checker to allow for validating an expected TSC frequency of 1Hz, which is achievable for L2 since L1 can use a completely arbitrary fractional ratio. To do so, set the minimum tolerance to 1, and make the thresholds inclusive, i.e. for an expected frequency of 1, allow the actual frequency to be in the range of [0,2]. Because scaling L2's frequency down to a single Hz makes the virtual TSC run soooo comically slow, and because all of the math is done with basic integer arithmetic, it's very easy for the test to observe what appears to be a frequency of 0.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +2 -2
@@ -36,11 +36,11 @@ static void host_check_tsc_freq(int level, u64 actual, u64 expected) {u64 tolerance, thresh_low, thresh_high; - tolerance = expected /100 ; + tolerance =max (expected /100 , (u64 )1 ); thresh_low = expected - tolerance; thresh_high = expected + tolerance; -TEST_ASSERT (thresh_low < actual && thresh_high > actual, +TEST_ASSERT (thresh_low <= actual && thresh_high >= actual,"L%u TSC freq is '%lu', expected to be between %lu and %lu" , level, actual, thresh_low, thresh_high); }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 13/21 KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
In reply to Sean ChristophersonUse KVM's reported default TSC frequency as L0's frequency instead of calibrating L0's frequency using RDTSC. Trying to calibrate the frequency in userspace is all but guaranteed to be wildly inaccurate, e.g. due to interrupts, preemption, etc, and no known CPU supports TSC scaling without also having a constant TSC. I.e. there's no need to calibrate the current frequency, as it should also be the same as KVM's default frequency.
Opportunistically sanity check KVM's reported frequency against what is observed via RDTSC, e.g. so that KVM_GET_TSC_KHZ or platform issues show up much earlier in the test, before VMs get involved.
--- .../kvm/x86/nested_tsc_scaling_test.c | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +17 -8
@@ -137,7 +137,7 @@ static void l1_guest_code(void *data)static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) { -u64 tsc_khz, l1_tsc_freq, l2_tsc_freq; +u64 l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva;
@@ -159,11 +159,8 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_vcpu_args_set (vcpu,1 , guest_gva); - tsc_khz =__vcpu_ioctl (vcpu,KVM_GET_TSC_KHZ ,NULL ); -TEST_ASSERT (tsc_khz !=-1 ,"vcpu ioctl KVM_GET_TSC_KHZ failed" ); -/* scale down L1's TSC frequency */ -vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (tsc_khz / l1_scale_factor)); +vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (l0_tsc_freq / l1_scale_factor));/* L1 will communicate its frequency before the L2 check.*/ l1_tsc_freq =0 ;
@@ -212,18 +209,30 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +struct kvm_vm *vm;TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) ||kvm_cpu_has (X86_FEATURE_SVM ));TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ()); +/* +* Create a dummy VM to get KVM's default TSC frequency. All CPUs that +* support TSC scaling should have a constant TSC, i.e. there's no need +* to calibrate the "real" TSC. But do sanity check that the observed +* TSC is within range of KVM's reported TSC frequency. +*/ + vm =vm_create_barebones (); + l0_tsc_freq = (u64 )__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ) *1000 ; +TEST_ASSERT (l0_tsc_freq,"vcpu ioctl KVM_GET_TSC_KHZ failed" ); +kvm_vm_free (vm); + +printf ("L0 TSC frequency is: %lu\n" , l0_tsc_freq); + tsc_start =rdtsc ();sleep (1 ); tsc_end =rdtsc (); - - l0_tsc_freq = tsc_end - tsc_start; -printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); +host_check_tsc_freq (0 , tsc_end - tsc_start, l0_tsc_freq);/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] kselftest: random test flakiness due to strict 1% tolerance on sleep(1) - [Low] kselftest…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] kselftest: random test flakiness due to strict 1% tolerance on sleep(1)
- [Low] kselftest: masked ioctl failure check due to u64 cast and multiplication --
commit 44c3625d55f5d8d3232ad8e1eb92f5cad1eaa9ab
Author: Sean Christopherson <[email protected]>
KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
This commit updates the nested TSC scaling test to use KVM's reported default TSC frequency instead of trying to calibrate it in userspace via RDTSC. It also introduces a sanity check to ensure the observed TSC frequency roughly matches KVM's reported frequency.
Quoted 4 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0
[ ... ]
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-0@@ -212,18 +209,30 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +struct kvm_vm *vm;TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) ||kvm_cpu_has (X86_FEATURE_SVM ));TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ());
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 227
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9
-0+/* +* Create a dummy VM to get KVM's default TSC frequency. All CPUs that +* support TSC scaling should have a constant TSC, i.e. there's no need +* to calibrate the "real" TSC. But do sanity check that the observed +* TSC is within range of KVM's reported TSC frequency. +*/ + vm =vm_create_barebones (); + l0_tsc_freq = (u64 )__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ) *1000 ; +TEST_ASSERT (l0_tsc_freq,"vcpu ioctl KVM_GET_TSC_KHZ failed" );
[Severity: Low]
Does this properly catch an ioctl failure? If __vm_ioctl() fails and returns -1, casting it to a u64 and multiplying by 1000 results in a very large non-zero value (0xfffffffffffffc18).
Since TEST_ASSERT only checks if l0_tsc_freq is non-zero, won't this pass using garbage data instead of failing the test?
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 235
Quoted 11 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +4
-3+kvm_vm_free (vm); + +printf ("L0 TSC frequency is: %lu\n" , l0_tsc_freq); tsc_start =rdtsc ();sleep (1 ); tsc_end =rdtsc (); - - l0_tsc_freq = tsc_end - tsc_start; -printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); +host_check_tsc_freq (0 , tsc_end - tsc_start, l0_tsc_freq);
[Severity: Medium]
Could this strict 1% tolerance on sleep(1) introduce random test flakiness on loaded systems?
When comparing the constant hardware frequency (l0_tsc_freq) against the cycle delta measured across a userspace sleep(1), scheduling delays could easily cause sleep to exceed 1.01 seconds.
If the test environment is heavily loaded, won't a 10ms scheduling delay guarantee that the measured cycles exceed the expected constant cycles by more than 1%, failing the host_check_tsc_freq() assertion?
The guest checks in test_tsc_scaling() that call host_check_tsc_freq() also appear to be subject to this same issue since l1_tsc_freq is measured using the guest's sleep(1).
Quoted 3 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Sean Christopherson
maintainerRe: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Quoted 48 lines · On Wed, Sep 30, 2026, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] kselftest: random test flakiness due to strict 1% tolerance on sleep(1)
- [Low] kselftest: masked ioctl failure check due to u64 cast and multiplication --commit
44c3625d55f5d8d3232ad8e1eb92f5cad1eaa9ab
Author: Sean Christopherson <[email protected]>KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
This commit updates the nested TSC scaling test to use KVM's reported default TSC frequency instead of trying to calibrate it in userspace via RDTSC. It also introduces a sanity check to ensure the observed TSC frequency roughly matches KVM's reported frequency.
Quoted 4 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +0
-0[ ... ]
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-0@@ -212,18 +209,30 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +struct kvm_vm *vm;TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) ||kvm_cpu_has (X86_FEATURE_SVM ));TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ());tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 227
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9
-0+/* +* Create a dummy VM to get KVM's default TSC frequency. All CPUs that +* support TSC scaling should have a constant TSC, i.e. there's no need +* to calibrate the "real" TSC. But do sanity check that the observed +* TSC is within range of KVM's reported TSC frequency. +*/ + vm =vm_create_barebones (); + l0_tsc_freq = (u64 )__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ) *1000 ; +TEST_ASSERT (l0_tsc_freq,"vcpu ioctl KVM_GET_TSC_KHZ failed" );[Severity: Low]
Does this properly catch an ioctl failure? If__vm_ioctl()fails and returns -1, casting it to a u64 and multiplying by 1000 results in a very large non-zero value (0xfffffffffffffc18).
Quoted 2 lines
Since
TEST_ASSERTonly checks ifl0_tsc_freqis non-zero, won't this pass using garbage data instead of failing the test?
Eh, yes and no. KVM_GET_TSC_KHZ existed long before TSC scaling, and while KVM doesn't sanity check tsc_khz, i.e. KVM_GET_TSC_KHZ could return a "legitimate" negative frequency (when those ~2.17PHz CPUs come along), in practice checking for a non-zero value is sufficient because any bogus value will quickly fail the functional checks.
Heh, actually, kvm_arch_vm_post_create() *already* validates KVM_GET_TSC_KHZ:
r = __vm_ioctl(vm, KVM_GET_TSC_KHZ, NULL); TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ did not provide a valid TSC frequency."); guest_tsc_khz = r;
so this is a non-issue. I agree it's not the prettiest, I just don't care enough to clean it up.
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +3
-0+kvm_vm_free (vm); + +printf ("L0 TSC frequency is: %lu\n" , l0_tsc_freq);
Quoted 11 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 235
Quoted 7 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-3tsc_start =rdtsc ();sleep (1 ); tsc_end =rdtsc (); - - l0_tsc_freq = tsc_end - tsc_start; -printf ("real TSC frequency is around: %lu\n" , l0_tsc_freq); +host_check_tsc_freq (0 , tsc_end - tsc_start, l0_tsc_freq);[Severity: Medium]
Could this strict 1% tolerance on sleep(1) introduce random test flakiness on loaded systems?
Yes, but that's pre-existing, and no one has complained, yet...
Re: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Quoted 38 lines · Sean Christopherson
Quoted 26 lines · On Wed, Sep 30, 2026, [email protected] wrote:
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-0@@ -212,18 +209,30 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +struct kvm_vm *vm;TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) ||kvm_cpu_has (X86_FEATURE_SVM ));TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ());tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 227
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9
-0+/* +* Create a dummy VM to get KVM's default TSC frequency. All CPUs that +* support TSC scaling should have a constant TSC, i.e. there's no need +* to calibrate the "real" TSC. But do sanity check that the observed +* TSC is within range of KVM's reported TSC frequency. +*/ + vm =vm_create_barebones (); + l0_tsc_freq = (u64 )__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ) *1000 ; +TEST_ASSERT (l0_tsc_freq,"vcpu ioctl KVM_GET_TSC_KHZ failed" );[Severity: Low]
Does this properly catch an ioctl failure? If__vm_ioctl()fails and returns -1, casting it to a u64 and multiplying by 1000 results in a very large non-zero value (0xfffffffffffffc18).Quoted 2 lines
Since
TEST_ASSERTonly checks ifl0_tsc_freqis non-zero, won't this pass using garbage data instead of failing the test?Eh, yes and no.
KVM_GET_TSC_KHZexisted long before TSC scaling, and while KVM doesn't sanity checktsc_khz, i.e.KVM_GET_TSC_KHZcould return a "legitimate" negative frequency (when those ~2.17PHz CPUs come along), in practice checking for a non-zero value is sufficient because any bogus value will quickly fail the functional checks.
Quoted 8 lines
Heh, actually,
kvm_arch_vm_post_create()*already* validatesKVM_GET_TSC_KHZ:r = __vm_ioctl(vm, KVM_GET_TSC_KHZ, NULL); TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ did not provide a valid TSC frequency."); guest_tsc_khz = r;so this is a non-issue. I agree it's not the prettiest, I just don't care enough to clean it up.
In my best Sashiko passive-aggressive tone:
Is it possible that vm_create_barebones() bypasses that check?
Sean Christopherson
maintainerRe: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Quoted 45 lines · Jim Mattson
Quoted 44 lines · Sean Christopherson
Quoted 26 lines · On Wed, Sep 30, 2026, [email protected] wrote:
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-0@@ -212,18 +209,30 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +struct kvm_vm *vm;TEST_REQUIRE (kvm_cpu_has (X86_FEATURE_VMX ) ||kvm_cpu_has (X86_FEATURE_SVM ));TEST_REQUIRE (kvm_has_cap (KVM_CAP_TSC_CONTROL ));TEST_REQUIRE (sys_clocksource_is_based_on_tsc ());tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 227
Quoted 10 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +9
-0+/* +* Create a dummy VM to get KVM's default TSC frequency. All CPUs that +* support TSC scaling should have a constant TSC, i.e. there's no need +* to calibrate the "real" TSC. But do sanity check that the observed +* TSC is within range of KVM's reported TSC frequency. +*/ + vm =vm_create_barebones (); + l0_tsc_freq = (u64 )__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ) *1000 ; +TEST_ASSERT (l0_tsc_freq,"vcpu ioctl KVM_GET_TSC_KHZ failed" );[Severity: Low]
Does this properly catch an ioctl failure? If__vm_ioctl()fails and returns -1, casting it to a u64 and multiplying by 1000 results in a very large non-zero value (0xfffffffffffffc18).Quoted 2 lines
Since
TEST_ASSERTonly checks ifl0_tsc_freqis non-zero, won't this pass using garbage data instead of failing the test?Eh, yes and no.
KVM_GET_TSC_KHZexisted long before TSC scaling, and while KVM doesn't sanity checktsc_khz, i.e.KVM_GET_TSC_KHZcould return a "legitimate" negative frequency (when those ~2.17PHz CPUs come along), in practice checking for a non-zero value is sufficient because any bogus value will quickly fail the functional checks.Heh, actually,
kvm_arch_vm_post_create()*already* validatesKVM_GET_TSC_KHZ:r = __vm_ioctl(vm, KVM_GET_TSC_KHZ, NULL); TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ did not provide a valid TSC frequency."); guest_tsc_khz = r;
Quoted 6 lines
Quoted 2 lines
so this is a non-issue. I agree it's not the prettiest, I just don't care enough to clean it up.
In my best Sashiko passive-aggressive tone:
Is it possible that
vm_create_barebones()bypasses that check?
LOL, *sigh*. I'll send a patch to add a wrapper, having to manually do the assert is annoying.
Re: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
In reply to Sean ChristophersonQuoted 31 lines · Sean Christopherson
Use KVM's reported default TSC frequency as L0's frequency instead of calibrating L0's frequency using RDTSC. Trying to calibrate the frequency in userspace is all but guaranteed to be wildly inaccurate, e.g. due to interrupts, preemption, etc, and no known CPU supports TSC scaling without also having a constant TSC. I.e. there's no need to calibrate the current frequency, as it should also be the same as KVM's default frequency.
Opportunistically sanity check KVM's reported frequency against what is observed via RDTSC, e.g. so that
KVM_GET_TSC_KHZor platform issues show up much earlier in the test, before VMs get involved.Signed-off-by Sean Christopherson <[email protected]>--- .../kvm/x86/nested_tsc_scaling_test.c | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-)tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-1@@ -137,7 +137,7 @@ static void l1_guest_code(void *data)static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) { -u64 tsc_khz, l1_tsc_freq, l2_tsc_freq; +u64 l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva;
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 163
Quoted 9 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-4@@ -159,11 +159,8 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_vcpu_args_set (vcpu,1 , guest_gva); - tsc_khz =__vcpu_ioctl (vcpu,KVM_GET_TSC_KHZ ,NULL ); -TEST_ASSERT (tsc_khz !=-1 ,"vcpu ioctl KVM_GET_TSC_KHZ failed" ); -/* scale down L1's TSC frequency */ -vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (tsc_khz / l1_scale_factor)); +vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (l0_tsc_freq / l1_scale_factor));
Our internal Sashiko complains:
[Severity: Medium]
Is l0_tsc_freq passed to KVM_SET_TSC_KHZ in Hz rather than KHz?
In main(), l0_tsc_freq is calculated in Hz by multiplying the KVM_GET_TSC_KHZ result by 1000. Passing this value divided by l1_scale_factor into KVM_SET_TSC_KHZ configures the guest TSC frequency 1000x larger than intended, since the ioctl expects a value in KHz.
I concur.
Sean Christopherson
maintainerRe: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Quoted 47 lines · Jim Mattson
Quoted 31 lines · Sean Christopherson
Use KVM's reported default TSC frequency as L0's frequency instead of calibrating L0's frequency using RDTSC. Trying to calibrate the frequency in userspace is all but guaranteed to be wildly inaccurate, e.g. due to interrupts, preemption, etc, and no known CPU supports TSC scaling without also having a constant TSC. I.e. there's no need to calibrate the current frequency, as it should also be the same as KVM's default frequency.
Opportunistically sanity check KVM's reported frequency against what is observed via RDTSC, e.g. so that
KVM_GET_TSC_KHZor platform issues show up much earlier in the test, before VMs get involved.Signed-off-by Sean Christopherson <[email protected]>--- .../kvm/x86/nested_tsc_scaling_test.c | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-)tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-1@@ -137,7 +137,7 @@ static void l1_guest_code(void *data)static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) { -u64 tsc_khz, l1_tsc_freq, l2_tsc_freq; +u64 l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva;tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 163
Quoted 9 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-4@@ -159,11 +159,8 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_vcpu_args_set (vcpu,1 , guest_gva); - tsc_khz =__vcpu_ioctl (vcpu,KVM_GET_TSC_KHZ ,NULL ); -TEST_ASSERT (tsc_khz !=-1 ,"vcpu ioctl KVM_GET_TSC_KHZ failed" ); -/* scale down L1's TSC frequency */ -vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (tsc_khz / l1_scale_factor)); +vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (l0_tsc_freq / l1_scale_factor));Our internal Sashiko complains:
[Severity: Medium]
Isl0_tsc_freqpassed toKVM_SET_TSC_KHZin Hz rather than KHz?
Quoted 6 lines
In
main(),l0_tsc_freqis calculated in Hz by multiplying theKVM_GET_TSC_KHZresult by 1000. Passing this value divided byl1_scale_factorintoKVM_SET_TSC_KHZconfigures the guest TSC frequency 1000x larger than intended, since the ioctl expects a value in KHz.I concur.
Drat, I thought I had fixed all of those goofs.
Re: [PATCH v3 13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked)
Quoted 49 lines · Sean Christopherson
Quoted 47 lines · Jim Mattson
Quoted 31 lines · Sean Christopherson
Use KVM's reported default TSC frequency as L0's frequency instead of calibrating L0's frequency using RDTSC. Trying to calibrate the frequency in userspace is all but guaranteed to be wildly inaccurate, e.g. due to interrupts, preemption, etc, and no known CPU supports TSC scaling without also having a constant TSC. I.e. there's no need to calibrate the current frequency, as it should also be the same as KVM's default frequency.
Opportunistically sanity check KVM's reported frequency against what is observed via RDTSC, e.g. so that
KVM_GET_TSC_KHZor platform issues show up much earlier in the test, before VMs get involved.Signed-off-by Sean Christopherson <[email protected]>--- .../kvm/x86/nested_tsc_scaling_test.c | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-)tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-1@@ -137,7 +137,7 @@ static void l1_guest_code(void *data)static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) { -u64 tsc_khz, l1_tsc_freq, l2_tsc_freq; +u64 l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva;tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_ · line 163
Quoted 9 lines
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1
-4@@ -159,11 +159,8 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_vcpu_args_set (vcpu,1 , guest_gva); - tsc_khz =__vcpu_ioctl (vcpu,KVM_GET_TSC_KHZ ,NULL ); -TEST_ASSERT (tsc_khz !=-1 ,"vcpu ioctl KVM_GET_TSC_KHZ failed" ); -/* scale down L1's TSC frequency */ -vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (tsc_khz / l1_scale_factor)); +vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (l0_tsc_freq / l1_scale_factor));Our internal Sashiko complains:
[Severity: Medium]
Isl0_tsc_freqpassed toKVM_SET_TSC_KHZin Hz rather than KHz?
Quoted 8 lines
Quoted 6 lines
In
main(),l0_tsc_freqis calculated in Hz by multiplying theKVM_GET_TSC_KHZresult by 1000. Passing this value divided byl1_scale_factorintoKVM_SET_TSC_KHZconfigures the guest TSC frequency 1000x larger than intended, since the ioctl expects a value in KHz.I concur.
Drat, I thought I had fixed all of those goofs.
Well, the issue is rendered moot by the next patch in the series.
Sean Christopherson
maintainerPATCH v3 14/21 KVM: selftests: Explicitly pass TSC frequencies to guts of TSC scaling test
In reply to Sean ChristophersonExplicitly pass the L1 and L2 TSC frequencies, along with the L2 multiplier, to the main test routine in the nested TSC scaling test. This will allow testing both "scale up" and "scale down" scenarios, and that KVM saturates L2's frequency if it's scale up/down beyond what hardware can support, in which case the expected frequency will not exactly match the L2 multiplier.
--- .../kvm/x86/nested_tsc_scaling_test.c | 42 ++++++++----------- 1 file changed, 17 insertions(+), 25 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +17 -25
@@ -8,7 +8,7 @@* both L1 and L2 are scaled using different ratios. For this test we scale * L1 down and scale L2 up. */ - +#include <linux/math64.h> #include <time.h> #include "kvm_util.h"
@@ -135,21 +135,19 @@ static void l1_guest_code(void *data)l1_svm_code (data); } -static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_scale_factor,u64 l2_scale_factor) +static void test_tsc_scaling (u64 l0_tsc_freq,u64 l1_tsc_freq,u64 l2_tsc_freq, +u64 __l2_multiplier) { -u64 l1_tsc_freq, l2_tsc_freq;struct kvm_vcpu *vcpu;struct kvm_vm *vm;gva_t guest_gva; -u8 frac_bits; -printf ("L1's scale down factor is: %lu\n" , l1_scale_factor); -printf ("L2's scale up factor is: %lu\n" , l2_scale_factor); - - frac_bits =kvm_cpu_has (X86_FEATURE_VMX ) ?48 :32 ; - l2_multiplier = l2_scale_factor << frac_bits; +printf ("Testing L0 freq = %lu, L1 freq = %lu, L2 freq = %lu, L2 mult = 0x%lx\n" , + l0_tsc_freq, l1_tsc_freq, l2_tsc_freq, __l2_multiplier); vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code); + + l2_multiplier = __l2_multiplier;sync_global_to_guest (vm, l2_multiplier);if (kvm_cpu_has (X86_FEATURE_VMX ))
@@ -159,11 +157,7 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_vcpu_args_set (vcpu,1 , guest_gva); -/* scale down L1's TSC frequency */ -vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *) (l0_tsc_freq / l1_scale_factor)); - -/* L1 will communicate its frequency before the L2 check.*/ - l1_tsc_freq =0 ; +vcpu_ioctl (vcpu,KVM_SET_TSC_KHZ , (void *)(l1_tsc_freq /1000 ));for (;;) {struct ucall uc;
@@ -180,18 +174,12 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_sleep (uc.args[1 ]);break ;case UCHECK_L1 : - l1_tsc_freq = uc.args[1 ]; -printf ("L1's TSC frequency is around: %lu\n" , l1_tsc_freq); - -host_check_tsc_freq (1 , l1_tsc_freq, - l0_tsc_freq / l1_scale_factor); +printf ("L1's observed TSC frequency: %lu\n" , uc.args[1 ]); +host_check_tsc_freq (1 , uc.args[1 ], l1_tsc_freq);break ;case UCHECK_L2 : - l2_tsc_freq = uc.args[1 ]; -printf ("L2's TSC frequency is around: %lu\n" , l2_tsc_freq); - -host_check_tsc_freq (2 , l2_tsc_freq, - l1_tsc_freq * l2_scale_factor); +printf ("L2's observed TSC frequency: %lu\n" , uc.args[1 ]); +host_check_tsc_freq (2 , uc.args[1 ], l2_tsc_freq);break ; }break ;
@@ -209,6 +197,7 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_scale_factor, u64 l2_scale_int main (int argc,char *argv[]) {u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale; +u8 frac_bits =kvm_cpu_has (X86_FEATURE_VMX ) ?48 :32 ;struct kvm_vm *vm; TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) ||
@@ -237,7 +226,10 @@ int main(int argc, char *argv[])/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; l2_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; -test_tsc_scaling (l0_tsc_freq, l1_scale, l2_scale); + +test_tsc_scaling (l0_tsc_freq, l0_tsc_freq / l1_scale, +mul_u64_u64_div64 (l0_tsc_freq, l2_scale, l1_scale), + (l2_scale << frac_bits));return 0 ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 15/21 KVM: selftests: Sanity check KVM's default TSC freq in nested TSC scaling test
In reply to Sean ChristophersonExplicitly verify that both L1 and L2 can run at KVM's default TSC frequency reported via KVM_GET_TSC_KHZ, e.g. that KVM_SET_TSC_KHZ can take in what KVM_GET_TSC_KHZ spits out.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 3 +++ 1 file changed, 3 insertions(+)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +3 -0
@@ -223,6 +223,9 @@ int main(int argc, char *argv[]) tsc_end =rdtsc ();host_check_tsc_freq (0 , tsc_end - tsc_start, l0_tsc_freq); +/* Sanity check the frequency reported by KVM_GET_TSC_KHZ. */ +test_tsc_scaling (l0_tsc_freq, l0_tsc_freq, l0_tsc_freq,BIT_ULL (frac_bits)); +/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; l2_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 16/21 KVM: selftests: Test L1 "up" and L2 "down" in nested TSC scaling test
In reply to Sean ChristophersonNow that the guts of the nested TSC scaling test support scaling L2 down, validate all four combinations of scaling up/down L1 and L2. Ideally, the test would also validate integer+fractional scale up scenarios, but prioritize keeping the test and math somewhat understandable.
Opportunistically drop the stale file comment about the test's name.
--- .../kvm/x86/nested_tsc_scaling_test.c | 33 ++++++++++++++++--- 1 file changed, 28 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +28 -5
@@ -1,12 +1,9 @@// SPDX-License-Identifier: GPL-2.0-only /* -* vmx_nested_tsc_scaling_test -* * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * This test case verifies that nested TSC scaling behaves as expected when -* both L1 and L2 are scaled using different ratios. For this test we scale -* L1 down and scale L2 up. +* both L1 and L2 are scaled using different ratios. */ #include <linux/math64.h> #include <time.h>
@@ -226,7 +223,21 @@ int main(int argc, char *argv[])/* Sanity check the frequency reported by KVM_GET_TSC_KHZ. */ test_tsc_scaling (l0_tsc_freq, l0_tsc_freq, l0_tsc_freq,BIT_ULL (frac_bits)); -/* Scale L1 "down" and L2 "up" at a random factor from 2 to 10. */ +/* +* Scale L1 and L2 up and down at random factors from 2 to 10. Current +* CPUs have two full orders of magnitude of "breathing room" before an +* ultrafast L0 TSC frequency multiplied by 10x will encounter KVM's +* signed 32-bit limit on L1's frequency, and it's highly unlikely a +* CPU that supports TSC scaling will show up running at 100MHz, i.e. +* underflowing KVM's minimum 1KHz frequency is extremely unlikely. +* +* For L2, the frequency is limited only by what hardware can support, +* not by KVM's limits. SVM provides 8 bits of integer scale up, and +* 32 bits of fractional scale down, i.e. can scale up 255x and down a +* comical amount, so scaling up 100x and down 1/100 is well within +* hardware's capabilities (VMX provides 16 bits of "up" and 48 bits of +* "down"). +*/ l1_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ; l2_scale = (kvm_random_u32 (&kvm_rng) %9 ) +2 ;
@@ -234,5 +245,17 @@ int main(int argc, char *argv[])mul_u64_u64_div64 (l0_tsc_freq, l2_scale, l1_scale), (l2_scale << frac_bits)); +test_tsc_scaling (l0_tsc_freq, l0_tsc_freq * l1_scale, +mul_u64_u64_div64 (l0_tsc_freq, l1_scale, l2_scale), + (1ull << frac_bits) / l2_scale); + +test_tsc_scaling (l0_tsc_freq, l0_tsc_freq / l1_scale, + l0_tsc_freq / l1_scale / l2_scale, + (1ull << frac_bits) / l2_scale); + +test_tsc_scaling (l0_tsc_freq, l0_tsc_freq * l1_scale, + l0_tsc_freq * l1_scale * l2_scale, + (l2_scale << frac_bits)); +return 0 ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 17/21 KVM: selftests: Verify that KVM saturates L2 TSC freq on {under,over}flow
In reply to Sean ChristophersonExtend the nested TSC scaling test to validate the extreme ends of what hardware supports, i.e. validate that KVM allows L1 to scale L2's virtual TSC frequency up/down to the architectural maximum/minimum, from L1's perspective, and that KVM saturates L2's effective frequency when L2's frequency can't be virtualized by hardware (which is possible when stacking multipliers/ratios across L1 and L2).
--- .../kvm/x86/nested_tsc_scaling_test.c | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +35 -0
@@ -193,6 +193,7 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_tsc_freq, u64 l2_tsc_freq,int main (int argc,char *argv[]) { +u64 min_freq, min_multiplier, max_freq, max_multiplier, l1_max_freq, l1_min_freq;u64 l0_tsc_freq, tsc_start, tsc_end, l1_scale, l2_scale;u8 frac_bits =kvm_cpu_has (X86_FEATURE_VMX ) ?48 :32 ;struct kvm_vm *vm;
@@ -257,5 +258,39 @@ int main(int argc, char *argv[]) l0_tsc_freq * l1_scale * l2_scale, (l2_scale << frac_bits)); +/* +* Test that KVM saturates L2's frequency on both ends if the resulting +* L2 TSC frequency would be below or above what hardware can support. +* Because L2 = L0 * (L1_mult >> frac) * (L2_mult >> frac) needs to be +* distilled down to a single multiplier, very small/large multipliers +* will underflow/overflow the minimum/maximum multiplier supported by +* hardware when L1 and L2 multipliers are combined. KVM's behavior is +* saturate on {under,over}flow, i.e. to run at the min/max frequency. +* +* Note, userspace can only program L1's frequency in KHz, i.e. can't +* specify an exact multiplier. As a result, the minimum and maximum +* frequencies are different for L1 vs L2, because L1 is constrained by +* hardware *and* KVM, whereas L2 is constrained only by hardware. +*/ + min_multiplier =1 ; + min_freq =mul_u64_u64_div64 (l0_tsc_freq, min_multiplier,BIT_ULL (frac_bits)); + min_freq =max (min_freq, (u64 )1 ); + l1_min_freq =max (min_freq, (u64 )1 *1000 ); +test_tsc_scaling (l0_tsc_freq, l1_min_freq, min_freq,1 ); + +/* +* SVM takes a 40-bit value (right shifted by 32), while VMX takes a +* 64-bit value (right shifted by 48). mul_u64_u64_shr() isn't (yet) +* available in selftests, but mul_u64_u64_div64() does nicely since, +* albeit more slowly (performance is obviously not a concern). Note, +* because KVM_GET_TSC_KHZ returns a signed 32-bit integer, KVM limits +* KVM_SET_TSC_KHZ to INT_MAX, even though hardware (both SVM and VMX) +* supports much higher frequencies. +*/ + max_multiplier =kvm_cpu_has (X86_FEATURE_VMX ) ?-1ull :GENMASK_U64 (39 ,0 ); + max_freq =mul_u64_u64_div64 (l0_tsc_freq, max_multiplier,BIT_ULL (frac_bits)); + l1_max_freq =min (max_freq, (u64 )INT32_MAX *1000 ); +test_tsc_scaling (l0_tsc_freq, l1_max_freq, max_freq, max_multiplier); +return 0 ; }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 18/21 KVM: selftests: Test non-zero TSC offset on SVM in nested TSC scaling test
In reply to Sean ChristophersonSet L2's TSC offset for SVM as well as VMX in the nested TSC scaling test. Odds are good skipping SVM was pure oversight.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 1 + 1 file changed, 1 insertion(+)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +1 -0
@@ -75,6 +75,7 @@ static void l1_svm_code(struct svm_test_data *svm)guest_check_tsc_freq (UCHECK_L1 );generic_svm_setup (svm, l2_guest_code); + svm->vmcb->control.tsc_offset =TSC_OFFSET_L2 ;/* enable TSC scaling for L2 */ wrmsr (MSR_AMD64_TSC_RATIO , l2_multiplier);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 19/21 KVM: selftests: Randomize L2's TSC offset in the nested TSC scaling test
In reply to Sean ChristophersonRandomize L2's TSC offset in the nested TSC scaling test, not because it's all that interesting, but because hardcoding a copmletely arbitrary value is all kinds of silly.
--- .../selftests/kvm/x86/nested_tsc_scaling_test.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +7 -5
@@ -13,10 +13,9 @@#include "svm_util.h" #include "kselftest.h" -/* L2's TSC multiplier, relative to L1. */ +/* L2's TSC multiplier and offset, relative to L1. */ static u64 l2_multiplier; - -#define TSC_OFFSET_L2 ((u64)-33125236320908) +static u64 l2_offset;enum {USLEEP ,UCHECK_L1 ,UCHECK_L2 };#define GUEST_SLEEP (sec) ucall(UCALL_SYNC, 2, USLEEP, sec)
@@ -75,7 +74,7 @@ static void l1_svm_code(struct svm_test_data *svm)guest_check_tsc_freq (UCHECK_L1 );generic_svm_setup (svm, l2_guest_code); - svm->vmcb->control.tsc_offset =TSC_OFFSET_L2 ; + svm->vmcb->control.tsc_offset = l2_offset;/* enable TSC scaling for L2 */ wrmsr (MSR_AMD64_TSC_RATIO , l2_multiplier);
@@ -112,7 +111,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages) control |=SECONDARY_EXEC_TSC_SCALING ;vmwrite (SECONDARY_VM_EXEC_CONTROL , control); -vmwrite (TSC_OFFSET ,TSC_OFFSET_L2 ); +vmwrite (TSC_OFFSET , l2_offset);vmwrite (TSC_MULTIPLIER , l2_multiplier);/* launch L2 */
@@ -145,6 +144,9 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_tsc_freq, u64 l2_tsc_freq, vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code); + l2_offset =kvm_random_u64 (&kvm_rng); +sync_global_to_guest (vm, l2_offset); + l2_multiplier = __l2_multiplier;sync_global_to_guest (vm, l2_multiplier);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 20/21 KVM: selftests: Use GUEST_SYNC2() in nested TSC scaling test
In reply to Sean ChristophersonUse GUEST_SYNC2() instead of open coding equivalent functionality using raw ucall() invocations in the nested TSC scaling test.
No functional change intended.
--- tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +2 -5
@@ -18,9 +18,6 @@ static u64 l2_multiplier;static u64 l2_offset;enum {USLEEP ,UCHECK_L1 ,UCHECK_L2 }; -#define GUEST_SLEEP (sec) ucall(UCALL_SYNC, 2, USLEEP, sec) -#define GUEST_CHECK (level, freq) ucall(UCALL_SYNC, 2, level, freq) -/* * This function checks whether the "actual" TSC frequency of a guest matches
@@ -52,12 +49,12 @@ static void guest_check_tsc_freq(int level)* be good enough for the purposes of this test. */ tsc_start =rdmsr (MSR_IA32_TSC ); -GUEST_SLEEP (1 ); +GUEST_SYNC2 (USLEEP ,1 ); tsc_end =rdmsr (MSR_IA32_TSC ); tsc_freq = tsc_end - tsc_start; -GUEST_CHECK (level, tsc_freq); +GUEST_SYNC2 (level, tsc_freq); }static void l2_guest_code (void )
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerPATCH v3 21/21 KVM: selftests: Spell out UCALL in nested TSC scaling test's enums
In reply to Sean ChristophersonExplicitly spell out UCALL instead of prefixing only U in the nested TSC scaling test's ucall commands, as it's all too easy to think USLEEP means "do usleep()", when it actually means "do sleep() in userspace".
Opportunistically name the enum and take an enum instead of a bare integer for the to-be-checked level, to make it more obvious that the fact that the level values happen to correspond to '1' and '2' for the nesting level is largely coincidental (or perhaps it was intentional, but it's not a hard requirement for the test).
--- .../kvm/x86/nested_tsc_scaling_test.c | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-)
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +12 -12
@@ -17,7 +17,7 @@static u64 l2_multiplier;static u64 l2_offset; -enum {USLEEP ,UCHECK_L1 ,UCHECK_L2 }; +enum ucall_cmd {UCALL_SLEEP ,UCALL_CHECK_L1 ,UCALL_CHECK_L2 };/* * This function checks whether the "actual" TSC frequency of a guest matches
@@ -38,7 +38,7 @@ static void host_check_tsc_freq(int level, u64 actual, u64 expected) level, actual, thresh_low, thresh_high); } -static void guest_check_tsc_freq (int level) +static void guest_check_tsc_freq (enum ucall_cmd check_level) {u64 tsc_start, tsc_end, tsc_freq;
@@ -49,17 +49,17 @@ static void guest_check_tsc_freq(int level)* be good enough for the purposes of this test. */ tsc_start =rdmsr (MSR_IA32_TSC ); -GUEST_SYNC2 (USLEEP ,1 ); +GUEST_SYNC2 (UCALL_SLEEP ,1 ); tsc_end =rdmsr (MSR_IA32_TSC ); tsc_freq = tsc_end - tsc_start; -GUEST_SYNC2 (level, tsc_freq); +GUEST_SYNC2 (check_level, tsc_freq); }static void l2_guest_code (void ) { -guest_check_tsc_freq (UCHECK_L2 ); +guest_check_tsc_freq (UCALL_CHECK_L2 );/* exit to L1 */ __asm__ __volatile__("vmcall" );
@@ -68,7 +68,7 @@ static void l2_guest_code(void)static void l1_svm_code (struct svm_test_data *svm) {/* check that L1's frequency looks alright before launching L2 */ -guest_check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCALL_CHECK_L1 );generic_svm_setup (svm, l2_guest_code); svm->vmcb->control.tsc_offset = l2_offset;
@@ -81,7 +81,7 @@ static void l1_svm_code(struct svm_test_data *svm)GUEST_ASSERT (svm->vmcb->control.exit_code ==SVM_EXIT_VMMCALL );/* check that L1's frequency still looks good */ -guest_check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCALL_CHECK_L1 );GUEST_DONE (); }
@@ -91,7 +91,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)u32 control;/* check that L1's frequency looks alright before launching L2 */ -guest_check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCALL_CHECK_L1 );prepare_for_vmx_operation (vmx_pages);load_vmcs (vmx_pages);
@@ -116,7 +116,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages)GUEST_ASSERT (vmread (VM_EXIT_REASON ) ==EXIT_REASON_VMCALL );/* check that L1's frequency still looks good */ -guest_check_tsc_freq (UCHECK_L1 ); +guest_check_tsc_freq (UCALL_CHECK_L1 );GUEST_DONE (); }
@@ -167,14 +167,14 @@ static void test_tsc_scaling(u64 l0_tsc_freq, u64 l1_tsc_freq, u64 l2_tsc_freq,REPORT_GUEST_ASSERT (uc);case UCALL_SYNC :switch (uc.args[0 ]) { -case USLEEP : +case UCALL_SLEEP :sleep (uc.args[1 ]);break ; -case UCHECK_L1 : +case UCALL_CHECK_L1 :printf ("L1's observed TSC frequency: %lu\n" , uc.args[1 ]);host_check_tsc_freq (1 , uc.args[1 ], l1_tsc_freq);break ; -case UCHECK_L2 : +case UCALL_CHECK_L2 :printf ("L2's observed TSC frequency: %lu\n" , uc.args[1 ]);host_check_tsc_freq (2 , uc.args[1 ], l2_tsc_freq);break ;
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Sean Christopherson
maintainerQuoted 8 lines · Sean Christopherson
Amirmohammad's fix to saturate L2's TSC frequency when stacking scale up/down ratios across L1+L2 yields a final scaling ratio that can't be plugged into hardware. The actual fix is functionally the same as v2, but I rewrote the comments and changelog to try and make them more approachable to KVM developers (AI has a habit of generating extremely verbose, "mathy" descriptions of bugs and fixes).
[...]
Applied to kvm-x86 selftests. Yes, shoving KVM commits into the selftests branch is awful, but the KVM changes are tiny in comparison, I don't want to deal with the massive nVMX selftests conflicts, and this is at the very end of the branch (I don't anticipate pulling anything else in at this point).
[01/21] KVM: x86: Saturate L2's TSC frequency if it exceeds hardware supports
https://github.com/kvm-x86/linux/commit/b701a8139fd6 [02/21] KVM: SVM: Fallback to the default TSC ratio if KVM tries to use a bad multiplier https://github.com/kvm-x86/linux/commit/967e40fb1b08 [03/21] KVM: x86: Allow userspace to set KVM's max supported guest TSC frequency https://github.com/kvm-x86/linux/commit/95eaeb7f202b [04/21] KVM: selftests: Use KVM's pRNG to randomize L1's TSC ratio in TSC scaling test https://github.com/kvm-x86/linux/commit/ed95a2725719 [05/21] KVM: selftests: Drop redundant VMWRITE of TSC_MULTIPLIER_HIGH https://github.com/kvm-x86/linux/commit/b955d8fdabfe [06/21] KVM: selftests: Drop unnecessary use of PRIu64 in nested TSC scaling test https://github.com/kvm-x86/linux/commit/17919fe61684 [07/21] KVM: selftests: Randomize L2's scale factor in nested TSC scaling test https://github.com/kvm-x86/linux/commit/19d963ed9a40 [08/21] KVM: selftests: Rename TSC freq checkers in nested TSC scaling test https://github.com/kvm-x86/linux/commit/771902aecbb4 [09/21] KVM: selftests: Extract guts of nested TSC scaling test to helper function https://github.com/kvm-x86/linux/commit/6ab38c295d06 [10/21] KVM: selftests: Track L2 multiplier, not scale-up factor, in nested TSC test https://github.com/kvm-x86/linux/commit/ff18d56a08b2 [11/21] KVM: selftests: Print out the failing L{0,1,2} level in nested TSC scaling test https://github.com/kvm-x86/linux/commit/df1613249f6f [12/21] KVM: selftests: Allow +/- 1 tolerance if expected TSC frequency is <100 https://github.com/kvm-x86/linux/commit/faa4f010187c [13/21] KVM: selftests: Use KVM's reported TSC KHz as L0's frequency (sanity checked) https://github.com/kvm-x86/linux/commit/43f58b1ee6fe [14/21] KVM: selftests: Explicitly pass TSC frequencies to guts of TSC scaling test https://github.com/kvm-x86/linux/commit/b77a34ac9d8f [15/21] KVM: selftests: Sanity check KVM's default TSC freq in nested TSC scaling test https://github.com/kvm-x86/linux/commit/6760d12380a4 [16/21] KVM: selftests: Test L1 "up" and L2 "down" in nested TSC scaling test https://github.com/kvm-x86/linux/commit/245932d3880c [17/21] KVM: selftests: Verify that KVM saturates L2 TSC freq on {under,over}flow https://github.com/kvm-x86/linux/commit/c2d3a10b90b1 [18/21] KVM: selftests: Test non-zero TSC offset on SVM in nested TSC scaling test https://github.com/kvm-x86/linux/commit/baada72b9c01 [19/21] KVM: selftests: Randomize L2's TSC offset in the nested TSC scaling test https://github.com/kvm-x86/linux/commit/5f13a511c782 [20/21] KVM: selftests: Use GUEST_SYNC2() in nested TSC scaling test https://github.com/kvm-x86/linux/commit/a744d1c1655d [21/21] KVM: selftests: Spell out UCALL in nested TSC scaling test's enums https://github.com/kvm-x86/linux/commit/bef953c06980