KVM: x86: Make bus clock frequency for vAPIC timer configurable
v9 cover KVM: x86: Make bus clock frequency for vAPIC timer configurable
Changes from v8:
- v8: https://lore.kernel.org/lkml/[email protected]/
- Many changes to new udelay() utility patch as well as the APIC bus frequency test aimed to make it more robust (additional ASSERTs, consistent types, eliminate duplicate code, etc.) and useful with support for more user configuration. Please refer to individual patches for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEAD e4e9e1067138e5620cf0500c3e5f6ebfb9d322c8.
Changes from v7:
- v7: https://lore.kernel.org/lkml/[email protected]/
- Use appropriate ioctl() return type.
- Let new tsc_khz used by KVM selftest be of the same type as same variable used in kernel.
- Please refer to individual patches for detailed changes.
Changes from v6:
- V6: https://lore.kernel.org/lkml/[email protected]/
- KVM changes were merged into kvm-x86 misc and subsequently dropped from this submission. This submission only contains the selftest changes.
https://lore.kernel.org/lkml/[email protected]/
- New patch to add udelay() utility to x86_64 KVM selftests. (Sean)
- Many changes to APIC bus frequency selftest. Please refer to the patch for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEAD af0903ab52ee6d6f0f63af67fa73d5eb00f79b9a.
Changes from v5:
- v5: https://lore.kernel.org/lkml/[email protected]/
- Rebased on latest of "next" branch of https://github.com/kvm-x86/linux.git
- Added Xiaoyao Li and Yuan Yao's Reviewed-by tags.
- Use the KVM selftest vm_create() wrapper instead of open coding it. (Zide)
- Improve grammar of selftest description. (Zide)
Changes from v4:
- v4: https://lore.kernel.org/lkml/[email protected]/
- Rename capability from KVM_CAP_X86_APIC_BUS_FREQUENCY to KVM_CAP_X86_APIC_BUS_CYCLES_NS. (Xiaoyao Li).
- Include "Testing" section in cover letter.
- Add Rick's Reviewed-by tags.
- Rebased on latest of "next" branch of https://github.com/kvm-x86/linux.git
Changes from v3:
- v3: https://lore.kernel.org/all/[email protected]/
- Reworked all changelogs.
- Regarding code changes: patches #1 and #2 are unchanged, patch #3 was reworked according to Sean's guidance, and patch #4 (the test) needed changes after rebase to kvm-x86/next and the implementation (patch #3) changes.
- Added Reviewed-by tags to patches #1, #2, and #4, but removed Maxim's Reviewed-by tag from patch #3 because it changed so much.
- Added a "Suggested-by" to patch #4 to reflect that it represents Sean's guidance.
- Reworked cover to match customs (in subject line) and reflect feedback to patches: capability renamed to KVM_CAP_X86_APIC_BUS_FREQUENCY, clarify distinction between "core crystal clock" and "bus clock", and update pro/con list.
- Please refer to individual patches for detailed changes.
Changes from v2:
- v2: https://lore.kernel.org/lkml/[email protected]/
- Removed APIC_BUS_FREQUENCY and apic_bus_frequency of struct kvm-arch.
- Update the commit messages.
- Added reviewed-by (Maxim Levitsky)
- Use 1.5GHz instead of 1GHz as frequency for the test case.
Changes from v1:
- v1: https://lore.kernel.org/all/[email protected]/
- Added a test case
- Fix a build error for i386 platform
- Add check if vcpu isn't created.
- Add check if lapic chip is in-kernel emulation.
- Updated api.rst
Summary
-------
Add KVM_CAP_X86_APIC_BUS_CYCLES_NS capability to configure the APIC bus clock frequency for APIC timer emulation.
Allow KVM_ENABLE_CAPABILITY(KVM_CAP_X86_APIC_BUS_CYCLES_NS) to set the frequency in nanoseconds. When using this capability, the user space VMM should configure CPUID leaf 0x15 to advertise the frequency.
Description
-----------
Vishal reported [1] that the TDX guest kernel expects a 25MHz APIC bus frequency but ends up getting interrupts at a significantly higher rate.
The TDX architecture hard-codes the core crystal clock frequency to 25MHz and mandates exposing it via CPUID leaf 0x15. The TDX architecture does not allow the VMM to override the value.
In addition, per Intel SDM:
"The APIC timer frequency will be the processor’s bus clock or core
crystal clock frequency (when TSC/core crystal clock ratio is enumerated in CPUID leaf 0x15) divided by the value specified in the divide configuration register."
The resulting 25MHz APIC bus frequency conflicts with the KVM hardcoded APIC bus frequency of 1GHz.
The KVM doesn't enumerate CPUID leaf 0x15 to the guest unless the user space VMM sets it using KVM_SET_CPUID. If the CPUID leaf 0x15 is enumerated, the guest kernel uses it as the APIC bus frequency. If not, the guest kernel measures the frequency based on other known timers like the ACPI timer or the legacy PIT. As reported in [1] the TDX guest kernel expects a 25MHz timer frequency but gets timer interrupt more frequently due to the 1GHz frequency used by KVM.
To ensure that the guest doesn't have a conflicting view of the APIC bus frequency, allow the userspace to tell KVM to use the same frequency that TDX mandates instead of the default 1Ghz.
There are several options to address this:
1. Make the KVM able to configure APIC bus frequency (this series).
Pro: It resembles the existing hardware. The recent Intel CPUs
adopts 25MHz.
Con: Require the VMM to emulate the APIC timer at 25MHz.
2. Make the TDX architecture enumerate CPUID leaf 0x15 to configurable frequency or not enumerate it.
Pro: Any APIC bus frequency is allowed.
Con: Deviates from TDX architecture.
3. Make the TDX guest kernel use 1GHz when it's running on KVM.
Con: The kernel ignores CPUID leaf 0x15.
4. Change CPUID leaf 0x15 under TDX to report the crystal clock frequency as 1 GHz.
Pro: This has been the virtual APIC frequency for KVM guests for 13
years.
Pro: This requires changing only one hard-coded constant in TDX.
Con: It doesn't work with other VMMs as TDX isn't specific to KVM.
Con: Core crystal clock frequency is also used to calculate TSC frequency.
Con: If it is configured to value different from hardware, it will
break the correctness of INTEL-PT Mini Time Count (MTC) packets in TDs.
Testing
-------
Tested on non-TDX host using included kselftest. Host running kernel with this series applied to "next" branch of
https://github.com/kvm-x86/linux.git
Tested on TDX host and TD using a modified version
of x86/apic.c:test_apic_timer_one_shot() available from
https://github.com/intel/kvm-unit-tests-tdx/blob/tdx/x86/apic.c.
Host running TDX KVM development patches and QEMU with corresponding TDX changes.
The test needed to be modified to (a) stop any lingering timers before the test starts, and (b) use CPUID 0x15 in TDX to accurately determine the TSC and APIC frequencies instead of making 1GHz assumption and use similar check as the kselftest test introduced in this series (while increasing the amount with which the frequency is allowed to deviate by 1%).
The core changes made to x86/apic.c:test_apic_timer_one_shot() for this testing are shown below for reference. Work is in progress to upstream these modifications.
@@ -477,11 +478,29 @@ static void lvtt_handler(isr_regs_t *regs)
static void test_apic_timer_one_shot(void)
{
- uint64_t tsc1, tsc2;
static const uint32_t interval = 0x10000;
+ uint64_t measured_apic_freq, tsc2, tsc1;
+ uint32_t tsc_freq = 0, apic_freq = 0;
+ struct cpuid cpuid_tsc = {};
#define APIC_LVT_TIMER_VECTOR (0xee)
+ /*
+ * CPUID 0x15 is not available in VMX, can use it to obtain
+ * TSC and APIC frequency for accurate testing
+ */
+ if (is_tdx_guest()) {
+ cpuid_tsc = raw_cpuid(0x15, 0);
+ tsc_freq = cpuid_tsc.c * cpuid_tsc.b / cpuid_tsc.a;
+ apic_freq = cpuid_tsc.c;
+ }
+ /*
+ stop already fired local timer + the test case can be negative failure if the timer fired + after installed lvtt_handler but *before* + write to TIMICT again.
+ */
+ apic_write(APIC_TMICT, 0);
handle_irq(APIC_LVT_TIMER_VECTOR, lvtt_handler);
/* One shot mode */
@@ -503,8 +522,16 @@ static void test_apic_timer_one_shot(void)
* cases, the following should satisfy on all modern
* processors.
*/
- report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
- "APIC LVT timer one shot");
+ if (is_tdx_guest()) {
+ measured_apic_freq = interval * (tsc_freq / (tsc2 - tsc1));
+ report((lvtt_counter == 1) &&
+ (measured_apic_freq < apic_freq * 102 / 100) && + (measured_apic_freq > apic_freq * 98 / 100), + "APIC LVT timer one shot");
+ } else {
+ report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
+ "APIC LVT timer one shot");
+ }
}
[1] https://lore.kernel.org/lkml/[email protected]/
Isaku Yamahata (1):
KVM: selftests: Add test for configure of x86 APIC bus frequency
Reinette Chatre (1):
KVM: selftests: Add x86_64 guest udelay() utility
tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/apic.h | 8 + .../selftests/kvm/include/x86_64/processor.h | 17 ++ .../selftests/kvm/lib/x86_64/processor.c | 11 + .../kvm/x86_64/apic_bus_clock_test.c | 202 ++++++++++++++++++ 5 files changed, 239 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c
base-commit: e4e9e1067138e5620cf0500c3e5f6ebfb9d322c8
Signature · 1 line
2.34.1
PATCH v9 1/2 KVM: selftests: Add x86_64 guest udelay() utility
Create udelay() utility for x86_64 tests to matchudelay() available to ARM and RISC-V tests.
Calibrate guest frequency using the KVM_GET_TSC_KHZ ioctl() and share it between host and guest with the new tsc_khz global variable.
---
Changes v9:
- Change tsc_kz type to uint32_t (was unsigned int). (Sean)
- Change type used to store number of cycles to uint64_t (was unsigned long). (Sean)
- Add GUEST_ASSERT() in udelay() used in guest to avoid risk of udelay() unexpectedly doing nothing if something went wrong during TSC frequency discovery. (Sean)
- Use TEST_ASSERT() when checking for KVM_CAP_GET_TSC_KHZ. (Sean)
- Use TEST_ASSERT() to enforce that discovery of TSC frequency must never fail. (Sean)
Changes v8:
- Use appropriate signed type to discover TSC freq from KVM.
- Switch type used to store TSC frequency from unsigned long to unsigned int to match the type used by the kernel.
Changes v7:
- New patch
--- .../selftests/kvm/include/x86_64/processor.h | 17 +++++++++++++++++ .../selftests/kvm/lib/x86_64/processor.c | 11 +++++++++++ 2 files changed, 28 insertions(+)
tools/testing/selftests/kvm/include/x86_64/processor.h +17 -0
@@ -23,6 +23,7 @@extern bool host_cpu_is_intel;extern bool host_cpu_is_amd; +extern uint32_t tsc_khz;/* Forced emulation prefix, used to invoke the emulator unconditionally. */ #define KVM_FEP "ud2; .byte 'k', 'v', 'm';"
@@ -816,6 +817,22 @@ static inline void cpu_relax(void) asmvolatile ("rep; nop" :::"memory" ); } +static inline void udelay (unsigned long usec) +{ +uint64_t start, now, cycles; + +GUEST_ASSERT (tsc_khz); + cycles = tsc_khz /1000 * usec; + + start =rdtsc (); +for (;;) { + now =rdtsc (); +if (now - start >= cycles) +break ; +cpu_relax (); + } +} +#define ud2 () \ __asm__ __volatile__( \ "ud2\n" \
tools/testing/selftests/kvm/lib/x86_64/processor.c +11 -0
@@ -25,6 +25,7 @@ vm_vaddr_t exception_handlers;bool host_cpu_is_amd;bool host_cpu_is_intel;bool is_forced_emulation_enabled; +uint32_t tsc_khz;static void regs_dump (FILE *stream,struct kvm_regs *regs,uint8_t indent) {
@@ -616,6 +617,11 @@ void assert_on_unhandled_exception(struct kvm_vcpu *vcpu)void kvm_arch_vm_post_create (struct kvm_vm *vm) { +int r; + +TEST_ASSERT (kvm_has_cap (KVM_CAP_GET_TSC_KHZ ), +"Require KVM_GET_TSC_KHZ to provide udelay() to guest." ); +vm_create_irqchip (vm);vm_init_descriptor_tables (vm);
@@ -628,6 +634,11 @@ void kvm_arch_vm_post_create(struct kvm_vm *vm)vm_sev_ioctl (vm,KVM_SEV_INIT2 , &init); } + + r =__vm_ioctl (vm,KVM_GET_TSC_KHZ ,NULL ); +TEST_ASSERT (r >0 ,"KVM_GET_TSC_KHZ did not provide a valid TSC frequency." ); + tsc_khz = r; +sync_global_to_guest (vm, tsc_khz); }void vcpu_arch_set_entry_point (struct kvm_vcpu *vcpu,void *guest_code)
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH V9 1/2] KVM: selftests: Add x86_64 guest udelay() utility
Quoted 10 lines · Reinette Chatre
--- .../selftests/kvm/include/x86_64/processor.h | 17 +++++++++++++++++ .../selftests/kvm/lib/x86_64/processor.c | 11 +++++++++++ 2 files changed, 28 insertions(+)tools/testing/selftests/kvm/include/x86_64/processor.h +0
-0
tools/testing/selftests/kvm/include/x86_64/processor.h line 26
Quoted 4 lines
tools/testing/selftests/kvm/include/x86_64/processor.h +1
-0@@ -23,6 +23,7 @@extern bool host_cpu_is_intel;extern bool host_cpu_is_amd; +extern uint32_t tsc_khz;
This should be guest_tsc_khz, because it most definitely isn't guaranteed to be the host TSC frequency. And because it's global, we should try to avoid variable shadowing, e.g. tsc_scaling_sync.c also defines tsc_khz.
Which, by the by, probably needs to be addressed, i.e. we should probably add a helper for setting KVM_SET_TSC_KHZ+guest_tsc_khz.
I think it also makes sense to have this be a 64-bit value, even though KVM *internally* tracks a 32-bit value. That way we don't have to worry about casting to avoid truncation.
Quoted 9 lines
tools/testing/selftests/kvm/include/x86_64/processor.h +3
-0/* Forced emulation prefix, used to invoke the emulator unconditionally. */ #define KVM_FEP "ud2; .byte 'k', 'v', 'm';"@@ -816,6 +817,22 @@ static inline void cpu_relax(void) asmvolatile ("rep; nop" :::"memory" ); } +static inline void udelay (unsigned long usec) +{ +uint64_t start, now, cycles;
tools/testing/selftests/kvm/include/x86_64/processor.h static inline void cpu_relax(void) · line 832
Quoted 10 lines
tools/testing/selftests/kvm/include/x86_64/processor.h +10
-0+ +GUEST_ASSERT (tsc_khz); + cycles = tsc_khz /1000 * usec; + + start =rdtsc (); +for (;;) { + now =rdtsc (); +if (now - start >= cycles) +break ; +cpu_relax ();
Given that this is guest code, we should omit the PAUSE so that it doesn't trigger PLE exits, i.e. to make the delay as accurate as possible. Then this simply becomes:
start = rdtsc();
do {
now = rdtsc();
} while (now - start < cycles);PATCH v9 2/2 KVM: selftests: Add test for configure of x86 APIC bus frequency
In reply to Reinette ChatreFrom: Isaku Yamahata <[email protected]>
Test if KVM emulates the APIC bus clock at the expected frequency when userspace configures the frequency via KVM_CAP_X86_APIC_BUS_CYCLES_NS.
Set APIC timer's initial count to the maximum value and busy wait for 100 msec (largely arbitrary) using the TSC. Read the APIC timer's "current count" to calculate the actual APIC bus clock frequency based on TSC frequency.
---
Changes v9:
- Use uint64_t to store APIC frequency (was unsigned long). (Sean)
- Introduce new helpers (apic_enable(), apic_read_reg(), and apic_write_reg()) that runs appropriate xAPIC or x2APIC functions based on new is_x2apic global. (Sean)
- Use new apic_* helpers in guest test code to eliminate duplicate code. (Sean)
- Use constant defined at beginning of function for ICR register value. (Sean)
- Drop unnecessary GUEST_ASSERT(tsc_cycles > 0). (Sean)
- Drop unnecessary comments, use provided comments. (Sean)
- Change APIC frequency command line option to "-f" (was "-a"). (Sean)
- Enable user to override delay used in guest with new "-d" command line option. (Sean)
- Open code test checks in what is now single guest test function.
Changes v8:
- With TSC frequency switching to unsigned int type the implicit type promotion during bus frequency computation results in overflow. Use explicit unsigned long type within computation to avoid overflow.
Changes v7:
- Drop Maxim Levitsky's Reviewed-by because of significant changes.
- Remove redefine of _GNU_SOURCE. (kernel test robot)
- Rewrite changelog and test description. (Sean)
- Enable user space to set APIC bus frequency. (Sean)
- Use GUEST_ASSERT() from guest instead of TEST_ASSERT() from host. (Sean)
- Test xAPIC as well as x2APIC. (Sean)
- Add check that KVM_CAP_X86_APIC_BUS_CYCLES_NS cannot be set after vCPU created. (Sean)
- Use new udelay() utility. (Sean)
- Drop CLI. Mask LVT timer independently. (Sean)
- Use correct LVT timer entry (0x350 -> 0x320) to enable oneshot operation.
- Remove unnecessary static functions from single file test.
- Be consistent in types by using uint32_t/uint64_t instead of u32/u64.
Changes v6:
- Use vm_create() wrapper instead of open coding it. (Zide)
- Improve grammar of test description. (Zide)
Changes v5:
- Update to new name of capability KVM_CAP_X86_APIC_BUS_FREQUENCY -> KVM_CAP_X86_APIC_BUS_CYCLES_NS. (Xiaoyao Li)
Changes v4:
- Rework changelog.
- Add Sean's "Suggested-by" to acknowledge guidance received in
https://lore.kernel.org/all/[email protected]/
- Add copyright.
- Add test description to file header.
- Consistent capitalization for acronyms.
- Rebase to kvm-x86/next.
- Update to v4 change of providing bus clock rate in nanoseconds.
- Add a "TEST_REQUIRE()" for the new capability so that the test can work on kernels that do not support the new capability.
- Address checkpatch warnings and use tabs instead of spaces in header file to match existing code.
Changes v3:
- Use 1.5GHz instead of 1GHz as frequency.
Changes v2:
- Newly added.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/apic.h | 8 + .../kvm/x86_64/apic_bus_clock_test.c | 202 ++++++++++++++++++ 3 files changed, 211 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -112,6 +112,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/vmx_invalid_nested_guest_state TEST_GEN_PROGS_x86_64 += x86_64/vmx_set_nested_state_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_tsc_adjust_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_nested_tsc_scaling_test +TEST_GEN_PROGS_x86_64 += x86_64/apic_bus_clock_test TEST_GEN_PROGS_x86_64 += x86_64/xapic_ipi_test TEST_GEN_PROGS_x86_64 += x86_64/xapic_state_test TEST_GEN_PROGS_x86_64 += x86_64/xcr0_cpuid_test
tools/testing/selftests/kvm/include/x86_64/apic.h +8 -0
@@ -60,6 +60,14 @@#define APIC_VECTOR_MASK 0x000FF#define APIC_ICR2 0x310#define SET_APIC_DEST_FIELD (x) ((x) << 24) +#define APIC_LVTT 0x320 +#define APIC_LVT_TIMER_ONESHOT (0 << 17) +#define APIC_LVT_TIMER_PERIODIC (1 << 17) +#define APIC_LVT_TIMER_TSCDEADLINE (2 << 17) +#define APIC_LVT_MASKED (1 << 16) +#define APIC_TMICT 0x380 +#define APIC_TMCCT 0x390 +#define APIC_TDCR 0x3E0void apic_disable (void );void xapic_enable (void );
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +202 -0
202 lines not on this page · raw message
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH V9 2/2] KVM: selftests: Add test for configure of x86 APIC bus frequency
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c line 27
Quoted 11 lines · Reinette Chatre
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +11
-0+/* +* Pick 25MHz for APIC bus frequency. Different enough from the default 1GHz. +* User can override via command line. +*/ +static uint64_t apic_hz =25 *1000 *1000 ; + +/* +* Delay in msec that guest uses to determine APIC bus frequency. +* User can override via command line. +*/ +static unsigned long delay_ms =100 ;
There's no need for these to be global, it's easy enough to pass them as params to apic_guest_code(). Making is_x2apic is wortwhile as it cuts down on the noise, but for these, I think it's better to keep them local.
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c line 35
Quoted 8 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +8
-0+ +/* +* Possible TDCR values with matching divide count. Used to modify APIC +* timer frequency. +*/ +static struct { + uint32_t tdcr; + uint32_t divide_count;
These can/should all be const.
Quoted 59 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +59
-0+} tdcrs[] = { + {0x0, 2}, + {0x1, 4}, + {0x2, 8}, + {0x3, 16}, + {0x8, 32}, + {0x9, 64}, + {0xa, 128}, + {0xb, 1}, +}; + +/* true if x2APIC test is running, false if xAPIC test is running. */ +static bool is_x2apic; + +static void apic_enable(void) +{ + if (is_x2apic) + x2apic_enable(); + else + xapic_enable(); +} + +static uint32_t apic_read_reg(unsigned int reg) +{ + return is_x2apic ? x2apic_read_reg(reg) : xapic_read_reg(reg); +} + +static void apic_write_reg(unsigned int reg, uint32_t val) +{ + if (is_x2apic) + x2apic_write_reg(reg, val); + else + xapic_write_reg(reg, val); +} + +static void apic_guest_code(void) +{ + uint64_t tsc_hz = (uint64_t)tsc_khz * 1000; + const uint32_t tmict = ~0u; + uint64_t tsc0, tsc1, freq; + uint32_t tmcct; + int i; + + apic_enable(); + + /* + * Setup one-shot timer. The vector does not matter because the + * interrupt should not fire. + */ + apic_write_reg(APIC_LVTT, APIC_LVT_TIMER_ONESHOT | APIC_LVT_MASKED); + + for (i = 0; i < ARRAY_SIZE(tdcrs); i++) { + + apic_write_reg(APIC_TDCR, tdcrs[i].tdcr); + apic_write_reg(APIC_TMICT, tmict); + + tsc0 = rdtsc(); + udelay(delay_ms * 1000); + tmcct = apic_read_reg(APIC_TMCCT);
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c line 104
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +10
-0+ tsc1 =rdtsc (); + +/* +* Stop the timer _after_ reading the current, final count, as +* writing the initial counter also modifies the current count. +*/ +apic_write_reg (APIC_TMICT ,0 ); + + freq = (tmict - tmcct) * tdcrs[i].divide_count * tsc_hz / (tsc1 - tsc0); +/* Check if measured frequency is within 1% of configured frequency. */
1% is likely too aggressive, i.e. we'll get false failures due to host activity. On our systems, even a single pr_warn() in the timer path causes failure. For now, I think 5% is good enough, e.g. it'll catch cases where KVM is waaay off. If we want to do better (or that's still too tight), then we can add a '-t <tolerance' param or something.
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c line 106
Quoted 2 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +2
-0+GUEST_ASSERT (freq < apic_hz *101 /100 ); +GUEST_ASSERT (freq > apic_hz *99 /100 );
Combine these into a single assert, and print the params, i.e. don't rely on the line number to figure out what's up.
__GUEST_ASSERT(freq < apic_hz * 105 / 100 && freq > apic_hz * 95 / 100, "Frequency = %lu (wanted %lu - %lu), bus = %lu, div = %u, tsc = %lu", freq, apic_hz * 95 / 100, apic_hz * 105 / 100, apic_hz, tdcrs[i].divide_count, tsc_hz);
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +10
-0+int main (int argc,char *argv[]) +{ +int opt; + +TEST_REQUIRE (kvm_has_cap (KVM_CAP_X86_APIC_BUS_CYCLES_NS )); + +while ((opt =getopt (argc, argv,"d:f:h" )) !=-1 ) { +switch (opt) { +case 'f' : + apic_hz =atol (optarg);
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c line 196
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +10
-0+break ; +case 'd' : + delay_ms =atol (optarg); +break ; +case 'h' : +help (argv[0 ]); +exit (0 ); +default : +help (argv[0 ]); +exit (1 );
Heh, selftests are anything but consistent, but this should be exit(KSFT_SKIP) for both.
Quoted 9 lines
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +6
-0+ } + } + +run_apic_bus_clock_test (false); +run_apic_bus_clock_test (true); +}Signature · 1 line
2.34.1
Sean Christopherson
maintainerVMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
In reply to Reinette ChatreForking this off to try and avoid confusion...
Quoted 17 lines · Reinette Chatre
(quoted diff) +17
-0+/* + * Possible TDCR values with matching divide count. Used to modify APIC + * timer frequency. + */ +static struct { + uint32_t tdcr; + uint32_t divide_count; +} tdcrs[] = { + {0x0, 2}, + {0x1, 4}, + {0x2, 8}, + {0x3, 16}, + {0x8, 32}, + {0x9, 64}, + {0xa, 128}, + {0xb, 1}, +};
...
Quoted 27 lines
(quoted diff) +27
-0+static void apic_guest_code(void) +{ + uint64_t tsc_hz = (uint64_t)tsc_khz * 1000; + const uint32_t tmict = ~0u; + uint64_t tsc0, tsc1, freq; + uint32_t tmcct; + int i; + + apic_enable(); + + /* + * Setup one-shot timer. The vector does not matter because the + * interrupt should not fire. + */ + apic_write_reg(APIC_LVTT, APIC_LVT_TIMER_ONESHOT | APIC_LVT_MASKED); + + for (i = 0; i < ARRAY_SIZE(tdcrs); i++) { + + apic_write_reg(APIC_TDCR, tdcrs[i].tdcr); + apic_write_reg(APIC_TMICT, tmict); + + tsc0 = rdtsc(); + udelay(delay_ms * 1000); + tmcct = apic_read_reg(APIC_TMCCT); + tsc1 = rdtsc(); + + /*
Review
Quoted 10 lines
(quoted diff) +10
-0+ * Stop the timer _after_ reading the current, final count, as + * writing the initial counter also modifies the current count. + */ + apic_write_reg(APIC_TMICT, 0); + + freq = (tmict - tmcct) * tdcrs[i].divide_count * tsc_hz / (tsc1 - tsc0); + /* Check if measured frequency is within 1% of configured frequency. */ + GUEST_ASSERT(freq < apic_hz * 101 / 100); + GUEST_ASSERT(freq > apic_hz * 99 / 100); + }
This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
As mentioned ad nauseum at this point, I'm offline all of next week, so hopefully there's enough info here to get a root cause...
A spurious VM-Exit will occur after programming a vmcs.PREEMPTION_TIMER_VALUE that shouldn't exit. Every observed failure occurs when bits 27:16 are zero, with a small value in bits 15:0, e.g. VM-Enter with a timer value of 0xe0003bf7 or 0xa0006db6 will cause a near-immediate VM-Exit.
Weirdly, it doesn't always happen, e.g. I've observed rollover from 0xa000xxxx to 0x9fffxxxx without failure. However, the *test* failure is 100% reproducible, i.e. it's not _that_ rare of a failure. So maybe there's state build-up required? E.g. in the failing cases, there are 10s of entries with a slightly larger timer value, with no preemption timer exit (the host's tick IRQ interrupts the guest, and then KVM reprograms the VMX timer).
Even more sketchy, the failure only occurs if APICv is enabled. Turning off APICv makes the problem go away (I initially suspected KVM was somehow botching the TMCCT emulation). I am 99.9% certain there is no KVM APICv bug that is causing problems. Our IVB servers don't fail (even with APICv enabled), nor does my Raptor Lake client box (with APICv enabled). And I confirmed that the VMX timer is still getting programmed with the same sequence that fails when APICv is enabled.
(Before I realized the pattern of values), I sanity checked the VMCS field just before VM-Enter, and again after VM-Exit (KVM runs without the CPU save vmcs.PREEMPTION_TIMER_VALUE on exit).
I also verified the CPU thinks the timer has expired by enabling "save on exit" and verifying vmcs.PREEMPTION_TIMER_VALUE is indeed '0', and that KVM really did get a PREEMPTION_TIMER exit.
Attached is the debug patch I used to get the below data (sort of; one of the post-exit prints is without saving vmcs.PREEMPTION_TIMER_VALUE on exit).
In kvm_hypercall (ignore the name, I piggybacked a tracepoint because trying to log to dmesg was too slow, guest literally couldn't make forward progress), a1 is the VMX timer value programmed by KVM (0xe0003bf7).
apic_bus_clock_-11419 [056] d..2. 146.771179: kvm_hypercall: nr 0x2c8a9e9cc6703c a0 0x2c8b0e9ce46c37 a1 0xe0003bf7 a2 0xe0003bf7 a3 0x7 apic_bus_clock_-11419 [056] d..2. 146.771242: kvm_exit: vcpu 0 reason PREEMPTION_TIMER rip 0x402065 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
VMX timer exit, VMCS = e0003bf7, delta = e0003690
and the post-exit print with the attached patch:
kvm_intel: VMX timer exit, EXIT_REASON = 34, VMCS = 0, delta = e00037e9
And I've been fiddling with the below hack to coerce KVM into programming VMX timer values. Had I more time, I would have booted kernels with the ability to properly fuzz the timer values.
Note, with TMICT=-1, only divide_count=1 will fail, because larger divide counts result in a timeout that doesn't fit into the 32-bit VMX timer field (don't ask me how long it took me to realize the divide count affects the time frequency, not the actual count, *sigh*).
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c +14 -9
@@ -22,13 +22,13 @@ static const struct {const uint32_t tdcr;const uint32_t divide_count; } tdcrs[] = { - {0x0, 2}, - {0x1, 4}, - {0x2, 8}, - {0x3, 16}, - {0x8, 32}, - {0x9, 64}, - {0xa, 128}, + // {0x0, 2}, + // {0x1, 4}, + // {0x2, 8}, + // {0x3, 16}, + // {0x8, 32}, + // {0x9, 64}, + // {0xa, 128}, {0xb, 1}, };
@@ -55,10 +55,11 @@ static void apic_write_reg(unsigned int reg, uint32_t val)xapic_write_reg (reg, val); } +uint32_t tmict = ~0u ; +static void apic_guest_code (uint64_t apic_hz,uint64_t delay_ms) {uint64_t tsc_hz = guest_tsc_khz *1000 ; -const uint32_t tmict = ~0u ;uint64_t tsc0, tsc1, freq;uint32_t tmcct;int i;
@@ -133,6 +134,7 @@ static void run_apic_bus_clock_test(uint64_t apic_hz, uint64_t delay_ms, vm =vm_create (1 );sync_global_to_guest (vm, is_x2apic); +sync_global_to_guest (vm, tmict);vm_enable_cap (vm,KVM_CAP_X86_APIC_BUS_CYCLES_NS ,NSEC_PER_SEC / apic_hz);
@@ -174,7 +176,7 @@ int main(int argc, char *argv[])TEST_REQUIRE (kvm_has_cap (KVM_CAP_X86_APIC_BUS_CYCLES_NS )); -while ((opt =getopt (argc, argv,"d:f:h" )) !=-1 ) { +while ((opt =getopt (argc, argv,"d:f:i:h" )) !=-1 ) {switch (opt) {case 'f' : apic_hz =atoi_positive ("APIC bus frequency" , optarg) *1000 *1000 ;
@@ -182,6 +184,9 @@ int main(int argc, char *argv[])case 'd' : delay_ms =atoi_positive ("Delay in milliseconds" , optarg);break ; +case 'i' : + tmict = ~0u -atoi_positive ("offset" , optarg); +break ;case 'h' :default :help (argv[0 ]);
From 42da584d5e4e2dfaf4296f2ea666f3a04c82052f Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Fri, 28 Jun 2024 16:33:32 -0700
Subject: [PATCH] debug
--- arch/x86/kvm/vmx/vmx.c | 33 +++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/vmx.h | 1 + arch/x86/kvm/x86.c | 1 + 3 files changed, 31 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/vmx.c +29 -4
@@ -4431,8 +4431,7 @@ static u32 vmx_vmexit_ctrl(void)* Not used by KVM and never set in vmcs01 or vmcs02, but emulated for * nested virtualization and thus allowed to be set in vmcs12. */ - vmexit_ctrl &= ~(VM_EXIT_SAVE_IA32_PAT |VM_EXIT_SAVE_IA32_EFER | -VM_EXIT_SAVE_VMX_PREEMPTION_TIMER ); + vmexit_ctrl &= ~(VM_EXIT_SAVE_IA32_PAT |VM_EXIT_SAVE_IA32_EFER );if (vmx_pt_mode_is_system ()) vmexit_ctrl &= ~(VM_EXIT_PT_CONCEAL_PIP |
@@ -5997,6 +5996,8 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit) {struct vcpu_vmx *vmx =to_vmx (vcpu); +u32 delta_tsc; +u64 tscl;/* * In the *extremely* unlikely scenario that this is a spurious VM-Exit
@@ -6020,6 +6021,16 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu,if (is_guest_mode (vcpu))return EXIT_FASTPATH_NONE ; + tscl =rdtsc (); +if (vmx->hv_deadline_tsc > tscl) + delta_tsc = (u32)((vmx->hv_deadline_tsc - tscl) >> cpu_preemption_timer_multi); +else + delta_tsc =0 ; + +pr_warn ("VMX timer exit, EXIT_REASON = %x, VMCS = %x, delta = %x\n" , +vmcs_read32 (VM_EXIT_REASON ), +vmcs_read32 (VMX_PREEMPTION_TIMER_VALUE ), delta_tsc); +kvm_lapic_expired_hv_timer (vcpu);return EXIT_FASTPATH_REENTER_GUEST ; }
@@ -7197,6 +7208,8 @@ static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exitu32 delta_tsc;if (force_immediate_exit) { +trace_kvm_hypercall (0 ,0 ,0 ,0 ,0 ); + vmx->preemption_timer =0 ;vmcs_write32 (VMX_PREEMPTION_TIMER_VALUE ,0 ); vmx->loaded_vmcs->hv_timer_soft_disabled = false; }else if (vmx->hv_deadline_tsc !=-1 ) {
@@ -7208,9 +7221,14 @@ static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exitelse delta_tsc =0 ; +trace_kvm_hypercall (tscl, vmx->hv_deadline_tsc, delta_tsc, + ((vmx->hv_deadline_tsc - tscl) >> cpu_preemption_timer_multi), + cpu_preemption_timer_multi); + vmx->preemption_timer = delta_tsc;vmcs_write32 (VMX_PREEMPTION_TIMER_VALUE , delta_tsc); vmx->loaded_vmcs->hv_timer_soft_disabled = false; }else if (!vmx->loaded_vmcs->hv_timer_soft_disabled) { +trace_kvm_hypercall (-1 ,-1 ,-1 ,-1 ,-1 );vmcs_write32 (VMX_PREEMPTION_TIMER_VALUE ,-1 ); vmx->loaded_vmcs->hv_timer_soft_disabled = true; }
@@ -7218,6 +7236,8 @@ static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) { +WARN_ON (!vmx->loaded_vmcs->hv_timer_soft_disabled && +vmcs_read32 (VMX_PREEMPTION_TIMER_VALUE ) != vmx->preemption_timer);if (unlikely (host_rsp != vmx->loaded_vmcs->host_state.rsp)) { vmx->loaded_vmcs->host_state.rsp = host_rsp;vmcs_writel (HOST_RSP , host_rsp);
@@ -8128,7 +8148,7 @@ int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, delta_tsc &&u64_shl_div_u64 (delta_tsc, kvm_caps.tsc_scaling_ratio_frac_bits, vcpu->arch.l1_tsc_scaling_ratio, &delta_tsc)) - return -ERANGE; + goto out_of_range;/* * If the delta tsc can't fit in the 32 bit after the multi shift,
@@ -8137,11 +8157,16 @@ int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc,* on every vmentry is costly so we just use an hrtimer. */ if (delta_tsc >> (cpu_preemption_timer_multi +32 )) -return -ERANGE ; + goto out_of_range; +trace_kvm_hypercall (tscl, vmx->hv_deadline_tsc, delta_tsc,-1 ,-1 ); vmx->hv_deadline_tsc = tscl + delta_tsc; *expired = !delta_tsc;return 0 ; + +out_of_range: + vmx->hv_deadline_tsc =-1 ; +return -ERANGE ; }void vmx_cancel_hv_timer (struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/vmx.h +1 -0
@@ -341,6 +341,7 @@ struct vcpu_vmx {/* apic deadline value in host tsc */ u64 hv_deadline_tsc; +u32 preemption_timer;unsigned long host_debugctlmsr;
arch/x86/kvm/x86.c +1 -0
@@ -14024,6 +14024,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_write_tsc_offset);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_ple_window_update);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_pml_full); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_hypercall);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_pi_irte_update);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_unaccelerated_access);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_incomplete_ipi);
base-commit: 128c71b7f489d8115d29a487367c648f8acc8374
Signature · 1 line
2.45.2.803.g4e1b14247a-goog
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Hi Sean,
Quoted 3 lines · Sean Christopherson
Forking this off to try and avoid confusion...
On Wed, Jun 12, 2024, Reinette Chatre wrote:
...
Quoted 10 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + freq = (tmict - tmcct) * tdcrs[i].divide_count * tsc_hz / (tsc1 - tsc0); + /* Check if measured frequency is within 1% of configured frequency. */ + GUEST_ASSERT(freq < apic_hz * 101 / 100); + GUEST_ASSERT(freq > apic_hz * 99 / 100); + }This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
It has been a few days. Just a note to let you know that we are investigating this. On my side I have not yet been able to reproduce this issue. I tested kvm-x86-next-2024.06.28 on an ICX and an CLX system by running 100 iterations of apic_bus_clock_test and they all passed. Since I have lack of experience here there are some Intel virtualization experts helping out with this investigation and I hope that they will be some insights from the analysis and testing that you already provided.
Reinette
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
(a short update ...)
Quoted 18 lines · Reinette Chatre
Quoted 3 lines · Sean Christopherson
Forking this off to try and avoid confusion...
On Wed, Jun 12, 2024, Reinette Chatre wrote:
...
Quoted 10 lines
Quoted 6 lines
+
+ freq = (tmict - tmcct) * tdcrs[i].divide_count *tsc_hz/ (tsc1 - tsc0); + /* Check if measured frequency is within 1% of configured frequency. */ +GUEST_ASSERT(freq < apic_hz * 101 / 100);
+GUEST_ASSERT(freq > apic_hz * 99 / 100);
+ }This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
Quoted 6 lines
It has been a few days. Just a note to let you know that we are investigating this. On my side I have not yet been able to reproduce this issue. I tested kvm-x86-next-2024.06.28 on an ICX and an CLX system by running 100 iterations of
apic_bus_clock_testand they all passed. Since I have lack of experience here there are some Intel virtualization experts helping out with this investigation and I hope that they will be some insights from the analysis and testing that you already provided.
I have now been able to test on SKX also and I am not yet able to reproduce. For
reference, the systems I tested on are:
SKX: https://ark.intel.com/content/www/us/en/ark/products/120507/intel-xeon-platinum-8170m-processor-35-75m-cache-2-10-ghz.html
ICX: https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html
CLX: https://ark.intel.com/content/www/us/en/ark/products/192476/intel-xeon-platinum-8260l-processor-35-75m-cache-2-40-ghz.html
Reinette
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
In reply to Reinette ChatreQuoted 20 lines · Reinette Chatre
Hi Sean,
Quoted 3 lines · Sean Christopherson
Forking this off to try and avoid confusion...
On Wed, Jun 12, 2024, Reinette Chatre wrote:
...
Quoted 10 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + freq = (tmict - tmcct) * tdcrs[i].divide_count * tsc_hz / (tsc1 - tsc0); + /* Check if measured frequency is within 1% of configured frequency. */ + GUEST_ASSERT(freq < apic_hz * 101 / 100); + GUEST_ASSERT(freq > apic_hz * 99 / 100); + }This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
Quoted 9 lines
It has been a few days. Just a note to let you know that we are investigating this. On my side I have not yet been able to reproduce this issue. I tested kvm-x86-next-2024.06.28 on an ICX and an CLX system by running 100 iterations of
apic_bus_clock_testand they all passed. Since I have lack of experience here there are some Intel virtualization experts helping out with this investigation and I hope that they will be some insights from the analysis and testing that you already provided.Reinette
I can reproduce this on my side, even with apicv disabled by 'insmod $kernel_path/arch/x86/kvm/kvm-intel.ko enable_apicv=N'. @Sean I think we're observing same issue, please see the details below:
apic_bus_clock_test can't reproduce this may because the preemption timer value calculation relies on apic bus frequency/TMICT/Divide count/host TSC frequency and ratio between preemption timer and host TSC frequency, too many factors to generate the 'magic' value there. So I changed KVM and added a small KVM kselftest tool to set the preemption timer value directly from guest, this makes the reproducing easily. The changes are attached at end of this comment.
The trace I captured below came form host with 1.7GHz TSC, the VM_EXIT_SAVE_VMX_PREEMPTION_TIMER is enabled to get the cpu saved vmcs.VMX_PREEMPTION_TIMER_VALUE after VMEXIT. I set the vmcs.VMX_PREEMPTION_TIMER_VALUE to 0x880042ad which is the 'magic' number on this 1.7Ghz TSC machine:
preempt_test 20677.199521: kvm:kvm_vmx_debug: kvm_vmx_debug: 3, a0:0x77fd5554 a1:0x880042ad a2:0x880042ad a3:0x20462e98d9b9
a0: The previous vmcs.VMX_PREEMPTION_TIMER_VALUE value
saved by CPU when VMEXIT.
a1: The new preemption timer value wrote to
vmcs.VMX_PREEMPTION_TIMER_VALUE.
a2: The value read back from
vmcs.VMX_PREEMPTION_TIMER_VALUE, for double confirmation.
a3: The host tsc at the time point, debug only.
preempt_test 20677.199579: kvm:kvm_exit: reason PREEMPTION_TIMER rip 0x40274d info 0 0 intr 0
preempt_test 20677.199579: kvm:kvm_vmx_debug: kvm_vmx_debug: 2, a0:0x34 a1:0x0 a2:0x87fea9b0 a3:0x20462e9a5749
a0: The VMEXIT reason, 0x34 is preemption timer VMEXIT.
a1: The read back vmcs.VMX_PREEMPTION_TIMER_VALUE value, here 0.
a2: The next preemption timer value should be written to
vmcs, calculates from the (target tsc - current tsc) >>
7. Now the preemption timer vmexit happend only after
~58 microseconds elapsed, it should happen after
~171.79 seconds but not such soon, the issue is
reproduced.
Another more easy way to observe this symptom w/o care the 'magic' preemption timer vlaue is use the maximum preemption timer value 0xffffffff, below log w/ 0xffffffff is captured from same machine:
preempt_test 20530.456589: kvm:kvm_vmx_debug: kvm_vmx_debug: 3, a0:0x77fd5551 a1:0xffffffff a2:0xffffffff a3:0x200c1971ca5d
a0: The previous vmcs.VMX_PREEMPTION_TIMER_VALUE value
saved by CPU when VMEXIT.
a1: The new preemption timer value wrote to
vmcs.VMX_PREEMPTION_TIMER_VALUE.
a2: The read back value from
vmcs.VMX_PREEMPTION_TIMER_VALUE, double confirmation.
a3: the host tsc at the time point, debug only.
preempt_test 20530.456690: kvm:kvm_exit: reason VMCALL rip 0x4131a0 info 0 0 intr 0
The preempt_test checks preemption timer state every 100us, this VMEXIT is expected behavior.
preempt_test 20530.456691: kvm:kvm_entry: vcpu 0, rip 0x4131a3
preempt_test 20530.456691: kvm:kvm_vmx_debug: kvm_vmx_debug: 3, a0:0x77ff82cc a1:0xfffe900b a2:0xfffe900b a3:0x200c19746d45
a0: The previous vmcs.VMX_PREEMPTION_TIMER_VALUE value
saved by CPU when VMEXIT. The difference value
shouldn't be such huge number 0xffffffff - 0x77ff82cc
= 0x88007D33 when just ~100us elapsed from previous
VMENTRY, the issue is reproduced.
Use 0x88000000 as preemption timer value to verify this preempt_test tool, the preemption timer VMEXIT happend after ~171.45 seconds which is expected behavior on the host 1.7Ghz TSC system:
The preemption timer VMEXIT should happen after:
2281701376 × 128 / 1700000000 = 171.79 seconds.
Attached my changes in KVM and kselftest tool for
reproducing here, based on:
https://github.com/kvm-x86/linux.git
tag:kvm-x86-next-2024.06.28
Patch 01:
From a977bf12a8cd1bbe401e68d3702c0b5aa3bf66e4 Mon Sep 17 00:00:00 2001
From: Yao Yuan <[email protected]>
Date: Thu, 4 Jul 2024 09:59:55 +0800
Subject: [PATCH 1/2] KVM: x86: Introudce trace_kvm_vmx_debug()
debug only common trace.
--- arch/x86/kvm/trace.h | 28 ++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + 2 files changed, 29 insertions(+)
arch/x86/kvm/trace.h +28 -0
@@ -34,6 +34,34 @@ TRACE_EVENT(kvm_entry, __entry->immediate_exit ? "[immediate exit]" : "") ); +TRACE_EVENT (kvm_vmx_debug, + TP_PROTO(unsigned long n, unsigned long a0, + unsigned long a1, + unsigned long a2, + unsigned long a3), +TP_ARGS (n, a0, a1, a2, a3), + +TP_STRUCT__entry ( + __field( unsigned long, n ) + __field( unsigned long, a0 ) + __field( unsigned long, a1 ) + __field( unsigned long, a2 ) + __field( unsigned long, a3 ) + ), + +TP_fast_assign ( + __entry->n = n; + __entry->a0 = a0; + __entry->a1 = a1; + __entry->a2 = a2; + __entry->a3 = a3; + ), + +TP_printk ("kvm_vmx_debug: %ld, a0:0x%lx a1:0x%lx a2:0x%lx a3:0x%lx" , + __entry->n, __entry->a0, __entry->a1, __entry->a2, __entry->a3) +); + +/* * Tracepoint for hypercall. */
arch/x86/kvm/x86.c +1 -0
@@ -14036,6 +14036,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_vmgexit_exit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_vmgexit_msr_protocol_enter);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_vmgexit_msr_protocol_exit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_rmp_fault); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_vmx_debug);static int __init kvm_x86_init (void ) {
--
2.27.0
Patch 02:
From a3bdce4f2f93810372f6068396776ac702946d16 Mon Sep 17 00:00:00 2001
From: Yao Yuan <[email protected]>
Date: Wed, 3 Jul 2024 14:08:02 +0800
Subject: [PATCH 2/2] [DEBUG] preempt timer debug test
A specific kselftesting based program to allow set the VMX preempt timer value from VM directly.
Introduce 2 hypercall 0x56780001/2, 01 to set the preempt timer value, 02 to wait for the preemption time expired.
Usage:
Reload kvm applied this change, then:
$KRNEL_SRC_ROOT/tools/testing/selftests/kvm/x86_64/preempt_test -p 'preempt_timer_vale'
'preempt_timer_vale' is the preempt timer value in DEC format, HEX is not supported.
For example:
perf record -e "kvm:*" tools/testing/selftests/kvm/x86_64/preempt_test -p 2281718445
Above set the preempt value to 2281718445(0x880042AD) and capture the trace, then check the kvm_vmx_debug in the trace to know the preempt timer behavior.
--- tools/testing/selftests/kvm/Makefile | 1 + arch/x86/kvm/vmx/vmx.h | 5 + arch/x86/kvm/vmx/vmx.c | 113 +++++++++++++++++- .../selftests/kvm/x86_64/preempt_test.c | 82 +++++++++++++ 4 files changed, 198 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/preempt_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -129,6 +129,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/amx_test TEST_GEN_PROGS_x86_64 += x86_64/max_vcpuid_cap_test TEST_GEN_PROGS_x86_64 += x86_64/triple_fault_event_test TEST_GEN_PROGS_x86_64 += x86_64/recalc_apic_map_test +TEST_GEN_PROGS_x86_64 += x86_64/preempt_test TEST_GEN_PROGS_x86_64 += access_tracking_perf_test TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test
arch/x86/kvm/vmx/vmx.h +5 -0
@@ -368,6 +368,11 @@ struct vcpu_vmx {/* ve_info must be page aligned. */ struct vmx_ve_information *ve_info; + +volatile bool debug_timer; +bool debug_timer_set_to_hardware; +u32 debug_timer_val; +u64 debug_timer_deadline_tsc; };struct kvm_vmx {
arch/x86/kvm/vmx/vmx.c +110 -3
148 lines not on this page · raw message
tools/testing/selftests/kvm/x86_64/preempt_test.c new file +82 -0
82 lines not on this page · raw message
--
2.27.0
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
Quoted 8 lines
As mentioned ad nauseum at this point, I'm offline all of next week, so hopefully there's enough info here to get a root cause...
A spurious VM-Exit will occur after programming a vmcs.PREEMPTION_TIMER_VALUE that shouldn't exit. Every observed failure occurs when bits 27:16 are zero, with a small value in bits 15:0, e.g. VM-Enter with a timer value of
0xe0003bf7or0xa0006db6will cause a near-immediate VM-Exit.
This behavior is documented as a CPU erratum. See
https://cdrdv2.intel.com/v1/dl/getContent/793902
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
Problem: When the VMX-preemption timer is programmed with certain large values, the timer may expire earlier than expected. Actual values vary by platform and Time Stamp Counter (TSC) frequency.
Implication: Due to this erratum, software that relies on long duration VMXpreemption timers may observe VM exits significantly earlier than the programmed interval. Intel has not observed this erratum with any commercially available software.
Workaround: A mitigation for this erratum is for software to program the VMXpreemption timer for values below 2^25 × CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0].
Sean Christopherson
maintainerRe: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 9 lines · Chao Gao
Quoted 8 lines · Sean Christopherson
This test fails on our SKX, CLX, and ICX systems due to what appears to be a CPU bug. It looks like something APICv related is clobbering internal VMX timer state? Or maybe there's a tearing or truncation issue?
As mentioned ad nauseum at this point, I'm offline all of next week, so hopefully there's enough info here to get a root cause...
Quoted 7 lines
Quoted 4 lines
A spurious VM-Exit will occur after programming a vmcs.PREEMPTION_TIMER_VALUE that shouldn't exit. Every observed failure occurs when bits 27:16 are zero, with a small value in bits 15:0, e.g. VM-Enter with a timer value of
0xe0003bf7or0xa0006db6will cause a near-immediate VM-Exit.This behavior is documented as a CPU erratum. See
https://cdrdv2.intel.com/v1/dl/getContent/793902
Ha!
Quoted 1 line
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Thanks much for following up on this!
Quoted 10 lines
Problem: When the VMX-preemption timer is programmed with certain large values, the timer may expire earlier than expected. Actual values vary by platform and Time Stamp Counter (TSC) frequency.
Implication: Due to this erratum, software that relies on long duration VMXpreemption timers may observe VM exits significantly earlier than the programmed interval. Intel has not observed this erratum with any commercially available software.
Workaround: A mitigation for this erratum is for software to program the VMXpreemption timer for values below 2^25 × CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0].
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 3 lines
Quoted 1 line
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Yes.
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 6 lines · Chao Gao
Quoted 4 lines
Quoted 2 lines
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Yes.
This test still fails on our SKX, CLX, and ICX systems.
Sean,
Were you thinking of enforcing a cap on delta_tsc in vmx_set_hv_timer()?
Sean Christopherson
maintainerRe: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 6 lines · Jim Mattson
Quoted 5 lines · Chao Gao
Quoted 4 lines
Quoted 2 lines
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Quoted 7 lines
Quoted 1 line
Yes.
This test still fails on our SKX, CLX, and ICX systems.
Sean,
Were you thinking of enforcing a cap on
delta_tscinvmx_set_hv_timer()?
Heh, to be honest, I wasn't thinking of a whole lot of nothing. Falling back to hrtimers does seem like the easiest solution.
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
?
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Jim Mattson
Quoted 6 lines · Chao Gao
Quoted 4 lines
Quoted 2 lines
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Yes.
Quoted 8 lines
Quoted 5 lines
This test still fails on our SKX, CLX, and ICX systems.
Sean,
Were you thinking of enforcing a cap on
delta_tscinvmx_set_hv_timer()?Heh, to be honest, I wasn't thinking of a whole lot of nothing. Falling back to hrtimers does seem like the easiest solution.
I think vmx_set_hv_timer() should return -EINVAL for values impacted by this erratum. However, the only documented issue is for EMR, and we have not observed the problem on EMR. That's unsettling.
Chao:
1) Should we just assume that all Intel CPUs are affected?
2) Is there any compelling reason not to simplify the limit to 2^25?
3) Is it just coincidence that 25 + IA32_VMX_MISC[4:0] (on EMR) == 32, or should the limit be calculated as 32 - IA32_VMX_MISC[4:0]?
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 17 lines · Jim Mattson
?
Quoted 14 lines · Sean Christopherson
Quoted 12 lines · Jim Mattson
Quoted 6 lines · Chao Gao
Quoted 4 lines
Quoted 2 lines
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Yes.
This test still fails on our SKX, CLX, and ICX systems.
Sean,
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Were you thinking of enforcing a cap on
delta_tscinvmx_set_hv_timer()?Heh, to be honest, I wasn't thinking of a whole lot of nothing. Falling back to hrtimers does seem like the easiest solution.
I think
vmx_set_hv_timer()should return-EINVALfor values impacted by this erratum. However, the only documented issue is for EMR, and we have not observed the problem on EMR. That's unsettling.
Could you clarify what tests you ran?
I am using the reproducer from Yuan:
https://lore.kernel.org/kvm/20240708055559.rl4w5xfhj3uru6j2@yy-desk-7060/
I write -1 to the VMX preemption timer, do VM-Enter, and have the guest execute VMCALL to force a VM-Exit. On VM-Exit, we read back the preemption timer. The delta should be very small; otherwise, the platform likely has the same issue.
I tested several platforms, including EMR. The results are consistent with the erratum, i.e., I observed premature VMX preemption-timer VM-Exits, and the documented limit did not trigger premature VMX preemption-timer VM-Exits in my testing.
Quoted 4 lines
Chao:
1) Should we just assume that all Intel CPUs are affected?
I think that is reasonable unless we have explicit evidence to exclude specific parts.
Quoted 2 lines
2) Is there any compelling reason not to simplify the limit to 2^25?
We can use 2^25 as a conservative bound, but it is much lower than necessary. The current bound comes from theoretical analysis and was validated on multiple platforms.
Quoted 3 lines
3) Is it just coincidence that 25 + IA32_VMX_MISC[4:0] (on EMR) == 32, or should the limit be calculated as 32 - IA32_VMX_MISC[4:0]?
My understanding is that hardware scales the preemption-timer value and converts it to a 32-bit core crystal clock counter, rather than directly using a 32-bit TSC delta. IA32_VMX_MISC[4:0] likely participates in that calculation.
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 21 lines · Chao Gao
Quoted 19 lines · Jim Mattson
?
Quoted 16 lines · Sean Christopherson
Quoted 13 lines · Jim Mattson
Quoted 6 lines · Chao Gao
Quoted 4 lines
Quoted 2 lines
EMR158. VMX-Preemption Timer May Expire Earlier With Certain Large Timer Values
I assume the same erratum applies to previous generations as well?
Yes.
This test still fails on our SKX, CLX, and ICX systems.
Sean,
Were you thinking of enforcing a cap on
delta_tscinvmx_set_hv_timer()?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Heh, to be honest, I wasn't thinking of a whole lot of nothing. Falling back to hrtimers does seem like the easiest solution.
I think
vmx_set_hv_timer()should return-EINVALfor values impacted by this erratum. However, the only documented issue is for EMR, and we have not observed the problem on EMR. That's unsettling.Could you clarify what tests you ran?
Just tools/testing/selftests/kvm/x86/apic_bus_clock_test.
It fails on SKX, CLX, and ICX. It passes on SPR, EMR, and GMR.
Quoted 22 lines
I am using the reproducer from Yuan:
https://lore.kernel.org/kvm/20240708055559.rl4w5xfhj3uru6j2@yy-desk-7060/I write -1 to the VMX preemption timer, do VM-Enter, and have the guest execute VMCALL to force a VM-Exit. On VM-Exit, we read back the preemption timer. The delta should be very small; otherwise, the platform likely has the same issue.
I tested several platforms, including EMR. The results are consistent with the erratum, i.e., I observed premature VMX preemption-timer VM-Exits, and the documented limit did not trigger premature VMX preemption-timer VM-Exits in my testing.
Quoted 4 lines
Chao:
1) Should we just assume that all Intel CPUs are affected?
I think that is reasonable unless we have explicit evidence to exclude specific parts.
Quoted 1 line
Quoted 5 lines
Quoted 1 line
2) Is there any compelling reason not to simplify the limit to 2^25?
We can use 2^25 as a conservative bound, but it is much lower than necessary. The current bound comes from theoretical analysis and was validated on multiple platforms.
Yes, but how often do guests program their local APIC timer to fire more than 2^(25 + IA32_VMX_MISC[4:0]) cycles in the future?
Quoted 8 lines
Quoted 3 lines
3) Is it just coincidence that 25 + IA32_VMX_MISC[4:0] (on EMR) == 32, or should the limit be calculated as 32 - IA32_VMX_MISC[4:0]?
My understanding is that hardware scales the preemption-timer value and converts it to a 32-bit core crystal clock counter, rather than directly using a 32-bit TSC delta. IA32_VMX_MISC[4:0] likely participates in that calculation.
That doesn't definitively answer my question. Let me try to rephrase it.
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 10 lines · Jim Mattson
Quoted 6 lines
Quoted 4 lines
I think
vmx_set_hv_timer()should return-EINVALfor values impacted by this erratum. However, the only documented issue is for EMR, and we have not observed the problem on EMR. That's unsettling.Could you clarify what tests you ran?
Just
tools/testing/selftests/kvm/x86/apic_bus_clock_test.It fails on SKX, CLX, and ICX. It passes on SPR, EMR, and GMR.
Thanks. In that case, that test likely does not trigger the issue.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Is there any compelling reason not to simplify the limit to 2^25?
We can use 2^25 as a conservative bound, but it is much lower than necessary. The current bound comes from theoretical analysis and was validated on multiple platforms.
Yes, but how often do guests program their local APIC timer to fire more than 2^(25 + IA32_VMX_MISC[4:0]) cycles in the future?
I had interpreted your earlier question as referring to the erratum write-up itself (i.e., why Intel did not publish 2^25 directly as the limit).
If we are talking about the VMM implementation, this should indeed be rare. I do not see a strong reason KVM could not use 2^25 as a conservative limit.
Quoted 12 lines
Quoted 9 lines
Quoted 4 lines
3) Is it just coincidence that 25 + IA32_VMX_MISC[4:0] (on EMR) == 32, or should the limit be calculated as 32 - IA32_VMX_MISC[4:0]?
My understanding is that hardware scales the preemption-timer value and converts it to a 32-bit core crystal clock counter, rather than directly using a 32-bit TSC delta. IA32_VMX_MISC[4:0] likely participates in that calculation.
That doesn't definitively answer my question. Let me try to rephrase it.
Quoted 7 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 2 lines · Chao Gao
On Thu, Jun 04, 2026 at 10:34:40PM -0700, Jim Mattson wrote:
...
Quoted 9 lines
Quoted 7 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Have you discovered anything?
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 8 lines · Jim Mattson
Quoted 2 lines · Chao Gao
On Thu, Jun 04, 2026 at 10:34:40PM -0700, Jim Mattson wrote:
...
Quoted 4 lines
Quoted 4 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Have you discovered anything?
Chao is on vacation, let's wait one more week.
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
In reply to Jim MattsonQuoted 8 lines · Jim Mattson
Quoted 2 lines · Chao Gao
On Thu, Jun 04, 2026 at 10:34:40PM -0700, Jim Mattson wrote:
...
Quoted 4 lines
Quoted 4 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Have you discovered anything?
I just got the final answer:
2^25*M/N is the max value of VMX preemption timer for _all_ affected processors. Our expectation is that for processors that have different value it will have the issue fixed (VMX preemption timer will implement full 32-bit). Use of 2^25*M/N will still be fine.
where M is CPUID.15H:EBX[31:0] and N is CPUID.15H:EAX[31:0].
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 18 lines · Chao Gao
Quoted 8 lines · Jim Mattson
Quoted 2 lines · Chao Gao
On Thu, Jun 04, 2026 at 10:34:40PM -0700, Jim Mattson wrote:
...
Quoted 4 lines
Quoted 4 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Have you discovered anything?
Quoted 8 lines
I just got the final answer:
2^25*M/N is the max value of VMX preemption timer for _all_ affected processors. Our expectation is that for processors that have different value it will have the issue fixed (VMX preemption timer will implement full 32-bit). Use of 2^25*M/N will still be fine.
where M is CPUID.15H:EBX[31:0] and N is CPUID.15H:EAX[31:0].
Great!
How will software identify processors that have the issue fixed?
Re: VMX Preemption Timer appears to be buggy on SKX, CLX, and ICX
Quoted 26 lines · Jim Mattson
Quoted 25 lines · Chao Gao
Quoted 8 lines · Jim Mattson
Quoted 2 lines · Chao Gao
On Thu, Jun 04, 2026 at 10:34:40PM -0700, Jim Mattson wrote:
...
Quoted 4 lines
Quoted 4 lines
With respect to EMR, you wrote previously, "A mitigation for this erratum is for software to program the VMX preemption timer for values below 2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]."
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My question is whether the exponent, 25, is a fixed value for all CPUs, regardless of their IA32_VMX_MISC[4:0]. It sounds like you are saying that the exponent may depend on IA32_VMX_MISC[4:0].
Let me double-check this with the internal team and get back to you.
Have you discovered anything?
I just got the final answer:
2^25*M/N is the max value of VMX preemption timer for _all_ affected processors. Our expectation is that for processors that have different value it will have the issue fixed (VMX preemption timer will implement full 32-bit). Use of 2^25*M/N will still be fine.
Quoted 5 lines
Quoted 1 line
where M is CPUID.15H:EBX[31:0] and N is CPUID.15H:EAX[31:0].
Great!
How will software identify processors that have the issue fixed?
That isn't clear at the moment.
Quoted 11 lines · Reinette Chatre
Changes from v8:
- v8: https://lore.kernel.org/lkml/[email protected]/
- Many changes to newudelay()utility patch as well as the APIC bus frequency test aimed to make it more robust (additional ASSERTs, consistent types, eliminate duplicate code, etc.) and useful with support for more user configuration. Please refer to individual patches for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEADe4e9e1067138e5620cf0500c3e5f6ebfb9d322c8.[...]
Applied to kvm-x86 misc, with all the changes mentioned in my earlier replies. I'm out next week, and don't want to merge the KVM changes without these tests, hence the rushed application.
Please holler if you disagree with anything (or if I broke something). I won't respond until July 8th at the earliest, but worst case scenario we can do fixup patches after 6.11-rc1.
[1/2] KVM: selftests: Add x86_64 guest udelay() utility
https://github.com/kvm-x86/linux/commit/6b878cbb87bf [2/2] KVM: selftests: Add test for configure of x86 APIC bus frequency https://github.com/kvm-x86/linux/commit/82222ee7e84c
Hi Sean,
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Reinette Chatre
Changes from v8:
- v8: https://lore.kernel.org/lkml/[email protected]/
- Many changes to newudelay()utility patch as well as the APIC bus frequency test aimed to make it more robust (additional ASSERTs, consistent types, eliminate duplicate code, etc.) and useful with support for more user configuration. Please refer to individual patches for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEADe4e9e1067138e5620cf0500c3e5f6ebfb9d322c8.[...]
Quoted 7 lines
Applied to kvm-x86 misc, with all the changes mentioned in my earlier replies. I'm out next week, and don't want to merge the KVM changes without these tests, hence the rushed application.
Please holler if you disagree with anything (or if I broke something). I won't respond until July 8th at the earliest, but worst case scenario we can do fixup patches after 6.11-rc1.
Thank you very much for taking the time to make the changes and apply the patches. All the changes look good to me and passes my testing.
Now that the x86 udelay() utility no longer use cpu_relax(), should ARM and RISC-V's udelay() be modified to match in this regard? I can prepare (unable to test) changes for you to consider on your return.
Reinette
Sean Christopherson
maintainerQuoted 24 lines · Reinette Chatre
Hi Sean,
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Reinette Chatre
Changes from v8:
- v8: https://lore.kernel.org/lkml/[email protected]/
- Many changes to newudelay()utility patch as well as the APIC bus frequency test aimed to make it more robust (additional ASSERTs, consistent types, eliminate duplicate code, etc.) and useful with support for more user configuration. Please refer to individual patches for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEADe4e9e1067138e5620cf0500c3e5f6ebfb9d322c8.[...]
Quoted 7 lines
Applied to kvm-x86 misc, with all the changes mentioned in my earlier replies. I'm out next week, and don't want to merge the KVM changes without these tests, hence the rushed application.
Please holler if you disagree with anything (or if I broke something). I won't respond until July 8th at the earliest, but worst case scenario we can do fixup patches after 6.11-rc1.
Quoted 6 lines
Thank you very much for taking the time to make the changes and apply the patches. All the changes look good to me and passes my testing.
Now that the x86
udelay()utility no longer usecpu_relax(), should ARM and RISC-V'sudelay()be modified to match in this regard? I can prepare (unable to test) changes for you to consider on your return.
I don't think so? IIUC, arm64's "yield", used by cpu_relax() doesn't trigger the "on spin" exists. Such exist are only triggered by "wfet" and friends.
Quoted 8 lines · Sean Christopherson
Quoted 3 lines · Reinette Chatre
Now that the x86
udelay()utility no longer usecpu_relax(), should ARM and RISC-V'sudelay()be modified to match in this regard? I can prepare (unable to test) changes for you to consider on your return.I don't think so? IIUC, arm64's "yield", used by
cpu_relax()doesn't trigger the "on spin" exists. Such exist are only triggered by "wfet" and friends.
ah, I see, thank you very much Sean.
Reinette