Skip to contentSkip to conversation

KVM: x86: Make bus clock frequency for vAPIC timer configurable

PATCH Superseded Reinette Chatre 1 participant 4 messages Jun 6, 2024 – Jun 8, 2024

Subsystems (2)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover KVM: x86: Make bus clock frequency for vAPIC timer configurable

Changes from v6:
- V6: https://lore.kernel.org/lkml/[email protected]/
- KVM changes were merged into kvm-x86 misc and subsequently dropped from this submission. This submission only contains the selftest changes.
https://lore.kernel.org/lkml/[email protected]/
- New patch to add udelay() utility to x86_64 KVM selftests. (Sean)
- Many changes to APIC bus frequency selftest. Please refer to the patch for detailed changes.
- Series applies cleanly to next branch of kvm-x86 with HEAD af0903ab52ee6d6f0f63af67fa73d5eb00f79b9a.

Changes from v5:
- v5: https://lore.kernel.org/lkml/[email protected]/
- Rebased on latest of "next" branch of https://github.com/kvm-x86/linux.git
- Added Xiaoyao Li and Yuan Yao's Reviewed-by tags.
- Use the KVM selftest vm_create() wrapper instead of open coding it. (Zide)
- Improve grammar of selftest description. (Zide)

Changes from v4:
- v4: https://lore.kernel.org/lkml/[email protected]/
- Rename capability from KVM_CAP_X86_APIC_BUS_FREQUENCY to KVM_CAP_X86_APIC_BUS_CYCLES_NS. (Xiaoyao Li).
- Include "Testing" section in cover letter.
- Add Rick's Reviewed-by tags.
- Rebased on latest of "next" branch of https://github.com/kvm-x86/linux.git

Changes from v3:
- v3: https://lore.kernel.org/all/[email protected]/
- Reworked all changelogs.
- Regarding code changes: patches #1 and #2 are unchanged, patch #3 was reworked according to Sean's guidance, and patch #4 (the test) needed changes after rebase to kvm-x86/next and the implementation (patch #3) changes.
- Added Reviewed-by tags to patches #1, #2, and #4, but removed Maxim's Reviewed-by tag from patch #3 because it changed so much.
- Added a "Suggested-by" to patch #4 to reflect that it represents Sean's guidance.
- Reworked cover to match customs (in subject line) and reflect feedback to patches: capability renamed to KVM_CAP_X86_APIC_BUS_FREQUENCY, clarify distinction between "core crystal clock" and "bus clock", and update pro/con list.
- Please refer to individual patches for detailed changes.

Changes from v2:
- v2: https://lore.kernel.org/lkml/[email protected]/
- Removed APIC_BUS_FREQUENCY and apic_bus_frequency of struct kvm-arch.
- Update the commit messages.
- Added reviewed-by (Maxim Levitsky)
- Use 1.5GHz instead of 1GHz as frequency for the test case.

Changes from v1:
- v1: https://lore.kernel.org/all/[email protected]/
- Added a test case
- Fix a build error for i386 platform
- Add check if vcpu isn't created.
- Add check if lapic chip is in-kernel emulation.
- Updated api.rst

Summary
-------
Add KVM_CAP_X86_APIC_BUS_CYCLES_NS capability to configure the APIC bus clock frequency for APIC timer emulation.
Allow KVM_ENABLE_CAPABILITY(KVM_CAP_X86_APIC_BUS_CYCLES_NS) to set the frequency in nanoseconds. When using this capability, the user space VMM should configure CPUID leaf 0x15 to advertise the frequency.

Description
-----------
Vishal reported [1] that the TDX guest kernel expects a 25MHz APIC bus frequency but ends up getting interrupts at a significantly higher rate.

The TDX architecture hard-codes the core crystal clock frequency to 25MHz and mandates exposing it via CPUID leaf 0x15. The TDX architecture does not allow the VMM to override the value.

In addition, per Intel SDM:
"The APIC timer frequency will be the processor’s bus clock or core
crystal clock frequency (when TSC/core crystal clock ratio is enumerated in CPUID leaf 0x15) divided by the value specified in the divide configuration register."

The resulting 25MHz APIC bus frequency conflicts with the KVM hardcoded APIC bus frequency of 1GHz.

The KVM doesn't enumerate CPUID leaf 0x15 to the guest unless the user space VMM sets it using KVM_SET_CPUID. If the CPUID leaf 0x15 is enumerated, the guest kernel uses it as the APIC bus frequency. If not, the guest kernel measures the frequency based on other known timers like the ACPI timer or the legacy PIT. As reported in [1] the TDX guest kernel expects a 25MHz timer frequency but gets timer interrupt more frequently due to the 1GHz frequency used by KVM.

To ensure that the guest doesn't have a conflicting view of the APIC bus frequency, allow the userspace to tell KVM to use the same frequency that TDX mandates instead of the default 1Ghz.

There are several options to address this:
1. Make the KVM able to configure APIC bus frequency (this series).
Pro: It resembles the existing hardware. The recent Intel CPUs
adopts 25MHz.
Con: Require the VMM to emulate the APIC timer at 25MHz.
2. Make the TDX architecture enumerate CPUID leaf 0x15 to configurable frequency or not enumerate it.
Pro: Any APIC bus frequency is allowed.
Con: Deviates from TDX architecture.
3. Make the TDX guest kernel use 1GHz when it's running on KVM.
Con: The kernel ignores CPUID leaf 0x15.
4. Change CPUID leaf 0x15 under TDX to report the crystal clock frequency as 1 GHz.
Pro: This has been the virtual APIC frequency for KVM guests for 13
years.
Pro: This requires changing only one hard-coded constant in TDX.
Con: It doesn't work with other VMMs as TDX isn't specific to KVM.
Con: Core crystal clock frequency is also used to calculate TSC frequency.
Con: If it is configured to value different from hardware, it will
break the correctness of INTEL-PT Mini Time Count (MTC) packets in TDs.

Testing
-------
Tested on non-TDX host using included kselftest. Host running kernel with this series applied to "next" branch of
https://github.com/kvm-x86/linux.git

Tested on TDX host and TD using a modified version
of x86/apic.c:test_apic_timer_one_shot() available from
https://github.com/intel/kvm-unit-tests-tdx/blob/tdx/x86/apic.c.
Host running TDX KVM development patches and QEMU with corresponding TDX changes.
The test needed to be modified to (a) stop any lingering timers before the test starts, and (b) use CPUID 0x15 in TDX to accurately determine the TSC and APIC frequencies instead of making 1GHz assumption and use similar check as the kselftest test introduced in this series (while increasing the amount with which the frequency is allowed to deviate by 1%).

The core changes made to x86/apic.c:test_apic_timer_one_shot() for this testing are shown below for reference. Work is in progress to upstream these modifications.

@@ -477,11 +478,29 @@ static void lvtt_handler(isr_regs_t *regs)

static void test_apic_timer_one_shot(void)
{
- uint64_t tsc1, tsc2;
static const uint32_t interval = 0x10000;
+ uint64_t measured_apic_freq, tsc2, tsc1;
+ uint32_t tsc_freq = 0, apic_freq = 0;
+ struct cpuid cpuid_tsc = {};

#define APIC_LVT_TIMER_VECTOR (0xee)

+ /*
+ * CPUID 0x15 is not available in VMX, can use it to obtain
+ * TSC and APIC frequency for accurate testing
+ */
+ if (is_tdx_guest()) {
+ cpuid_tsc = raw_cpuid(0x15, 0);
+ tsc_freq = cpuid_tsc.c * cpuid_tsc.b / cpuid_tsc.a;
+ apic_freq = cpuid_tsc.c;
+ }
+ /*

+	   stop already fired local timer
+	   the test case can be negative failure if the timer fired
+	   after installed lvtt_handler but *before*
+	   write to TIMICT again.

+ */
+ apic_write(APIC_TMICT, 0);
handle_irq(APIC_LVT_TIMER_VECTOR, lvtt_handler);

/* One shot mode */
@@ -503,8 +522,16 @@ static void test_apic_timer_one_shot(void)
* cases, the following should satisfy on all modern
* processors.
*/
- report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
- "APIC LVT timer one shot");
+ if (is_tdx_guest()) {
+ measured_apic_freq = interval * (tsc_freq / (tsc2 - tsc1));
+ report((lvtt_counter == 1) &&

+		       (measured_apic_freq < apic_freq * 102 / 100) &&
+		       (measured_apic_freq > apic_freq * 98 / 100),
+		       "APIC LVT timer one shot");

+ } else {
+ report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
+ "APIC LVT timer one shot");
+ }
}

[1] https://lore.kernel.org/lkml/[email protected]/

Isaku Yamahata (1):
KVM: selftests: Add test for configure of x86 APIC bus frequency

Reinette Chatre (1):
KVM: selftests: Add x86_64 guest udelay() utility

 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/include/x86_64/apic.h       |   8 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 ++
 .../selftests/kvm/lib/x86_64/processor.c      |   8 +
 .../kvm/x86_64/apic_bus_clock_test.c          | 218 ++++++++++++++++++
 5 files changed, 250 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v7 1/2 KVM: selftests: Add x86_64 guest udelay() utility

Create udelay() utility for x86_64 tests to match
udelay() available to ARM and RISC-V tests.

Calibrate guest frequency using the KVM_GET_TSC_KHZ ioctl() and share it between host and guest with the new tsc_khz global variable.

---
 .../selftests/kvm/include/x86_64/processor.h      | 15 +++++++++++++++
 .../testing/selftests/kvm/lib/x86_64/processor.c  |  8 ++++++++
 2 files changed, 23 insertions(+)
tools/testing/selftests/kvm/include/x86_64/processor.h +15 -0
index 8eb57de0b587..f3a5881dd821 100644
@@ -23,6 +23,7 @@
 
 extern bool host_cpu_is_intel;
 extern bool host_cpu_is_amd;
+extern unsigned long tsc_khz;
 
 /* Forced emulation prefix, used to invoke the emulator unconditionally. */
 #define KVM_FEP "ud2; .byte 'k', 'v', 'm';"
@@ -815,6 +816,20 @@ static inline void cpu_relax(void)
 	asm volatile("rep; nop" ::: "memory");
 }
 
+static inline void udelay(unsigned long usec)
+{
+	unsigned long cycles = tsc_khz / 1000 * usec;
+	uint64_t start, now;
+
+	start = rdtsc();
+	for (;;) {
+		now = rdtsc();
+		if (now - start >= cycles)
+			break;
+		cpu_relax();
+	}
+}
+
 #define ud2()			\
 	__asm__ __volatile__(	\
 		"ud2\n"	\
tools/testing/selftests/kvm/lib/x86_64/processor.c +8 -0
index c664e446136b..6558fece8a36 100644
@@ -25,6 +25,7 @@ vm_vaddr_t exception_handlers;
 bool host_cpu_is_amd;
 bool host_cpu_is_intel;
 bool is_forced_emulation_enabled;
+unsigned long tsc_khz;
 
 static void regs_dump(FILE *stream, struct kvm_regs *regs, uint8_t indent)
 {
@@ -628,6 +629,13 @@ void kvm_arch_vm_post_create(struct kvm_vm *vm)
 
 		vm_sev_ioctl(vm, KVM_SEV_INIT2, &init);
 	}
+
+	if (kvm_has_cap(KVM_CAP_GET_TSC_KHZ)) {
+		tsc_khz = __vm_ioctl(vm, KVM_GET_TSC_KHZ, NULL);
+		if (tsc_khz < 0)
+			tsc_khz = 0;
+		sync_global_to_guest(vm, tsc_khz);
+	}
 }
 
 void vcpu_arch_set_entry_point(struct kvm_vcpu *vcpu, void *guest_code)
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH V7 1/2] KVM: selftests: Add x86_64 guest udelay() utility

Quoted 59 lines · Reinette Chatre

Create udelay() utility for x86_64 tests to match
udelay() available to ARM and RISC-V tests.

Calibrate guest frequency using the KVM_GET_TSC_KHZ ioctl() and share it between host and guest with the new tsc_khz global variable.

---
  .../selftests/kvm/include/x86_64/processor.h      | 15 +++++++++++++++
  .../testing/selftests/kvm/lib/x86_64/processor.c  |  8 ++++++++
  2 files changed, 23 insertions(+)
tools/testing/selftests/kvm/include/x86_64/processor.h +15 -0
index 8eb57de0b587..f3a5881dd821 100644
@@ -23,6 +23,7 @@
 
  extern bool host_cpu_is_intel;
  extern bool host_cpu_is_amd;
+extern unsigned long tsc_khz;
 
  /* Forced emulation prefix, used to invoke the emulator unconditionally. */
  #define KVM_FEP "ud2; .byte 'k', 'v', 'm';"
@@ -815,6 +816,20 @@ static inline void cpu_relax(void)
  	asm volatile("rep; nop" ::: "memory");
  }
 
+static inline void udelay(unsigned long usec)
+{
+	unsigned long cycles = tsc_khz / 1000 * usec;
+	uint64_t start, now;
+
+	start = rdtsc();
+	for (;;) {
+		now = rdtsc();
+		if (now - start >= cycles)
+			break;
+		cpu_relax();
+	}
+}
+
  #define ud2()			\
  	__asm__ __volatile__(	\
  		"ud2\n"	\
tools/testing/selftests/kvm/lib/x86_64/processor.c +1 -0
index c664e446136b..6558fece8a36 100644
@@ -25,6 +25,7 @@ vm_vaddr_t exception_handlers;
  bool host_cpu_is_amd;
  bool host_cpu_is_intel;
  bool is_forced_emulation_enabled;
+unsigned long tsc_khz;
 
  static void regs_dump(FILE *stream, struct kvm_regs *regs, uint8_t indent)
  {

tools/testing/selftests/kvm/lib/x86_64/processor.c void kvm_arch_vm_post_create(struct kvm_vm *vm) · line 635

Quoted 7 lines
tools/testing/selftests/kvm/lib/x86_64/processor.c +4 -0
@@ -628,6 +629,13 @@ void kvm_arch_vm_post_create(struct kvm_vm *vm)
 
  		vm_sev_ioctl(vm, KVM_SEV_INIT2, &init);
  	}
+
+	if (kvm_has_cap(KVM_CAP_GET_TSC_KHZ)) {
+		tsc_khz = __vm_ioctl(vm, KVM_GET_TSC_KHZ, NULL);
+		if (tsc_khz < 0)

This incorrect type usage will be fixed in next version that I plan to send on Monday.

tools/testing/selftests/kvm/lib/x86_64/processor.c void kvm_arch_vm_post_create(struct kvm_vm *vm) · line 641

Quoted 6 lines
tools/testing/selftests/kvm/lib/x86_64/processor.c +3 -0
+			tsc_khz = 0;
+		sync_global_to_guest(vm, tsc_khz);
+	}
  }
 
  void vcpu_arch_set_entry_point(struct kvm_vcpu *vcpu, void *guest_code)

Reinette

#3 lore raw

PATCH v7 2/2 KVM: selftests: Add test for configure of x86 APIC bus frequency

In reply to Reinette Chatre

From: Isaku Yamahata <[email protected]>

Test if KVM emulates the APIC bus clock at the expected frequency when userspace configures the frequency via KVM_CAP_X86_APIC_BUS_CYCLES_NS.

Set APIC timer's initial count to the maximum value and busy wait for 100 msec (largely arbitrary) using the TSC. Read the APIC timer's "current count" to calculate the actual APIC bus clock frequency based on TSC frequency.

---
Changes v7:
- Drop Maxim Levitsky's Reviewed-by because of significant changes.
- Remove redefine of _GNU_SOURCE. (kernel test robot)
- Rewrite changelog and test description. (Sean)
- Enable user space to set APIC bus frequency. (Sean)
- Use GUEST_ASSERT() from guest instead of TEST_ASSERT() from host. (Sean)
- Test xAPIC as well as x2APIC. (Sean)
- Add check that KVM_CAP_X86_APIC_BUS_CYCLES_NS cannot be set after vCPU created. (Sean)
- Use new udelay() utility. (Sean)
- Drop CLI. Mask LVT timer independently. (Sean)
- Use correct LVT timer entry (0x350 -> 0x320) to enable oneshot operation.
- Remove unnecessary static functions from single file test.
- Be consistent in types by using uint32_t/uint64_t instead of u32/u64.

Changes v6:
- Use vm_create() wrapper instead of open coding it. (Zide)
- Improve grammar of test description. (Zide)

Changes v5:
- Update to new name of capability KVM_CAP_X86_APIC_BUS_FREQUENCY -> KVM_CAP_X86_APIC_BUS_CYCLES_NS. (Xiaoyao Li)

Changes v4:
- Rework changelog.
- Add Sean's "Suggested-by" to acknowledge guidance received in
https://lore.kernel.org/all/[email protected]/
- Add copyright.
- Add test description to file header.
- Consistent capitalization for acronyms.
- Rebase to kvm-x86/next.
- Update to v4 change of providing bus clock rate in nanoseconds.
- Add a "TEST_REQUIRE()" for the new capability so that the test can work on kernels that do not support the new capability.
- Address checkpatch warnings and use tabs instead of spaces in header file to match existing code.

Changes v3:
- Use 1.5GHz instead of 1GHz as frequency.

Changes v2:
- Newly added.

---
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/include/x86_64/apic.h       |   8 +
 .../kvm/x86_64/apic_bus_clock_test.c          | 218 ++++++++++++++++++
 3 files changed, 227 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c
tools/testing/selftests/kvm/Makefile +1 -0
index ce8ff8e8ce3a..ad8b5d15f2bd 100644
@@ -112,6 +112,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/vmx_invalid_nested_guest_state
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_set_nested_state_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_tsc_adjust_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_nested_tsc_scaling_test
+TEST_GEN_PROGS_x86_64 += x86_64/apic_bus_clock_test
 TEST_GEN_PROGS_x86_64 += x86_64/xapic_ipi_test
 TEST_GEN_PROGS_x86_64 += x86_64/xapic_state_test
 TEST_GEN_PROGS_x86_64 += x86_64/xcr0_cpuid_test
tools/testing/selftests/kvm/include/x86_64/apic.h +8 -0
index bed316fdecd5..0f268b55fa06 100644
@@ -60,6 +60,14 @@
 #define		APIC_VECTOR_MASK	0x000FF
 #define	APIC_ICR2	0x310
 #define		SET_APIC_DEST_FIELD(x)	((x) << 24)
+#define APIC_LVTT	0x320
+#define		APIC_LVT_TIMER_ONESHOT		(0 << 17)
+#define		APIC_LVT_TIMER_PERIODIC		(1 << 17)
+#define		APIC_LVT_TIMER_TSCDEADLINE	(2 << 17)
+#define		APIC_LVT_MASKED			(1 << 16)
+#define	APIC_TMICT	0x380
+#define	APIC_TMCCT	0x390
+#define	APIC_TDCR	0x3E0
 
 void apic_disable(void);
 void xapic_enable(void);
tools/testing/selftests/kvm/x86_64/apic_bus_clock_test.c new file +218 -0

218 lines not on this page · raw message

Signature · 1 line
2.34.1