Skip to contentSkip to conversation

x86: add HPET counter tests

PATCH In review Igor Mammedov 2 participants 13 messages Jul 25, 2025 – Sep 8, 2025 KVM

lore
Share & tools
lore
#0 lore raw

v4 cover x86: add HPET counter tests

Changelog:
v4:
* fix smap/pku/pks tests that were failing CI
* on_cpus() wasn't running task on CPUs above 255, fix x86/smp code to correctly account for APIC IDs above 255
* bump max cpus limit to 1024
* hpet: count 0 latency as failure to avoid missing APs that doesn't run reader task
* hpet: replace on_cpus() with on_cpus_async() to to run reader task only on explicitly selected APs.
v3:
* fix test running long time due to control thread also running read test and stalling starting other threads
* improve latency accuracy
* increase max number of vcpus to 448
(that's what I had in hands for testing)

previous rev:
"[kvm-unit-tests PATCH v3 0/2] x86: add HPET counter tests"
https://yhbt.net/lore/all/[email protected]/T/#t

Igor Mammedov (5):
x86: resize id_map[] elements to u32
x86: fix APs with APIC ID more that 255 not showing in id_map
x86: move USERBASE to 32Mb in smap/pku/pks tests
x86: bump number of max cpus to 1024
x86: add HPET counter read micro benchmark and enable/disable torture
tests

 lib/x86/apic.h       |  2 +-
 lib/x86/smp.h        |  2 +-
 lib/x86/apic.c       |  2 +-
 lib/x86/setup.c      |  2 +-
 x86/Makefile.common  |  2 +
 x86/cstart.S         |  2 +-
 x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 x86/pks.c            |  2 +-
 x86/pku.c            |  2 +-
 x86/smap.c           |  2 +-
 10 files changed, 103 insertions(+), 8 deletions(-)
 create mode 100644 x86/hpet_read_test.c
Signature · 1 line
2.47.1
#1 lore raw

PATCH v4 1/5 x86: resize id_map[] elements to u32

currently item size is u8, which would truncate APIC IDs that are greater than 255.
Make it u32 so that it cna hold x2apic IDs as well.

---
 lib/x86/apic.h | 2 +-
 lib/x86/apic.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lib/x86/apic.h +1 -1
index cac6eab1..a76b1138 100644
@@ -8,7 +8,7 @@
 #include "processor.h"
 #include "smp.h"
 
-extern u8 id_map[MAX_TEST_CPUS];
+extern u32 id_map[MAX_TEST_CPUS];
 
 typedef struct {
     uint8_t vector;
lib/x86/apic.c +1 -1
index 0d151476..c538fb5f 100644
@@ -9,7 +9,7 @@
 static void *g_apic = (void *)APIC_DEFAULT_PHYS_BASE;
 static void *g_ioapic = (void *)IO_APIC_DEFAULT_PHYS_BASE;
 
-u8 id_map[MAX_TEST_CPUS];
+u32 id_map[MAX_TEST_CPUS];
 
 struct apic_ops {
 	u32 (*reg_read)(unsigned reg);
Signature · 1 line
2.47.1
Reviewed-by on v4 1/5
#2 lore raw

Re: [kvm-unit-tests PATCH v4 1/5] x86: resize id_map[] elements to u32

Quoted 5 lines · Igor Mammedov

currently item size is u8, which would truncate APIC IDs that are greater than 255.
Make it u32 so that it cna hold x2apic IDs as well.

Signature · 1 line
Peter Xu
#3 lore raw

PATCH v4 2/5 x86: fix APs with APIC ID more that 255 not showing in id_map

In reply to Igor Mammedov

save_id() is called too early and uses xapic ops, which is fine up to 2566 cpus.
However with that any CPU with APIC ID more than 255 will set wrong bit in online_cpus (since xapic ops only handle 8bit IDs, thus losing all higher bit x2apic might have).
As result CPUs with id higher than 255 are not registered (they will trumple over elements in range 0-255 instead).

To fix it move save_id() after the point where APs have switched to x2apic ops, to get non-truncated ID.

---
 lib/x86/setup.c | 2 +-
 x86/cstart.S    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lib/x86/setup.c +1 -1
index 122f0af3..c2f1c6d0 100644
@@ -391,9 +391,9 @@ void ap_start64(void)
 	setup_gdt_tss();
 	reset_apic();
 	load_idt();
-	save_id();
 	enable_apic();
 	enable_x2apic();
+	save_id();
 	ap_online();
 }
 
x86/cstart.S +1 -1
index 8fb7bdef..dafb330d 100644
@@ -99,9 +99,9 @@ ap_start32:
 	call load_idt
 	call prepare_32
 	call reset_apic
-	call save_id
 	call enable_apic
 	call enable_x2apic
+	call save_id
 	call ap_online
 
 	/* ap_online() should never return */
Signature · 1 line
2.47.1
Reviewed-by on v4 2/5
#4 lore raw

Re: [kvm-unit-tests PATCH v4 2/5] x86: fix APs with APIC ID more that 255 not showing in id_map

Quoted 8 lines · Igor Mammedov

save_id() is called too early and uses xapic ops, which is fine up to 2566 cpus.
However with that any CPU with APIC ID more than 255 will set wrong bit in online_cpus (since xapic ops only handle 8bit IDs, thus losing all higher bit x2apic might have).
As result CPUs with id higher than 255 are not registered (they will trumple over elements in range 0-255 instead).

Quoted 4 lines

To fix it move save_id() after the point where APs have switched to x2apic ops, to get non-truncated ID.

Signature · 1 line
Peter Xu
#5 lore raw

PATCH v4 3/5 x86: move USERBASE to 32Mb in smap/pku/pks tests

In reply to Igor Mammedov

If number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.

Move pages used by tests to 32Mb to fix it.

---
 x86/pks.c  | 2 +-
 x86/pku.c  | 2 +-
 x86/smap.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
x86/pks.c +1 -1
index f4d6ac83..9b9519ba 100644
@@ -6,7 +6,7 @@
 #include "x86/msr.h"
 
 #define PTE_PKEY_BIT     59
-#define SUPER_BASE        (1 << 23)
+#define SUPER_BASE        (2 << 24)
 #define SUPER_VAR(v)      (*((__typeof__(&(v))) (((unsigned long)&v) + SUPER_BASE)))
 
 volatile int pf_count = 0;
x86/pku.c +1 -1
index 6c0d72cc..544c6f24 100644
@@ -6,7 +6,7 @@
 #include "x86/msr.h"
 
 #define PTE_PKEY_BIT     59
-#define USER_BASE        (1 << 23)
+#define USER_BASE        (2 << 24)
 #define USER_VAR(v)      (*((__typeof__(&(v))) (((unsigned long)&v) + USER_BASE)))
 
 volatile int pf_count = 0;
x86/smap.c +1 -1
index 9a823a55..53c9f4ce 100644
@@ -45,7 +45,7 @@ asm ("pf_tss:\n"
         "jmp pf_tss\n\t");
 
 
-#define USER_BASE	(1 << 23)
+#define USER_BASE	(2 << 24)
 #define USER_VAR(v)	(*((__typeof__(&(v))) (((unsigned long)&v) + USER_BASE)))
 #define USER_ADDR(v)   ((void *)((unsigned long)(&v) + USER_BASE))
 
Signature · 1 line
2.47.1
Reviewed-by on v4 3/5
#6 lore raw

Re: [kvm-unit-tests PATCH v4 3/5] x86: move USERBASE to 32Mb in smap/pku/pks tests

Quoted 7 lines · Igor Mammedov

If number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.

Move pages used by tests to 32Mb to fix it.

Quoted 11 lines
---
 x86/pks.c  | 2 +-
 x86/pku.c  | 2 +-
 x86/smap.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
x86/pks.c +0 -0
index f4d6ac83..9b9519ba 100644

x86/pks.c line 9

Quoted 5 lines
x86/pks.c +1 -1
@@ -6,7 +6,7 @@
 #include "x86/msr.h"
 
 #define PTE_PKEY_BIT     59
-#define SUPER_BASE        (1 << 23)
+#define SUPER_BASE        (2 << 24)

Nitpick: maybe 1<<25 would be easier to read.

Below are some random thoughts when reading these tests..

I'm not sure whether I understand them correctly here: all of them so far depend on the "test" var present in the .bss section, and they all assumed that the var's physical address (likely together with the whole .bss) will be under SUPER_BASE after loaded in the VM.

Based on that, there's yet another restriction versus the need to reserve (SUPER_BASE, SUPER_BASE*2), because the tests want to map the same (0, SUPER_BASE) memory twice in that virtual address range, so here the tests do not really need the phys pages in the back but kind of a way to reserve virtual addresses..

Instead of these tricks, I wonder whether we can do alloc_page() once, then put the test var on the page allocated. Then we can build the required PKU/PKS/SMAP special pgtables on top, mapping to the page allocated. It should make sure system changes (like growing num_cpus) never affect it anymore.

That (even if feasible.. or maybe I missed something) can definitely involve more changes, so not something to ask for on adding the hpet test / x2apic support. Just some pure thoughts when reading it..

Thanks,

Signature · 1 line
Peter Xu
#7 lore raw

Re: [kvm-unit-tests PATCH v4 3/5] x86: move USERBASE to 32Mb in smap/pku/pks tests

Quoted 24 lines · On Wed, 30 Jul 2025 16:56:18 -0400 Peter Xu <[email protected]> wrote:
Quoted 7 lines · Igor Mammedov

If number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.

Move pages used by tests to 32Mb to fix it.

Quoted 13 lines
---
 x86/pks.c  | 2 +-
 x86/pku.c  | 2 +-
 x86/smap.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
x86/pks.c +0 -0
index f4d6ac83..9b9519ba 100644
@@ -6,7 +6,7 @@
 #include "x86/msr.h"
 
Quoted 5 lines

x86/pks.c line 9

Quoted 3 lines
x86/pks.c +1 -1
 #define PTE_PKEY_BIT     59
-#define SUPER_BASE        (1 << 23)
+#define SUPER_BASE        (2 << 24)  

Nitpick: maybe 1<<25 would be easier to read.

I can try with, if I have to respin.

Quoted 19 lines

Below are some random thoughts when reading these tests..

I'm not sure whether I understand them correctly here: all of them so far depend on the "test" var present in the .bss section, and they all assumed that the var's physical address (likely together with the whole .bss) will be under SUPER_BASE after loaded in the VM.

Based on that, there's yet another restriction versus the need to reserve (SUPER_BASE, SUPER_BASE*2), because the tests want to map the same (0, SUPER_BASE) memory twice in that virtual address range, so here the tests do not really need the phys pages in the back but kind of a way to reserve virtual addresses..

Instead of these tricks, I wonder whether we can do alloc_page() once, then put the test var on the page allocated. Then we can build the required PKU/PKS/SMAP special pgtables on top, mapping to the page allocated. It should make sure system changes (like growing num_cpus) never affect it anymore.

Quoted 3 lines

That (even if feasible.. or maybe I missed something) can definitely involve more changes, so not something to ask for on adding the hpet test / x2apic support. Just some pure thoughts when reading it..

I had the same thoughts but I'm not confident enough to rewrite those tests. Hence a knee-jerk reaction of just fixing up base to unbreak affected test.

Quoted 3 lines

Thanks,

#8 lore raw

PATCH v4 4/5 x86: bump number of max cpus to 1024

In reply to Igor Mammedov

this should allow run tests with more thatn 256 cpus

---
 lib/x86/smp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/x86/smp.h +1 -1
index bbe90daa..272aa5ee 100644
@@ -1,7 +1,7 @@
 #ifndef _X86_SMP_H_
 #define _X86_SMP_H_
 
-#define MAX_TEST_CPUS (255)
+#define MAX_TEST_CPUS (1024)
 
 /*
  * Allocate 12KiB of data for per-CPU usage.  One page for per-CPU data, and
Signature · 1 line
2.47.1
#10 lore raw

PATCH v4 5/5 x86: add HPET counter read micro benchmark and enable/disable torture tests

In reply to Igor Mammedov

test is to be used for benchmarking/validating HPET main counter reading

how to run:
QEMU=/foo/qemu-system-x86_64 x86/run x86/hpet_read_test.flat -smp X
where X is desired number of logical CPUs to test with

it will 1st execute concurrent read benchmark
and after that it will run torture test enabling/disabling HPET counter, while running readers in parallel. Goal is to verify counter that always goes up.

---
v4:
* use global for test cycles and siplify code a bit
* use cpu number instead of APCI ID as index into latency array
* report failure if a cpu measured 0 latency
* replace on_cpus() with on_cpu_async() to avoid BSP interrupting itself
* drop volatile

v3:
* measure lat inside threads so that, threads startup time wouldn't throw off results
* fix BSP iterrupting itself by running read test and stalling other cpus as result. (fix it by exiting read test earlier if it's running on BSP)
v2:
* fix broken timer going backwards check
* report # of fails
* warn if number of vcpus is not sufficient for torture test and skip it
* style fixups

---
 x86/Makefile.common  |  2 +
 x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 x86/hpet_read_test.c
x86/Makefile.common +2 -0
index 5663a65d..ef0e09a6 100644
@@ -101,6 +101,8 @@ tests-common += $(TEST_DIR)/realmode.$(exe) \
 realmode_bits := $(if $(call cc-option,-m16,""),16,32)
 endif
 
+tests-common += $(TEST_DIR)/hpet_read_test.$(exe)
+
 test_cases: $(tests-common) $(tests)
 
 $(TEST_DIR)/%.o: CFLAGS += -std=gnu99 -ffreestanding -I $(SRCDIR)/lib -I $(SRCDIR)/lib/x86 -I lib
x86/hpet_read_test.c new file +93 -0

93 lines not on this page · raw message

Signature · 1 line
2.47.1
Reviewed-by on v4 5/5
#11 lore raw

Re: [kvm-unit-tests PATCH v4 5/5] x86: add HPET counter read micro benchmark and enable/disable torture tests

Quoted 138 lines · Igor Mammedov

test is to be used for benchmarking/validating HPET main counter reading

how to run:

   QEMU=/foo/qemu-system-x86_64 x86/run x86/hpet_read_test.flat -smp X

where X is desired number of logical CPUs to test with

it will 1st execute concurrent read benchmark
and after that it will run torture test enabling/disabling HPET counter, while running readers in parallel. Goal is to verify counter that always goes up.

---
v4:
* use global for test cycles and siplify code a bit
* use cpu number instead of APCI ID as index into latency array
* report failure if a cpu measured 0 latency
* replace on_cpus() with on_cpu_async() to avoid BSP interrupting itself
* drop volatile

v3:
* measure lat inside threads so that, threads startup time wouldn't throw off results
* fix BSP iterrupting itself by running read test and stalling other cpus as result. (fix it by exiting read test earlier if it's running on BSP)
v2:
* fix broken timer going backwards check
* report # of fails
* warn if number of vcpus is not sufficient for torture test and skip it
* style fixups

---
 x86/Makefile.common  |  2 +
 x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 x86/hpet_read_test.c
x86/Makefile.common +2 -0
index 5663a65d..ef0e09a6 100644
@@ -101,6 +101,8 @@ tests-common += $(TEST_DIR)/realmode.$(exe) \
 realmode_bits := $(if $(call cc-option,-m16,""),16,32)
 endif
 
+tests-common += $(TEST_DIR)/hpet_read_test.$(exe)
+
 test_cases: $(tests-common) $(tests)
 
 $(TEST_DIR)/%.o: CFLAGS += -std=gnu99 -ffreestanding -I $(SRCDIR)/lib -I $(SRCDIR)/lib/x86 -I lib
x86/hpet_read_test.c new file +80 -0
new file mode 100644
index 00000000..a44cdac2
@@ -0,0 +1,93 @@
+#include "libcflat.h"
+#include "smp.h"
+#include "apic.h"
+#include "asm/barrier.h"
+#include "x86/atomic.h"
+#include "vmalloc.h"
+#include "alloc.h"
+
+#define HPET_ADDR         0xFED00000L
+#define HPET_COUNTER_ADDR ((uint8_t *)HPET_ADDR + 0xF0UL)
+#define HPET_CONFIG_ADDR  ((uint8_t *)HPET_ADDR + 0x10UL)
+#define HPET_ENABLE_BIT   0x01UL
+#define HPET_CLK_PERIOD   10
+
+#define TEST_CYCLES 100000
+
+static atomic_t fail;
+static uint64_t latency[MAX_TEST_CPUS];
+
+static void hpet_reader(void *data)
+{
+	long i;
+	uint64_t old_counter = 0, new_counter;
+	long id = (long)data;
+
+	latency[id] = *(uint64_t *)HPET_COUNTER_ADDR;
+	for (i = 0; i < TEST_CYCLES; ++i) {
+		new_counter = *(uint64_t *)HPET_COUNTER_ADDR;
+		if (new_counter < old_counter)
+			atomic_inc(&fail);
+		old_counter = new_counter;
+	}
+	/* claculate job latency in ns */
+	latency[id] = (*(uint64_t *)HPET_COUNTER_ADDR - latency[id])
+		* HPET_CLK_PERIOD / TEST_CYCLES;
+}
+
+static void hpet_writer(void *data)
+{
+	int i;
+
+	for (i = 0; i < TEST_CYCLES; ++i)
+		if (i % 2)
+			*(uint64_t *)HPET_CONFIG_ADDR |= HPET_ENABLE_BIT;
+		else
+			*(uint64_t *)HPET_CONFIG_ADDR &= ~HPET_ENABLE_BIT;
+}
+
+int main(void)
+{
+	unsigned long cpu, time_ns, lat = 0;
+	uint64_t start, end;
+	int ncpus = cpu_count();
+
+	do {
+		printf("* starting concurrent read bench on %d cpus\n", ncpus);
+		*(uint64_t *)HPET_CONFIG_ADDR |= HPET_ENABLE_BIT;
+		start = *(uint64_t *)HPET_COUNTER_ADDR;
+
+		for (cpu = cpu_count() - 1; cpu > 0; --cpu)
+			on_cpu_async(cpu, hpet_reader, (void *)cpu);
+		while (cpus_active() > 1)
+			pause();
+
+		end = (*(uint64_t *)HPET_COUNTER_ADDR);
+		time_ns = (end - start) * HPET_CLK_PERIOD;
+
+		for (cpu = 1; cpu < ncpus; cpu++)
+			if (latency[cpu])
+				lat += latency[cpu];
+			else
+				report_fail("cpu %lu reported invalid latency (0)\n", cpu);
+		lat = lat / ncpus;
+
+		report(time_ns && !atomic_read(&fail),
+			"read test took %lu ms, avg read: %lu ns\n", time_ns/1000000,  lat);
+	} while (0);
+
+	do {
+		printf("* starting enable/disable with concurrent readers torture\n");

x86/hpet_read_test.c line 90

Quoted 10 lines
x86/hpet_read_test.c new file +10 -0
+		if (ncpus > 2) {
+			for (cpu = 2; cpu < ncpus; cpu++)
+				on_cpu_async(cpu, hpet_reader, (void *)TEST_CYCLES);
+
+			on_cpu(1, hpet_writer, (void *)TEST_CYCLES);
+			report(!atomic_read(&fail), "torture test, fails: %u\n",
+				atomic_read(&fail));
+		} else
+			printf("SKIP: torture test: '-smp X' should be greater than 2\n");
+	} while (0);

Nitpick: IMHO the "do... while" indentation isn't strictly needed. Maybe it implies the blobs can be put separately into two functions, e.g. as test_hpet_concurrent_ro() / test_hpet_concurrent_rw().

Quoted 6 lines
x86/hpet_read_test.c new file +3 -0
+
+	return report_summary();
+}
Signature · 1 line
2.47.1
Signature · 1 line
Peter Xu
#12 lore raw
In reply to Igor Mammedov

On Fri, 25 Jul 2025 11:54:24 +0200
Igor Mammedov <[email protected]> wrote:

Paolo,
gentle ping

Quoted 43 lines

Changelog:
v4:
* fix smap/pku/pks tests that were failing CI
* on_cpus() wasn't running task on CPUs above 255, fix x86/smp code to correctly account for APIC IDs above 255
* bump max cpus limit to 1024
* hpet: count 0 latency as failure to avoid missing APs that doesn't run reader task
* hpet: replace on_cpus() with on_cpus_async() to to run reader task only on explicitly selected APs.
v3:
* fix test running long time due to control thread also running read test and stalling starting other threads
* improve latency accuracy
* increase max number of vcpus to 448
(that's what I had in hands for testing)

previous rev:
"[kvm-unit-tests PATCH v3 0/2] x86: add HPET counter tests"
https://yhbt.net/lore/all/[email protected]/T/#t

Igor Mammedov (5):
x86: resize id_map[] elements to u32
x86: fix APs with APIC ID more that 255 not showing in id_map
x86: move USERBASE to 32Mb in smap/pku/pks tests
x86: bump number of max cpus to 1024
x86: add HPET counter read micro benchmark and enable/disable torture
tests

 lib/x86/apic.h       |  2 +-
 lib/x86/smp.h        |  2 +-
 lib/x86/apic.c       |  2 +-
 lib/x86/setup.c      |  2 +-
 x86/Makefile.common  |  2 +
 x86/cstart.S         |  2 +-
 x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 x86/pks.c            |  2 +-
 x86/pku.c            |  2 +-
 x86/smap.c           |  2 +-
 10 files changed, 103 insertions(+), 8 deletions(-)
 create mode 100644 x86/hpet_read_test.c