x86: add HPET counter tests
v4 cover x86: add HPET counter tests
Changelog:
v4:
* fix smap/pku/pks tests that were failing CI
* on_cpus() wasn't running task on CPUs above 255, fix x86/smp code to correctly account for APIC IDs above 255
* bump max cpus limit to 1024
* hpet: count 0 latency as failure to avoid missing APs that doesn't run reader task
* hpet: replace on_cpus() with on_cpus_async() to to run reader task only on explicitly selected APs.
v3:
* fix test running long time due to control thread also running read test and stalling starting other threads
* improve latency accuracy
* increase max number of vcpus to 448
(that's what I had in hands for testing)
previous rev:
"[kvm-unit-tests PATCH v3 0/2] x86: add HPET counter tests"
https://yhbt.net/lore/all/[email protected]/T/#t
Igor Mammedov (5):
x86: resize id_map[] elements to u32
x86: fix APs with APIC ID more that 255 not showing in id_map
x86: move USERBASE to 32Mb in smap/pku/pks tests
x86: bump number of max cpus to 1024
x86: add HPET counter read micro benchmark and enable/disable torture
tests
lib/x86/apic.h | 2 +- lib/x86/smp.h | 2 +- lib/x86/apic.c | 2 +- lib/x86/setup.c | 2 +- x86/Makefile.common | 2 + x86/cstart.S | 2 +- x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++ x86/pks.c | 2 +- x86/pku.c | 2 +- x86/smap.c | 2 +- 10 files changed, 103 insertions(+), 8 deletions(-) create mode 100644 x86/hpet_read_test.c
Signature · 1 line
2.47.1
PATCH v4 1/5 x86: resize id_map[] elements to u32
currently item size is u8, which would truncate APIC IDs that are greater than 255.
Make it u32 so that it cna hold x2apic IDs as well.
--- lib/x86/apic.h | 2 +- lib/x86/apic.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/x86/apic.h +1 -1
@@ -8,7 +8,7 @@#include "processor.h" #include "smp.h" -extern u8 id_map[MAX_TEST_CPUS ]; +extern u32 id_map[MAX_TEST_CPUS ]; typedef struct { uint8_t vector;
lib/x86/apic.c +1 -1
@@ -9,7 +9,7 @@static void *g_apic = (void *)APIC_DEFAULT_PHYS_BASE ;static void *g_ioapic = (void *)IO_APIC_DEFAULT_PHYS_BASE ; -u8 id_map[MAX_TEST_CPUS ]; +u32 id_map[MAX_TEST_CPUS ];struct apic_ops {u32 (*reg_read)(unsigned reg);
Signature · 1 line
2.47.1
Re: [kvm-unit-tests PATCH v4 1/5] x86: resize id_map[] elements to u32
Quoted 5 lines · Igor Mammedov
currently item size is u8, which would truncate APIC IDs that are greater than 255.
Make it u32 so that it cna hold x2apic IDs as well.Signed-off-by Igor Mammedov <[email protected]>
Signature · 1 line
Peter Xu
PATCH v4 2/5 x86: fix APs with APIC ID more that 255 not showing in id_map
In reply to Igor Mammedovsave_id() is called too early and uses xapic ops, which is fine up to 2566 cpus.
However with that any CPU with APIC ID more than 255 will set wrong bit in online_cpus (since xapic ops only handle 8bit IDs, thus losing all higher bit x2apic might have).
As result CPUs with id higher than 255 are not registered (they will trumple over elements in range 0-255 instead).
To fix it move save_id() after the point where APs have switched to x2apic ops, to get non-truncated ID.
--- lib/x86/setup.c | 2 +- x86/cstart.S | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/x86/setup.c +1 -1
@@ -391,9 +391,9 @@ void ap_start64(void)setup_gdt_tss ();reset_apic ();load_idt (); -save_id ();enable_apic ();enable_x2apic (); +save_id ();ap_online (); }
x86/cstart.S +1 -1
@@ -99,9 +99,9 @@ ap_start32: call load_idt call prepare_32 call reset_apic - call save_id call enable_apic call enable_x2apic + call save_id call ap_online /* ap_online() should never return */
Signature · 1 line
2.47.1
Re: [kvm-unit-tests PATCH v4 2/5] x86: fix APs with APIC ID more that 255 not showing in id_map
Quoted 8 lines · Igor Mammedov
save_id()is called too early and uses xapic ops, which is fine up to 2566 cpus.
However with that any CPU with APIC ID more than 255 will set wrong bit inonline_cpus(since xapic ops only handle 8bit IDs, thus losing all higher bit x2apic might have).
As result CPUs with id higher than 255 are not registered (they will trumple over elements in range 0-255 instead).
Quoted 4 lines
To fix it move
save_id()after the point where APs have switched to x2apic ops, to get non-truncated ID.Signed-off-by Igor Mammedov <[email protected]>
Signature · 1 line
Peter Xu
PATCH v4 3/5 x86: move USERBASE to 32Mb in smap/pku/pks tests
In reply to Igor MammedovIf number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.
Move pages used by tests to 32Mb to fix it.
--- x86/pks.c | 2 +- x86/pku.c | 2 +- x86/smap.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
x86/pks.c +1 -1
@@ -6,7 +6,7 @@#include "x86/msr.h" #define PTE_PKEY_BIT 59 -#define SUPER_BASE (1 << 23) +#define SUPER_BASE (2 << 24)#define SUPER_VAR (v) (*((__typeof__(&(v))) (((unsigned long)&v) + SUPER_BASE)))volatile int pf_count =0 ;
x86/pku.c +1 -1
@@ -6,7 +6,7 @@#include "x86/msr.h" #define PTE_PKEY_BIT 59 -#define USER_BASE (1 << 23) +#define USER_BASE (2 << 24)#define USER_VAR (v) (*((__typeof__(&(v))) (((unsigned long)&v) + USER_BASE)))volatile int pf_count =0 ;
x86/smap.c +1 -1
@@ -45,7 +45,7 @@ asm ("pf_tss:\n" "jmp pf_tss\n\t"); -#define USER_BASE (1 << 23) +#define USER_BASE (2 << 24)#define USER_VAR (v) (*((__typeof__(&(v))) (((unsigned long)&v) + USER_BASE)))#define USER_ADDR (v) ((void *)((unsigned long)(&v) + USER_BASE))
Signature · 1 line
2.47.1
Re: [kvm-unit-tests PATCH v4 3/5] x86: move USERBASE to 32Mb in smap/pku/pks tests
Quoted 7 lines · Igor Mammedov
If number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.
Move pages used by tests to 32Mb to fix it.
Signed-off-by Igor Mammedov <[email protected]>
Quoted 11 lines
--- x86/pks.c | 2 +- x86/pku.c | 2 +- x86/smap.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)x86/pks.c +0
-0
x86/pks.c line 9
Quoted 5 lines
x86/pks.c +1
-1@@ -6,7 +6,7 @@#include "x86/msr.h" #define PTE_PKEY_BIT 59 -#define SUPER_BASE (1 << 23) +#define SUPER_BASE (2 << 24)
Nitpick: maybe 1<<25 would be easier to read.
Below are some random thoughts when reading these tests..
I'm not sure whether I understand them correctly here: all of them so far depend on the "test" var present in the .bss section, and they all assumed that the var's physical address (likely together with the whole .bss) will be under SUPER_BASE after loaded in the VM.
Based on that, there's yet another restriction versus the need to reserve (SUPER_BASE, SUPER_BASE*2), because the tests want to map the same (0, SUPER_BASE) memory twice in that virtual address range, so here the tests do not really need the phys pages in the back but kind of a way to reserve virtual addresses..
Instead of these tricks, I wonder whether we can do alloc_page() once, then put the test var on the page allocated. Then we can build the required PKU/PKS/SMAP special pgtables on top, mapping to the page allocated. It should make sure system changes (like growing num_cpus) never affect it anymore.
That (even if feasible.. or maybe I missed something) can definitely involve more changes, so not something to ask for on adding the hpet test / x2apic support. Just some pure thoughts when reading it..
Thanks,
Signature · 1 line
Peter Xu
Re: [kvm-unit-tests PATCH v4 3/5] x86: move USERBASE to 32Mb in smap/pku/pks tests
Quoted 24 lines · On Wed, 30 Jul 2025 16:56:18 -0400 Peter Xu <[email protected]> wrote:
Quoted 7 lines · Igor Mammedov
If number of CPUs is increased up to 2048, it will push available pages above 16Mb range and make smap/pku/pks tests fail with 'Could not reserve memory' error.
Move pages used by tests to 32Mb to fix it.
Signed-off-by Igor Mammedov <[email protected]>Reviewed-by Peter Xu <[email protected]>Quoted 13 lines
--- x86/pks.c | 2 +- x86/pku.c | 2 +- x86/smap.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)x86/pks.c +0
-0@@ -6,7 +6,7 @@#include "x86/msr.h"
Quoted 5 lines
x86/pks.c line 9
Quoted 3 lines
x86/pks.c +1
-1#define PTE_PKEY_BIT 59 -#define SUPER_BASE (1 << 23) +#define SUPER_BASE (2 << 24)Nitpick: maybe 1<<25 would be easier to read.
I can try with, if I have to respin.
Quoted 19 lines
Below are some random thoughts when reading these tests..
I'm not sure whether I understand them correctly here: all of them so far depend on the "test" var present in the .bss section, and they all assumed that the var's physical address (likely together with the whole .bss) will be under
SUPER_BASEafter loaded in the VM.Based on that, there's yet another restriction versus the need to reserve (
SUPER_BASE, SUPER_BASE*2), because the tests want to map the same (0,SUPER_BASE) memory twice in that virtual address range, so here the tests do not really need the phys pages in the back but kind of a way to reserve virtual addresses..Instead of these tricks, I wonder whether we can do
alloc_page()once, then put the test var on the page allocated. Then we can build the required PKU/PKS/SMAP special pgtables on top, mapping to the page allocated. It should make sure system changes (like growingnum_cpus) never affect it anymore.
Quoted 3 lines
That (even if feasible.. or maybe I missed something) can definitely involve more changes, so not something to ask for on adding the hpet test / x2apic support. Just some pure thoughts when reading it..
I had the same thoughts but I'm not confident enough to rewrite those tests. Hence a knee-jerk reaction of just fixing up base to unbreak affected test.
Quoted 3 lines
Thanks,
PATCH v4 4/5 x86: bump number of max cpus to 1024
In reply to Igor Mammedovthis should allow run tests with more thatn 256 cpus
--- lib/x86/smp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/x86/smp.h +1 -1
@@ -1,7 +1,7 @@#ifndef _X86_SMP_H_#define _X86_SMP_H_ -#define MAX_TEST_CPUS (255) +#define MAX_TEST_CPUS (1024)/* * Allocate 12KiB of data for per-CPU usage. One page for per-CPU data, and
Signature · 1 line
2.47.1
Re: [kvm-unit-tests PATCH v4 4/5] x86: bump number of max cpus to 1024
Quoted 3 lines · Igor Mammedov
this should allow run tests with more thatn 256 cpus
Signed-off-by Igor Mammedov <[email protected]>
Signature · 1 line
Peter Xu
PATCH v4 5/5 x86: add HPET counter read micro benchmark and enable/disable torture tests
In reply to Igor Mammedovtest is to be used for benchmarking/validating HPET main counter reading
how to run:
QEMU=/foo/qemu-system-x86_64 x86/run x86/hpet_read_test.flat -smp X
where X is desired number of logical CPUs to test with
it will 1st execute concurrent read benchmark
and after that it will run torture test enabling/disabling HPET counter, while running readers in parallel. Goal is to verify counter that always goes up.
---
v4:
* use global for test cycles and siplify code a bit
* use cpu number instead of APCI ID as index into latency array
* report failure if a cpu measured 0 latency
* replace on_cpus() with on_cpu_async() to avoid BSP interrupting itself
* drop volatile
v3:
* measure lat inside threads so that, threads startup time wouldn't throw off results
* fix BSP iterrupting itself by running read test and stalling other cpus as result. (fix it by exiting read test earlier if it's running on BSP)
v2:
* fix broken timer going backwards check
* report # of fails
* warn if number of vcpus is not sufficient for torture test and skip it
* style fixups
--- x86/Makefile.common | 2 + x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 x86/hpet_read_test.c
x86/Makefile.common +2 -0
@@ -101,6 +101,8 @@ tests-common += $(TEST_DIR)/realmode.$(exe) \ realmode_bits := $(if $(call cc-option,-m16,""),16,32) endif +tests-common += $(TEST_DIR)/hpet_read_test.$(exe) + test_cases: $(tests-common) $(tests) $(TEST_DIR)/%.o: CFLAGS += -std=gnu99 -ffreestanding -I $(SRCDIR)/lib -I $(SRCDIR)/lib/x86 -I lib
x86/hpet_read_test.c new file +93 -0
93 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [kvm-unit-tests PATCH v4 5/5] x86: add HPET counter read micro benchmark and enable/disable torture tests
Quoted 138 lines · Igor Mammedov
test is to be used for benchmarking/validating HPET main counter reading
how to run:
QEMU=/foo/qemu-system-x86_64 x86/run x86/hpet_read_test.flat -smp Xwhere X is desired number of logical CPUs to test with
it will 1st execute concurrent read benchmark
and after that it will run torture test enabling/disabling HPET counter, while running readers in parallel. Goal is to verify counter that always goes up.Signed-off-by Igor Mammedov <[email protected]>---
v4:
* use global for test cycles and siplify code a bit
* use cpu number instead of APCI ID as index into latency array
* report failure if a cpu measured 0 latency
* replaceon_cpus()withon_cpu_async()to avoid BSP interrupting itself
* drop volatilev3:
* measure lat inside threads so that, threads startup time wouldn't throw off results
* fix BSP iterrupting itself by running read test and stalling other cpus as result. (fix it by exiting read test earlier if it's running on BSP)
v2:
* fix broken timer going backwards check
* report # of fails
* warn if number of vcpus is not sufficient for torture test and skip it
* style fixups--- x86/Makefile.common | 2 + x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 x86/hpet_read_test.cx86/Makefile.common +2
-0@@ -101,6 +101,8 @@ tests-common += $(TEST_DIR)/realmode.$(exe) \ realmode_bits := $(if $(call cc-option,-m16,""),16,32) endif +tests-common += $(TEST_DIR)/hpet_read_test.$(exe) + test_cases: $(tests-common) $(tests) $(TEST_DIR)/%.o: CFLAGS += -std=gnu99 -ffreestanding -I $(SRCDIR)/lib -I $(SRCDIR)/lib/x86 -I libx86/hpet_read_test.c new file +80
-0@@ -0,0 +1,93 @@ +#include "libcflat.h" +#include "smp.h" +#include "apic.h" +#include "asm/barrier.h" +#include "x86/atomic.h" +#include "vmalloc.h" +#include "alloc.h" + +#define HPET_ADDR 0xFED00000L +#define HPET_COUNTER_ADDR ((uint8_t *)HPET_ADDR + 0xF0UL) +#define HPET_CONFIG_ADDR ((uint8_t *)HPET_ADDR + 0x10UL) +#define HPET_ENABLE_BIT 0x01UL +#define HPET_CLK_PERIOD 10 + +#define TEST_CYCLES 100000 + +static atomic_t fail; +static uint64_t latency[MAX_TEST_CPUS ]; + +static void hpet_reader (void *data) +{ +long i; +uint64_t old_counter =0 , new_counter; +long id = (long )data; + + latency[id] = *(uint64_t *)HPET_COUNTER_ADDR ; +for (i =0 ; i <TEST_CYCLES ; ++i) { + new_counter = *(uint64_t *)HPET_COUNTER_ADDR ; +if (new_counter < old_counter) +atomic_inc (&fail); + old_counter = new_counter; + } +/* claculate job latency in ns */ + latency[id] = (*(uint64_t *)HPET_COUNTER_ADDR - latency[id]) + *HPET_CLK_PERIOD /TEST_CYCLES ; +} + +static void hpet_writer (void *data) +{ +int i; + +for (i =0 ; i <TEST_CYCLES ; ++i) +if (i %2 ) + *(uint64_t *)HPET_CONFIG_ADDR |=HPET_ENABLE_BIT ; +else + *(uint64_t *)HPET_CONFIG_ADDR &= ~HPET_ENABLE_BIT ; +} + +int main (void ) +{ +unsigned long cpu, time_ns, lat =0 ; +uint64_t start, end; +int ncpus =cpu_count (); + +do { +printf ("* starting concurrent read bench on %d cpus\n" , ncpus); + *(uint64_t *)HPET_CONFIG_ADDR |=HPET_ENABLE_BIT ; + start = *(uint64_t *)HPET_COUNTER_ADDR ; + +for (cpu =cpu_count () -1 ; cpu >0 ; --cpu) +on_cpu_async (cpu, hpet_reader, (void *)cpu); +while (cpus_active () >1 ) +pause (); + + end = (*(uint64_t *)HPET_COUNTER_ADDR ); + time_ns = (end - start) *HPET_CLK_PERIOD ; + +for (cpu =1 ; cpu < ncpus; cpu++) +if (latency[cpu]) + lat += latency[cpu]; +else +report_fail ("cpu %lu reported invalid latency (0)\n" , cpu); + lat = lat / ncpus; + +report (time_ns && !atomic_read (&fail), +"read test took %lu ms, avg read: %lu ns\n" , time_ns/1000000 , lat); + }while (0 ); + + do { + printf("* starting enable/disable with concurrent readers torture\n");
x86/hpet_read_test.c line 90
Quoted 10 lines
x86/hpet_read_test.c new file +10
-0+if (ncpus >2 ) { +for (cpu =2 ; cpu < ncpus; cpu++) +on_cpu_async (cpu, hpet_reader, (void *)TEST_CYCLES ); + +on_cpu (1 , hpet_writer, (void *)TEST_CYCLES ); +report (!atomic_read (&fail),"torture test, fails: %u\n" , +atomic_read (&fail)); + }else +printf ("SKIP: torture test: '-smp X' should be greater than 2\n" ); + }while (0 );
Nitpick: IMHO the "do... while" indentation isn't strictly needed. Maybe it implies the blobs can be put separately into two functions, e.g. as test_hpet_concurrent_ro() / test_hpet_concurrent_rw().
Quoted 6 lines
x86/hpet_read_test.c new file +3
-0+ +return report_summary (); +}Signature · 1 line
2.47.1
Signature · 1 line
Peter Xu
On Fri, 25 Jul 2025 11:54:24 +0200
Igor Mammedov <[email protected]> wrote:
Paolo,
gentle ping
Quoted 43 lines
Changelog:
v4:
* fix smap/pku/pks tests that were failing CI
*on_cpus()wasn't running task on CPUs above 255, fix x86/smp code to correctly account for APIC IDs above 255
* bump max cpus limit to 1024
* hpet: count 0 latency as failure to avoid missing APs that doesn't run reader task
* hpet: replaceon_cpus()withon_cpus_async()to to run reader task only on explicitly selected APs.
v3:
* fix test running long time due to control thread also running read test and stalling starting other threads
* improve latency accuracy
* increase max number of vcpus to 448
(that's what I had in hands for testing)previous rev:
"[kvm-unit-tests PATCH v3 0/2] x86: add HPET counter tests"
https://yhbt.net/lore/all/[email protected]/T/#tIgor Mammedov (5):
x86: resize id_map[] elements to u32
x86: fix APs with APIC ID more that 255 not showing inid_map
x86: move USERBASE to 32Mb in smap/pku/pks tests
x86: bump number of max cpus to 1024
x86: add HPET counter read micro benchmark and enable/disable torture
testslib/x86/apic.h | 2 +- lib/x86/smp.h | 2 +- lib/x86/apic.c | 2 +- lib/x86/setup.c | 2 +- x86/Makefile.common | 2 + x86/cstart.S | 2 +- x86/hpet_read_test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++ x86/pks.c | 2 +- x86/pku.c | 2 +- x86/smap.c | 2 +- 10 files changed, 103 insertions(+), 8 deletions(-) create mode 100644 x86/hpet_read_test.c