x86: fix async page fault issues
cover x86: fix async page fault issues
When running asyncpf test, it gets skipped without a clear reason:
./asyncpf
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 107f000
cr4 = 20
install handler
enable async pf
alloc memory
start loop
end loop
start loop
end loop
SUMMARY: 0 tests
SKIP asyncpf (0 tests)
The reason is that KVM changed to use interrupt-based 'page-ready' notification and abandoned #PF-based 'page-ready' notification mechanism. Interrupt-based 'page-ready' notification requires KVM_ASYNC_PF_DELIVERY_AS_INT to be set as well in MSR_KVM_ASYNC_PF_EN to enable asyncpf.
This series tries to fix the problem by separating two testcases for different mechanisms.
- For old #PF-based notification, changes current asyncpf.c to add CPUID check at the beginning. It checks (KVM_FEATURE_ASYNC_PF && !KVM_FEATURE_ASYNC_PF_INT), otherwise it gets skipped.
- For new interrupt-based notification, add a new test, asyncpf-int.c, to check (KVM_FEATURE_ASYNC_PF && KVM_FEATURE_ASYNC_PF_INT) and implement interrupt-based 'page-ready' handler.
Dan Wu (3):
x86: Add a common header asyncpf.h
x86: Add async page fault int test
x86/asyncpf: Add CPUID feature bits check to ensure feature is
available
ci/cirrus-ci-fedora.yml | 1 + lib/x86/processor.h | 6 ++ x86/Makefile.common | 3 +- x86/asyncpf.c | 31 ++++++---- x86/asyncpf.h | 23 ++++++++ x86/asyncpf_int.c | 127 ++++++++++++++++++++++++++++++++++++++++ x86/unittests.cfg | 6 +- 7 files changed, 185 insertions(+), 12 deletions(-) create mode 100644 x86/asyncpf.h create mode 100644 x86/asyncpf_int.c
Signature · 1 line
2.39.3
PATCH 1/3 x86: Add a common header asyncpf.h
Grab the ASYNC PF related bit definitions from asyncpf.c and put them into asyncpf.h. Add some new definitions and they are going to be used by asyncpf-int test which will be added later.
--- x86/asyncpf.c | 9 +-------- x86/asyncpf.h | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+), 8 deletions(-) create mode 100644 x86/asyncpf.h
x86/asyncpf.c +1 -8
@@ -27,14 +27,7 @@#include "libcflat.h" #include "vmalloc.h" #include <stdint.h> - -#define KVM_PV_REASON_PAGE_NOT_PRESENT 1 -#define KVM_PV_REASON_PAGE_READY 2 - -#define MSR_KVM_ASYNC_PF_EN 0x4b564d02 - -#define KVM_ASYNC_PF_ENABLED (1 << 0) -#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) +#include "asyncpf.h" volatile uint32_t apf_reason __attribute__((aligned(64)));char *buf;
x86/asyncpf.h new file +23 -0
@@ -0,0 +1,23 @@ +#define KVM_PV_REASON_PAGE_NOT_PRESENT 1 +#define KVM_PV_REASON_PAGE_READY 2 + +#define MSR_KVM_ASYNC_PF_EN 0x4b564d02 +#define MSR_KVM_ASYNC_PF_INT 0x4b564d06 +#define MSR_KVM_ASYNC_PF_ACK 0x4b564d07 + +#define KVM_ASYNC_PF_ENABLED (1 << 0) +#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) +#define KVM_ASYNC_PF_DELIVERY_AS_INT (1 << 3) + +#define HYPERVISOR_CALLBACK_VECTOR 0xf3 + +struct kvm_vcpu_pv_apf_data { +/* Used for 'page not present' events delivered via #PF */ +uint32_t flags; + +/* Used for 'page ready' events delivered via interrupt notification */ +uint32_t token; + +uint8_t pad[56 ]; +uint32_t enabled; +} __attribute__((aligned (64 )));
Signature · 1 line
2.39.3
PATCH 2/3 x86: Add async page fault int test
In reply to Dan WuKVM switched to use interrupt for 'page ready' APF event since Linux v5.10 and the legacy mechanism using #PF was deprecated. Add a new test for interrupt based 'page ready' APF event delivery.
---
The test is based on asyncpf.c and simplifies implementation.
--- ci/cirrus-ci-fedora.yml | 1 + lib/x86/processor.h | 6 ++ x86/Makefile.common | 3 +- x86/asyncpf_int.c | 127 ++++++++++++++++++++++++++++++++++++++++ x86/unittests.cfg | 4 ++ 5 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 x86/asyncpf_int.c
ci/cirrus-ci-fedora.yml +1 -0
@@ -22,6 +22,7 @@ fedora_task: - ./run_tests.sh access asyncpf + asyncpf_int debug emulator ept
lib/x86/processor.h +6 -0
@@ -263,6 +263,12 @@ static inline bool is_intel(void)#define X86_FEATURE_ARCH_CAPABILITIES (CPUID(0x7, 0, EDX, 29))#define X86_FEATURE_PKS (CPUID(0x7, 0, ECX, 31)) +/* +* KVM defined leafs +*/ +#define KVM_FEATURE_ASYNC_PF (CPUID(0x40000001, 0, EAX, 4)) +#define KVM_FEATURE_ASYNC_PF_INT (CPUID(0x40000001, 0, EAX, 14)) +/* * Extended Leafs, a.k.a. AMD defined */
x86/Makefile.common +2 -1
@@ -90,7 +90,8 @@ tests-common = $(TEST_DIR)/vmexit.$(exe) $(TEST_DIR)/tsc.$(exe) \ $(TEST_DIR)/emulator.$(exe) \ $(TEST_DIR)/eventinj.$(exe) \ $(TEST_DIR)/smap.$(exe) \ - $(TEST_DIR)/umip.$(exe) + $(TEST_DIR)/umip.$(exe) \ + $(TEST_DIR)/asyncpf_int.$(exe) # The following test cases are disabled when building EFI tests because they # use absolute addresses in their inline assembly code, which cannot compile
x86/asyncpf_int.c new file +127 -0
127 lines not on this page · raw message
x86/unittests.cfg +4 -0
@@ -174,6 +174,10 @@ extra_params = -cpu max file = asyncpf.flat extra_params = -m 2048 +[asyncpf_int] +file = asyncpf_int.flat +extra_params = -cpu host -m 2048 + [emulator] file = emulator.flat
Signature · 1 line
2.39.3
PATCH 3/3 x86/asyncpf: Add CPUID feature bits check to ensure feature is available
In reply to Dan WuAsyncpf test only works for #PF-based 'page-ready' notification. This case becomes invalid for KVM that starts to enumerate interrupt-based 'page-ready' notification.
Add CPUID feature check at the beginning, and skip the test when KVM_FEATURE_ASYNC_PF is not available or it enumerates KVM_FEATURE_ASYNC_PF_INT.
To run this test, add the QEMU option "-cpu host" to check CPUID, since KVM_FEATURE_ASYNC_PF_INT can't be detected without "-cpu host".
Update the usage of how to setup cgroup for different cgroup versions.
---
Unlike patch #2, this patch retains the original test without reducing the memory access round from 2 to 1. If anyone thinks it worthes a go, please call out.
--- x86/asyncpf.c | 22 ++++++++++++++++++++-- x86/unittests.cfg | 2 +- 2 files changed, 21 insertions(+), 3 deletions(-)
x86/asyncpf.c +20 -2
@@ -1,18 +1,26 @@/* * Async PF test. For the test to actually do anything it needs to be started -* in memory cgroup with 512M of memory and with more then 1G memory provided +* in memory cgroup with 512M of memory and with more than 1G memory provided * to the guest. * +* To identify the cgroup version on Linux: +* stat -fc %T /sys/fs/cgroup/ +* +* If the output is tmpfs, your system is using cgroup v1: * To create cgroup do as root: * mkdir /dev/cgroup * mount -t cgroup none -omemory /dev/cgroup * chmod a+rxw /dev/cgroup/ -* * From a shell you will start qemu from: * mkdir /dev/cgroup/1 * echo $$ > /dev/cgroup/1/tasks * echo 512M > /dev/cgroup/1/memory.limit_in_bytes * +* If the output is cgroup2fs, your system is using cgroup v2: +* mkdir /sys/fs/cgroup/cg1 +* echo $$ > /sys/fs/cgroup/cg1/cgroup.procs +* echo 512M > /sys/fs/cgroup/cg1/memory.max +* */ #include "x86/msr.h" #include "x86/processor.h"
@@ -79,6 +87,16 @@ static void pf_isr(struct ex_regs *r)int main (int ac,char **av) { +if (!this_cpu_has (KVM_FEATURE_ASYNC_PF )) { +report_skip ("KVM_FEATURE_ASYNC_PF is not supported\n" ); +return report_summary (); + } + +if (this_cpu_has (KVM_FEATURE_ASYNC_PF_INT )) { +report_skip ("interrupt-based page-ready event is enumerated, use asyncpf-int instead.\n" ); +return report_summary (); + } +int loop =2 ;setup_vm ();
x86/unittests.cfg +1 -1
@@ -172,7 +172,7 @@ extra_params = -cpu max [asyncpf] file = asyncpf.flat -extra_params = -m 2048 +extra_params = -cpu host -m 2048 [asyncpf_int] file = asyncpf_int.flat
Signature · 1 line
2.39.3
Sean Christopherson
reviewerQuoted 27 lines · Dan Wu
When running asyncpf test, it gets skipped without a clear reason:
./asyncpf
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 107f000
cr4 = 20
install handler
enable async pf
alloc memory
start loop
end loop
start loop
end loop
SUMMARY: 0 tests
SKIP asyncpf (0 tests)The reason is that KVM changed to use interrupt-based 'page-ready' notification and abandoned #PF-based 'page-ready' notification mechanism. Interrupt-based 'page-ready' notification requires
KVM_ASYNC_PF_DELIVERY_AS_INTto be set as well inMSR_KVM_ASYNC_PF_ENto enable asyncpf.This series tries to fix the problem by separating two testcases for different mechanisms.
Quoted 7 lines
- For old #PF-based notification, changes current
asyncpf.cto add CPUID check at the beginning. It checks (KVM_FEATURE_ASYNC_PF&& !KVM_FEATURE_ASYNC_PF_INT), otherwise it gets skipped.- For new interrupt-based notification, add a new test,
asyncpf-int.c, to check (KVM_FEATURE_ASYNC_PF&&KVM_FEATURE_ASYNC_PF_INT) and implement interrupt-based 'page-ready' handler.
Using #PF to deliver page-ready is completely dead, no? Unless I'm mistaken, let's just drop the existing support and replace it with the interrupted-based mechanism. I see no reason to continue maintaining the old crud. If someone wants to verify an old, broken KVM, then they can use the old version of KUT.
Quoted 32 lines · Sean Christopherson
Quoted 31 lines · Dan Wu
When running asyncpf test, it gets skipped without a clear reason:
./asyncpf
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 107f000
cr4 = 20
install handler
enable async pf
alloc memory
start loop
end loop
start loop
end loop
SUMMARY: 0 tests
SKIP asyncpf (0 tests)The reason is that KVM changed to use interrupt-based 'page-ready' notification and abandoned #PF-based 'page-ready' notification mechanism. Interrupt-based 'page-ready' notification requires
KVM_ASYNC_PF_DELIVERY_AS_INTto be set as well inMSR_KVM_ASYNC_PF_ENto enable asyncpf.This series tries to fix the problem by separating two testcases for different mechanisms.
- For old #PF-based notification, changes current
asyncpf.cto add CPUID check at the beginning. It checks (KVM_FEATURE_ASYNC_PF&& !KVM_FEATURE_ASYNC_PF_INT), otherwise it gets skipped.
Quoted 7 lines
Quoted 3 lines
- For new interrupt-based notification, add a new test,
asyncpf-int.c, to check (KVM_FEATURE_ASYNC_PF&&KVM_FEATURE_ASYNC_PF_INT) and implement interrupt-based 'page-ready' handler.Using #PF to deliver page-ready is completely dead, no? Unless I'm mistaken, let's just drop the existing support and replace it with the interrupted-based mechanism. I see no reason to continue maintaining the old crud. If someone wants to verify an old, broken KVM, then they can use the old version of KUT.
Yes, since Linux v5.10 the feature asyncpf is deprecated.
So, just drop asyncpf.c and add asyncpf_int.c is enough, right?
Quoted 1 line
Sean Christopherson
reviewerQuoted 37 lines · On Wed, Dec 13, 2023, Dan1 Wu wrote:
Quoted 35 lines · Sean Christopherson
Quoted 34 lines · Dan Wu
When running asyncpf test, it gets skipped without a clear reason:
./asyncpf
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 107f000
cr4 = 20
install handler
enable async pf
alloc memory
start loop
end loop
start loop
end loop
SUMMARY: 0 tests
SKIP asyncpf (0 tests)The reason is that KVM changed to use interrupt-based 'page-ready' notification and abandoned #PF-based 'page-ready' notification mechanism. Interrupt-based 'page-ready' notification requires
KVM_ASYNC_PF_DELIVERY_AS_INTto be set as well inMSR_KVM_ASYNC_PF_ENto enable asyncpf.This series tries to fix the problem by separating two testcases for different mechanisms.
- For old #PF-based notification, changes current
asyncpf.cto add CPUID check at the beginning. It checks (KVM_FEATURE_ASYNC_PF&& !KVM_FEATURE_ASYNC_PF_INT), otherwise it gets skipped.- For new interrupt-based notification, add a new test,
asyncpf-int.c, to check (KVM_FEATURE_ASYNC_PF&&KVM_FEATURE_ASYNC_PF_INT) and implement interrupt-based 'page-ready' handler.
Quoted 8 lines
Quoted 4 lines
Using #PF to deliver page-ready is completely dead, no? Unless I'm mistaken, let's just drop the existing support and replace it with the interrupted-based mechanism. I see no reason to continue maintaining the old crud. If someone wants to verify an old, broken KVM, then they can use the old version of KUT.
Yes, since Linux v5.10 the feature asyncpf is deprecated.
So, just drop
asyncpf.cand addasyncpf_int.cis enough, right?
I would rather not add asyncpf_int.c, and instead keep asyncpf.c and modify it to use ASYNC_PF_INT. It _might_ be a bit more churn, but modifying the existing code instead of dropping in a new file will better preserve the history, and may also allow for finer grained patches (not sure on that one).
Quoted 41 lines · Sean Christopherson
Quoted 40 lines · On Wed, Dec 13, 2023, Dan1 Wu wrote:
Quoted 39 lines · Sean Christopherson
Quoted 27 lines · Dan Wu
When running asyncpf test, it gets skipped without a clear reason:
./asyncpf
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 107f000
cr4 = 20
install handler
enable async pf
alloc memory
start loop
end loop
start loop
end loop
SUMMARY: 0 tests
SKIP asyncpf (0 tests)The reason is that KVM changed to use interrupt-based 'page-ready' notification and abandoned #PF-based 'page-ready' notification mechanism. Interrupt-based 'page-ready' notification requires
KVM_ASYNC_PF_DELIVERY_AS_INTto be set as well inMSR_KVM_ASYNC_PF_ENto enable asyncpf.This series tries to fix the problem by separating two testcases for different mechanisms.
Quoted 7 lines
- For old #PF-based notification, changes current
asyncpf.cto add CPUID check at the beginning. It checks (KVM_FEATURE_ASYNC_PF&& !KVM_FEATURE_ASYNC_PF_INT), otherwise it gets skipped.- For new interrupt-based notification, add a new test,
asyncpf-int.c, to check(KVM_FEATURE_ASYNC_PF && KVM_FEATURE_ASYNC_PF_INT) and implement interrupt-based 'page-ready' handler.Using #PF to deliver page-ready is completely dead, no? Unless I'm mistaken, let's just drop the existing support and replace it with the interrupted-based mechanism. I see no reason to continue maintaining the old crud. If someone wants to verify an old, broken KVM, then they can use the old version of KUT.
Quoted 7 lines
Quoted 3 lines
Yes, since Linux v5.10 the feature asyncpf is deprecated.
So, just drop
asyncpf.cand addasyncpf_int.cis enough, right?I would rather not add
asyncpf_int.c, and instead keepasyncpf.cand modify it to useASYNC_PF_INT. It _might_ be a bit more churn, but modifying the existing code instead of dropping in a new file will better preserve the history, and may also allow for finer grained patches (not sure on that one).
ok, I will modify it in the next version. Thanks for your review.