KVM: SVM: Set PFERR_GUEST_{PAGE,FINAL}_MASK for nested NPF and add selftest
cover KVM: SVM: Set PFERR_GUEST_{PAGE,FINAL}_MASK for nested NPF and add selftest
This series fixes the setting of PFERR_GUEST_PAGE_MASK and PFERR_GUEST_FINAL_MASK when injecting a nested page fault to L1, and adds a selftest to verify the behavior.
When a nested page fault occurs, L1 needs to know whether the fault happened during the page table walk (on a PT page) or on the final data page translation. This information is conveyed through PFERR_GUEST_PAGE_MASK (bit 33) and PFERR_GUEST_FINAL_MASK (bit 32) in exit_info_1.
Currently, these bits are not set when an NPF is injected during instruction emulation where the original exit reason was not NPF (e.g., during string I/O emulation). This series fixes that and adds test coverage.
Patch 1 adds the PFERR_GUEST_{PAGE,FINAL}_MASK bits to the fault error code in paging_tmpl.h when the GPA->HPA translation fails during a guest page table walk or final page translation.
Patch 2 adds TDP unmap helper functions to the selftest library, enabling tests to selectively unmap pages from the NPT to trigger nested page faults.
Patch 3 adds a selftest that exercises the nested NPF injection path by having L2 execute an OUTS instruction with the source address unmapped from L1's NPT. The test verifies that the correct PFERR_GUEST_* bit is set and that exit_info_2 contains the faulting GPA.
Kevin Cheng (3):
KVM: SVM: Fix nested NPF injection to set
PFERR_GUEST_{PAGE,FINAL}_MASK
KVM: selftests: Add TDP unmap helpers
KVM: selftests: Add nested NPF injection test for SVM
arch/x86/kvm/kvm_emulate.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 22 ++- arch/x86/kvm/svm/nested.c | 11 +- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/include/x86/processor.h | 6 + .../testing/selftests/kvm/lib/x86/processor.c | 53 ++++++ .../selftests/kvm/x86/svm_nested_npf_test.c | 154 ++++++++++++++++++ 7 files changed, 230 insertions(+), 19 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_npf_test.c
base-commit: 38f626812b20ad22ab6dc9cfe6d811850f2d8244 --
2.52.0.457.g6b5491de43-goog
PATCH 1/3 KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation), nested_svm_inject_npf_exit() injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always sets PFERR_GUEST_FINAL_MASK even when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.
Set PFERR_GUEST_PAGE_MASK in the error code when a nested page fault occurs during a guest page table walk, and PFERR_GUEST_FINAL_MASK when the fault occurs on the final GPA-to-HPA translation.
Widen error_code in struct x86_exception from u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).
Update nested_svm_inject_npf_exit() to use fault->error_code directly instead of hardcoding the upper bits. Also add a WARN_ON_ONCE if neither PFERR_GUEST_FINAL_MASK nor PFERR_GUEST_PAGE_MASK is set, as this would indicate a bug in the page fault handling code.
--- arch/x86/kvm/kvm_emulate.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 11 +++++------ 3 files changed, 16 insertions(+), 19 deletions(-)
arch/x86/kvm/kvm_emulate.h +1 -1
@@ -22,7 +22,7 @@ enum x86_intercept_stage;struct x86_exception {u8 vector;bool error_code_valid; -u16 error_code; +u64 error_code;bool nested_page_fault;u64 address;/* cr2 or nested page fault gpa */ u8 async_page_fault;
arch/x86/kvm/mmu/paging_tmpl.h +10 -12
@@ -379,18 +379,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit -* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; +#endif return 0 ; + } slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (real_gpa));if (!kvm_is_visible_memslot (slot))
@@ -446,8 +440,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,#endif real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), access, &walker->fault); -if (real_gpa ==INVALID_GPA ) +if (real_gpa ==INVALID_GPA ) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; +#endif return 0 ; + } walker->gfn = real_gpa >>PAGE_SHIFT ;
arch/x86/kvm/svm/nested.c +5 -6
@@ -40,18 +40,17 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code; +WARN_ON_ONCE (!(vmcb->control.exit_info_1 & + (PFERR_GUEST_FINAL_MASK |PFERR_GUEST_PAGE_MASK ))); +nested_svm_vmexit (svm); }
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 8 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.
Quoted 6 lines
Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).
Please do this in a separate patch. Intel CPUs straight up don't support 32-bit error codes, let alone 64-bit error codes, so this seemingly innocuous change needs to be accompanied by a lengthy changelog that effectively audits all usage to "prove" this change is ok.
Quoted 38 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>--- arch/x86/kvm/kvm_emulate.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 11 +++++------ 3 files changed, 16 insertions(+), 19 deletions(-)arch/x86/kvm/kvm_emulate.h +1
-1@@ -22,7 +22,7 @@ enum x86_intercept_stage;struct x86_exception {u8 vector;bool error_code_valid; -u16 error_code; +u64 error_code;bool nested_page_fault;u64 address;/* cr2 or nested page fault gpa */ u8 async_page_fault;arch/x86/kvm/mmu/paging_tmpl.h +0
-5@@ -379,18 +379,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit -* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause,
arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 385
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +4
-6-* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; +#endif
Why exclude EPT? EPT doesn't use the error code _verbatim_, but EPT shares the concept/reporting of intermediate vs. final walks.
Quoted 6 lines
arch/x86/kvm/mmu/paging_tmpl.h +1
-0return 0 ; + } slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (real_gpa));if (!kvm_is_visible_memslot (slot))
arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 446
Quoted 8 lines
arch/x86/kvm/mmu/paging_tmpl.h +4
-1@@ -446,8 +440,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,#endif real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), access, &walker->fault); -if (real_gpa ==INVALID_GPA ) +if (real_gpa ==INVALID_GPA ) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; +#endif
Same thing here.
Quoted 16 lines
arch/x86/kvm/mmu/paging_tmpl.h +1
-0return 0 ; + } walker->gfn = real_gpa >>PAGE_SHIFT ;arch/x86/kvm/svm/nested.c +0
-3@@ -40,18 +40,17 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1.
arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 49
Quoted 10 lines
arch/x86/kvm/svm/nested.c +2
-3-*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;
So... what happens when exit_info_1 already has PFERR_GUEST_PAGE_MASK, and then @fault sets PFERR_GUEST_FINAL_MASK? Presumably that can't/shouldn't happen, but nothing in the changelog explains why such a scenario is impossible, and nothing in the code hardens KVM against such goofs.
Quoted 9 lines
arch/x86/kvm/svm/nested.c +3
-0+WARN_ON_ONCE (!(vmcb->control.exit_info_1 & + (PFERR_GUEST_FINAL_MASK |PFERR_GUEST_PAGE_MASK ))); +nested_svm_vmexit (svm); }Signature · 1 line
2.52.0.457.g6b5491de43-goog
Re: [PATCH 1/3] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.
Quoted 7 lines
Quoted 2 lines
Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Please do this in a separate patch. Intel CPUs straight up don't support 32-bit error codes, let alone 64-bit error codes, so this seemingly innocuous change needs to be accompanied by a lengthy changelog that effectively audits all usage to "prove" this change is ok.
Semi-jokingly, we can add error_code_hi to track the high bits and side-step the problem for Intel (dejavu?).
Quoted 7 lines
Quoted 7 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>
[..]
Quoted 18 lines
Quoted 18 lines
arch/x86/kvm/svm/nested.c +1
-6@@ -40,18 +40,17 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }
Quoted 8 lines
arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 49
Quoted 3 lines
arch/x86/kvm/svm/nested.c +1
-0+ vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;So... what happens when
exit_info_1already hasPFERR_GUEST_PAGE_MASK, and then @fault setsPFERR_GUEST_FINAL_MASK? Presumably that can't/shouldn't happen, but nothing in the changelog explains why such a scenario is impossible, and nothing in the code hardens KVM against such goofs.
I guess we can update the WARN below to check for that as well, and fallback to the current behavior (set PFERR_GUEST_FINAL_MASK):
fault_stage = vmcb->control.exit_info_1 & (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK); if (WARN_ON_ONCE(fault_stage != PFERR_GUEST_FINAL_MASK && fault_stage != PFERR_GUEST_PAGE_MASK)) vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK;
Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/svm/nested.c +3
-0+WARN_ON_ONCE (!(vmcb->control.exit_info_1 & + (PFERR_GUEST_FINAL_MASK |PFERR_GUEST_PAGE_MASK ))); +nested_svm_vmexit (svm); }Signature · 1 line
2.52.0.457.g6b5491de43-goog
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 17 lines · Yosry Ahmed
Quoted 16 lines · Sean Christopherson
Quoted 14 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).
Quoted 7 lines
Quoted 4 lines
Please do this in a separate patch. Intel CPUs straight up don't support 32-bit error codes, let alone 64-bit error codes, so this seemingly innocuous change needs to be accompanied by a lengthy changelog that effectively audits all usage to "prove" this change is ok.
Semi-jokingly, we can add
error_code_hito track the high bits and side-step the problem for Intel (dejavu?).
Technically, it would require three fields: u16 error_code, u16 error_code_hi, and u32 error_code_ultra_hi. :-D
Isolating the (ultra) hi flags is very tempting, but I worry that it would lead to long term pain, e.g. because inevitably we'll forget to grab the hi flags at some point. I'd rather audit the current code and ensure that KVM truncates the error code as needed.
VMX is probably a-ok, e.g. see commit eba9799b5a6e ("KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS"). I'd be more worred SVM, where it's legal to shove a 32-bit value into the error code, i.e. where KVM might not have existing explicit truncation.
Quoted 34 lines
Quoted 6 lines
Quoted 6 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>[..]
Quoted 18 lines
Quoted 18 lines
arch/x86/kvm/svm/nested.c +1
-6@@ -40,18 +40,17 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }Quoted 8 lines
arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 49
Quoted 3 lines
arch/x86/kvm/svm/nested.c +1
-0+ vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;So... what happens when
exit_info_1already hasPFERR_GUEST_PAGE_MASK, and then @fault setsPFERR_GUEST_FINAL_MASK? Presumably that can't/shouldn't happen, but nothing in the changelog explains why such a scenario is impossible, and nothing in the code hardens KVM against such goofs.
Quoted 8 lines
I guess we can update the WARN below to check for that as well, and fallback to the current behavior (set
PFERR_GUEST_FINAL_MASK):fault_stage = vmcb->control.exit_info_1 & (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK); if (WARN_ON_ONCE(fault_stage != PFERR_GUEST_FINAL_MASK && fault_stage != PFERR_GUEST_PAGE_MASK)) vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK;
Except that doesn't do the right thing if both bits are set. And we can use hweight64(), which is a single POPCNT on modern CPUs.
Might be easiest to add something like PFERR_GUEST_FAULT_STAGE_MASK, then do:
/*
* All nested page faults should be annotated as occuring on the final
* translation *OR* the page walk. Arbitrarily choose "final" if KVM
* is buggy and enumerated both or none.
*/
if (WARN_ON_ONCE(hweight64(vmcb->control.exit_info_1 &
PFERR_GUEST_FAULT_STAGE_MASK) != 1)) {
vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK;
vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK;
}Re: [PATCH 1/3] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 35 lines · Sean Christopherson
Quoted 17 lines · Yosry Ahmed
Quoted 16 lines · Sean Christopherson
Quoted 14 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Quoted 7 lines
Quoted 4 lines
Please do this in a separate patch. Intel CPUs straight up don't support 32-bit error codes, let alone 64-bit error codes, so this seemingly innocuous change needs to be accompanied by a lengthy changelog that effectively audits all usage to "prove" this change is ok.
Semi-jokingly, we can add
error_code_hito track the high bits and side-step the problem for Intel (dejavu?).Technically, it would require three fields: u16
error_code, u16error_code_hi, and u32error_code_ultra_hi. :-DIsolating the (ultra) hi flags is very tempting, but I worry that it would lead to long term pain, e.g. because inevitably we'll forget to grab the hi flags at some point. I'd rather audit the current code and ensure that KVM truncates the error code as needed.
Quoted 4 lines
VMX is probably a-ok, e.g. see commit
eba9799b5a6e("KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS"). I'd be more worred SVM, where it's legal to shove a 32-bit value into the error code, i.e. where KVM might not have existing explicit truncation.
As I understand it, intel CPUs don't allow for setting bits 31:16 of the error code, but AMD CPUs allow bits 31:16 to be set. The 86_exception error_code field is u16 currently so it is always truncated to u16 by default. In that case, after widening the error code to 64 bits, do I have to ensure that any usage of the error that isn't for NPF, has to truncate it to 16 bits? Or do I just need to verify that all SVM usages of the error_code for exceptions truncate the 64 bits down to 32 bits and all VMX usages truncate to 16 bits?
Just wanted to clarify because I think the wording of that statement is confusing me into thinking that maybe there is something wrong with 32 bit error codes for SVM?
If the only usage of the widened field is NPF, wouldn't it be better to go with an additional field like Yosry suggested (I see that VMX has the added exit_qualification field in the struct)?
Quoted 59 lines
Quoted 35 lines
Quoted 7 lines
Quoted 7 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>[..]
Quoted 18 lines
Quoted 18 lines
arch/x86/kvm/svm/nested.c +1
-6@@ -40,18 +40,17 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }Quoted 8 lines
arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 49
Quoted 3 lines
arch/x86/kvm/svm/nested.c +1
-0+ vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;So... what happens when
exit_info_1already hasPFERR_GUEST_PAGE_MASK, and then @fault setsPFERR_GUEST_FINAL_MASK? Presumably that can't/shouldn't happen, but nothing in the changelog explains why such a scenario is impossible, and nothing in the code hardens KVM against such goofs.Quoted 8 lines
I guess we can update the WARN below to check for that as well, and fallback to the current behavior (set
PFERR_GUEST_FINAL_MASK):fault_stage = vmcb->control.exit_info_1 & (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK); if (WARN_ON_ONCE(fault_stage != PFERR_GUEST_FINAL_MASK && fault_stage != PFERR_GUEST_PAGE_MASK)) vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK;Except that doesn't do the right thing if both bits are set. And we can use
hweight64(), which is a single POPCNT on modern CPUs.Might be easiest to add something like
PFERR_GUEST_FAULT_STAGE_MASK, then do:/* * All nested page faults should be annotated as occuring on the final * translation *OR* the page walk. Arbitrarily choose "final" if KVM * is buggy and enumerated both or none. */ if (WARN_ON_ONCE(hweight64(vmcb->control.exit_info_1 & PFERR_GUEST_FAULT_STAGE_MASK) != 1)) { vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK; vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK; }
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 36 lines · Kevin Cheng
Quoted 35 lines · Sean Christopherson
Quoted 17 lines · Yosry Ahmed
Quoted 16 lines · Sean Christopherson
Quoted 14 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Quoted 7 lines
Quoted 4 lines
Please do this in a separate patch. Intel CPUs straight up don't support 32-bit error codes, let alone 64-bit error codes, so this seemingly innocuous change needs to be accompanied by a lengthy changelog that effectively audits all usage to "prove" this change is ok.
Semi-jokingly, we can add
error_code_hito track the high bits and side-step the problem for Intel (dejavu?).Technically, it would require three fields: u16
error_code, u16error_code_hi, and u32error_code_ultra_hi. :-DIsolating the (ultra) hi flags is very tempting, but I worry that it would lead to long term pain, e.g. because inevitably we'll forget to grab the hi flags at some point. I'd rather audit the current code and ensure that KVM truncates the error code as needed.
Quoted 7 lines
Quoted 4 lines
VMX is probably a-ok, e.g. see commit
eba9799b5a6e("KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS"). I'd be more worred SVM, where it's legal to shove a 32-bit value into the error code, i.e. where KVM might not have existing explicit truncation.As I understand it, intel CPUs don't allow for setting bits 31:16 of the error code, but AMD CPUs allow bits 31:16 to be set.
Yep.
Quoted 8 lines
The 86_exception
error_codefield is u16 currently so it is always truncated to u16 by default. In that case, after widening the error code to 64 bits, do I have to ensure that any usage of the error that isn't for NPF, has to truncate it to 16 bits?Or do I just need to verify that all SVM usages of the
error_codefor exceptions truncate the 64 bits down to 32 bits and all VMX usages truncate to 16 bits?
Hmm, good question.
I was going to say "the second one", but that's actually meaningless because
(a) "struct kvm_queued_exception" stores the error code as a u32, which it should, and (b) event_inj_err is also a u32, i.e. it's impossible to shove a 64-bit error code into hardware on SVM.
And thinking through this more, if there is _existing_ code that tries to set bits > 15 in the error_code, then UBSAN would likely have detected the issue, e.g. due to trying to OR in a "bad" value.
Aha! A serendipitous quirk in this patch is that it does NOT change the local error_code in FNAME(walk_addr_generic) from a u16 to a u64.
We should double down on that with a comment. That'd give me enough confidence that we aren't likely to break legacy shadow paging now or in the future. E.g. in a patch to change x86_exception.error_code to a u64, also do:
arch/x86/kvm/mmu/paging_tmpl.h +6 -0
@@ -317,6 +317,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,const int write_fault = access &PFERR_WRITE_MASK ;const int user_fault = access &PFERR_USER_MASK ;const int fetch_fault = access &PFERR_FETCH_MASK ; +/* +* Note! Track the error_code that's common to legacy shadow paging +* and NPT shadow paging as a u16 to guard against unintentionally +* setting any of bits 63:16. Architecturally, the #PF error code is +* 32 bits, and Intel CPUs don't support settings bits 31:16. +*/ u16 errcode =0 ;gpa_t real_gpa;gfn_t gfn;
Quoted 3 lines
Just wanted to clarify because I think the wording of that statement is confusing me into thinking that maybe there is something wrong with 32 bit error codes for SVM?
It's more that I am confident that either KVM already truncates the error code on VMX, or that we'll notice *really* quickly, because failure to truncate an error code will generate a VM-Fail.
On SVM, we could royally screw up an error code and it's entirely possible we wouldn't notice until some random guest breaks in some weird way.
Quoted 3 lines
If the only usage of the widened field is NPF, wouldn't it be better to go with an additional field like Yosry suggested (I see that VMX has the added
exit_qualificationfield in the struct)?
No? paging_tmpl.h is used to shadow all flavors of nested NPT as well as all flavors of legacy paging. And more importantly, unlike EPT, nested NPT isn't otherwise special cased. As shown by this patch, it _is_ possible to identify nested NPT in select flows, and we could certainly do so more generically by checking if the MMU is nested, but I'd prefer not to special case any particular type of shadow paging without a strong reason to do so.
And more importantly, if we have two (or three) error code fields, then we need to remember to pull data from all error code fields. I.e. in an effort to avoid introducing bugs, we would actually make it easier to introduce _other_ bugs.
PATCH 2/3 KVM: selftests: Add TDP unmap helpers
In reply to Kevin ChengAdd __virt_pg_unmap(), __tdp_unmap(), and tdp_unmap() as counterparts to the existing __virt_pg_map(), __tdp_map(), and tdp_map() functions. These helpers allow tests to selectively unmap pages from the TDP/NPT, enabling testing of NPT faults for unmapped pages.
--- .../selftests/kvm/include/x86/processor.h | 6 +++ .../testing/selftests/kvm/lib/x86/processor.c | 53 +++++++++++++++++++ 2 files changed, 59 insertions(+)
tools/testing/selftests/kvm/include/x86/processor.h +6 -0
@@ -1487,6 +1487,12 @@ void tdp_map(struct kvm_vm *vm, uint64_t nested_paddr, uint64_t paddr, uint64_tvoid tdp_identity_map_default_memslots (struct kvm_vm *vm);void tdp_identity_map_1g (struct kvm_vm *vm,uint64_t addr,uint64_t size); +void __virt_pg_unmap (struct kvm_vm *vm,struct kvm_mmu *mmu,uint64_t vaddr, +int level); +void __tdp_unmap (struct kvm_vm *vm,uint64_t nested_paddr,uint64_t size, +int level); +void tdp_unmap (struct kvm_vm *vm,uint64_t nested_paddr,uint64_t size); +/* * Basic CPU control in CR0 */
tools/testing/selftests/kvm/lib/x86/processor.c +53 -0
65 lines not on this page · raw message
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Sean Christopherson
maintainerRe: [PATCH 2/3] KVM: selftests: Add TDP unmap helpers
Quoted 4 lines · Kevin Cheng
Add
__virt_pg_unmap(),__tdp_unmap(), andtdp_unmap()as counterparts to the existing__virt_pg_map(),__tdp_map(), andtdp_map()functions. These helpers allow tests to selectively unmap pages from the TDP/NPT, enabling testing of NPT faults for unmapped pages.
This is both overkill and insufficient, just do:
*tdp_get_pte(vm, <addr>) &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
Then when a test wants to validate more than just !PRESENT we don't need to add another API.
PATCH 3/3 KVM: selftests: Add nested NPF injection test for SVM
In reply to Kevin ChengAdd a test that exercises nested NPF injection when the original VM exit was not an NPF. This tests the code path in nested_svm_inject_npf_exit() where exit_code != SVM_EXIT_NPF.
L2 executes an OUTS instruction with the source address mapped in L2's page tables but not in L1's NPT. KVM emulates the string I/O, and when it tries to read the source operand, the GPA->HPA translation fails. KVM then injects an NPF to L1 even though the original exit was IOIO.
The test verifies that:
- The exit code is converted to SVM_EXIT_NPF
- exit_info_1 has the appropriate PFERR_GUEST_* bit set
- exit_info_2 contains the correct faulting GPA
Two test cases are implemented:
- Test 1: Unmap the final data page from NPT (PFERR_GUEST_FINAL_MASK)
- Test 2: Unmap a PT page from NPT (PFERR_GUEST_PAGE_MASK)
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/svm_nested_npf_test.c | 154 ++++++++++++++++++ 2 files changed, 155 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_npf_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -112,6 +112,7 @@ TEST_GEN_PROGS_x86 += x86/svm_vmcall_test TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test +TEST_GEN_PROGS_x86 += x86/svm_nested_npf_test TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync TEST_GEN_PROGS_x86 += x86/sync_regs_test TEST_GEN_PROGS_x86 += x86/ucna_injection_test
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +154 -0
154 lines not on this page · raw message
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Sean Christopherson
maintainerRe: [PATCH 3/3] KVM: selftests: Add nested NPF injection test for SVM
Quoted 11 lines · Kevin Cheng
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/svm_nested_npf_test.c | 154 ++++++++++++++++++ 2 files changed, 155 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_npf_test.ctools/testing/selftests/kvm/Makefile.kvm +0
-0
tools/testing/selftests/kvm/Makefile.kvm TEST_GEN_PROGS_x86 += x86/svm_vmcall_test · line 115
Quoted 4 lines
tools/testing/selftests/kvm/Makefile.kvm +1
-0@@ -112,6 +112,7 @@ TEST_GEN_PROGS_x86 += x86/svm_vmcall_test TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test +TEST_GEN_PROGS_x86 += x86/svm_nested_npf_test
a, b, c, d, e, f, g, h, i, j, k, l, m, N, o, p, q, r, S, t, u, v, w, x, y, z
Quoted 13 lines
tools/testing/selftests/kvm/Makefile.kvm +0
-0TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync TEST_GEN_PROGS_x86 += x86/sync_regs_test TEST_GEN_PROGS_x86 += x86/ucna_injection_testtools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +4
-0@@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* svm_nested_npf_test +*
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c line 14
Quoted 10 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +10
-0+* Test nested NPF injection when the original VM exit was not an NPF. +* This exercises nested_svm_inject_npf_exit() with exit_code != SVM_EXIT_NPF. +* +* L2 executes OUTS with the source address mapped in L2's page tables but +* not in L1's NPT. KVM emulates the string I/O instruction, and when it +* tries to read the source operand, the GPA->HPA translation fails. KVM +* then injects an NPF to L1 even though the original exit was IOIO. +* +* Test 1: Final data page GPA not in NPT (PFERR_GUEST_FINAL_MASK) +* Test 2: Page table page GPA not in NPT (PFERR_GUEST_PAGE_MASK)
Please don't add file-level comments (the Copyright is fine), because things like the name of the test/file inevitably become stale, and they're useless, and the description of _what_ the test is doing is almost always more helpful if it's the comment is closer to the code it's documenting.
Quoted 7 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +7
-0+* +* Copyright (C) 2025, Google, Inc. +*/ + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h"
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c line 31
Quoted 10 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +10
-0+#include "svm_util.h" + +#define L2_GUEST_STACK_SIZE 64 + +enum test_type { +TEST_FINAL_PAGE_UNMAPPED ,/* Final data page GPA not in NPT */ +TEST_PT_PAGE_UNMAPPED ,/* Page table page GPA not in NPT */ +}; + +static void *l2_test_page;
Why store it as a "void *"? Just track a vm_addr_t and avoid a bunch of casts.
Quoted 4 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +4
-0+ +#define TEST_IO_PORT 0x80 +#define TEST1_VADDR 0x8000000ULL +#define TEST2_VADDR 0x10000000ULL
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c line 45
Quoted 10 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +10
-0+ +/* +* L2 executes OUTS with source at l2_test_page, triggering a nested NPF. +* The address is mapped in L2's page tables, but either the data page or +* a PT page is unmapped from L1's NPT, causing the fault. +*/ +static void l2_guest_code (void *unused) +{ + asmvolatile ("outsb" ::"S" (l2_test_page),"d" (TEST_IO_PORT ) :"memory" ); +GUEST_ASSERT (0 );
GUEST_FAIL
Quoted 2 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +2
-0+} +
...
Quoted 27 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +27
-0+static void run_test (enum test_type type) +{ +vm_paddr_t expected_fault_gpa; +uint64_t exit_info_1_mask; +vm_vaddr_t svm_gva; + +struct kvm_vcpu *vcpu; +struct kvm_vm *vm; +struct ucall uc; + + vm =vm_create_with_one_vcpu (&vcpu, l1_guest_code); +vm_enable_npt (vm); +vcpu_alloc_svm (vm, &svm_gva); + +if (type ==TEST_FINAL_PAGE_UNMAPPED ) { +/* +* Test 1: Unmap the final data page from NPT. The page table +* walk succeeds, but the final GPA->HPA translation fails. +*/ + l2_test_page = + (void *)vm_vaddr_alloc (vm, vm->page_size,TEST1_VADDR ); + expected_fault_gpa =addr_gva2gpa (vm, (vm_vaddr_t )l2_test_page); + exit_info_1_mask =PFERR_GUEST_FINAL_MASK ; + }else { +/* +* Test 2: Unmap a PT page from NPT. The hardware page table +* walk fails when translating the PT page's GPA through NPT.
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c line 126
Quoted 10 lines
tools/testing/selftests/kvm/x86/svm_nested_npf_test.c new file +10
-0+*/ + l2_test_page = + (void *)vm_vaddr_alloc (vm, vm->page_size,TEST2_VADDR ); + expected_fault_gpa = +get_pt_gpa_for_vaddr (vm, (vm_vaddr_t )l2_test_page); + exit_info_1_mask =PFERR_GUEST_PAGE_MASK ; + } + +tdp_identity_map_default_memslots (vm); +tdp_unmap (vm, expected_fault_gpa, vm->page_size);
Hrm. This should really be a vendor agnostic test. There exactly results are vendor specific, but thye core concept and pretty much all of the configuration is nearly identical.
It'd also be nice to support more than just !PRESENT, e.g. to verify protection violations and other things that set PFERR/EXIT_QUAL bits.