KVM: X86: Correctly populate nested page fault
v2 cover KVM: X86: Correctly populate nested page fault
This series fixes how KVM populates error information whne injecting nested page faults (NPF on SVM, EPT violations on VMX) to L1 during instruction emulation, and adds a selftest to exercise these paths.
When a nested page fault occurs, L1 needs to know whether the fault happened during the page table walk (on a PT page) or on the final data page translation. Two issues exist today:
1. SVM: The page table walker does not set PFERR_GUEST_{PAGE,FINAL}_MASK bits in the error code, and nested_svm_inject_page_fault() hardcodes the PFERR_GUEST_FINAL_MASK error code.
2. VMX: nested_ept_inject_page_fault() OR's bit 7-8 from the original exit qualification, which has no relation to the synthesized EPT violation regardless of whether it was originally an EPT violation or not.
Patch 1 widens x86_exception.error_code from u16 to u64 so it can carry the PFERR_GUEST_* bits (bits 32-33).
Patch 2 sets PFERR_GUEST_PAGE_MASK and PFERR_GUEST_FINAL_MASK in the walker at the kvm_translate_gpa() failure sites, and updates nested_svm_inject_npf_exit() to use the walker-provided error code.
Patch 3 removes the OR with the hardware exit qualification in nested_ept_inject_page_fault(), and populates EPT_VIOLATION_GVA_IS_VALID and EPT_VIOLATION_GVA_TRANSLATED in the walker alongside the NPF bits.
Patch 4 adds a selftest covering both SVM and VMX with three scenarios:
- Final data page unmapped (final translation fault)
- Page table page unmapped (page walk fault)
- Final data page write-protected (protection violation)
- Page table page write-protected (protection violation)
v1 -> v2:
- Split out the widening of the x86_exception error code into a separate patch as per Sean.
- Added a WARN if both PFERR_GUEST_* bits are set and force the exit_info_1 to PFERR_GUEST_FINAL_MASK if this occurs.
- Removed the selftest TDP helpers as per Sean
- Added a patch to populate the EPT violation bits for VMX nested page faults as per Sean.
- Expanded the added selftest to support VMX and also added a test case for write protected pages using the INS instruction.
base-commit: 183bb0ce8c77b0fd1fb25874112bc8751a461e49
Kevin Cheng (4):
KVM: x86: Widen x86_exception's error_code to 64 bits
KVM: SVM: Fix nested NPF injection to set
PFERR_GUEST_{PAGE,FINAL}_MASK
KVM: VMX: Don't consult original exit qualification for nested EPT
violation injection
KVM: selftests: Add nested page fault injection test
arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/kvm_emulate.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 44 ++- arch/x86/kvm/svm/nested.c | 19 +- arch/x86/kvm/vmx/nested.c | 3 - tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/nested_npf_test.c | 374 ++++++++++++++++++ 7 files changed, 422 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86/nested_npf_test.c
--
2.53.0.414.gf7e9f6c205-goog
PATCH v2 1/4 KVM: x86: Widen x86_exception's error_code to 64 bits
Widen the error_code field in struct x86_exception from u16 to u64 to accommodate AMD's NPF error code, which defines information bits above bit 31, e.g. PFERR_GUEST_FINAL_MASK (bit 32), and PFERR_GUEST_PAGE_MASK (bit 33).
Retain the u16 type for the local errcode variable in walk_addr_generic as the walker synthesizes conventional #PF error codes that are architecturally limited to bits 15:0.
--- arch/x86/kvm/kvm_emulate.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/kvm/kvm_emulate.h +1 -1
@@ -22,7 +22,7 @@ enum x86_intercept_stage;struct x86_exception {u8 vector;bool error_code_valid; -u16 error_code; +u64 error_code;bool nested_page_fault;u64 address;/* cr2 or nested page fault gpa */ u8 async_page_fault;
arch/x86/kvm/mmu/paging_tmpl.h +6 -0
@@ -317,6 +317,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,const int write_fault = access &PFERR_WRITE_MASK ;const int user_fault = access &PFERR_USER_MASK ;const int fetch_fault = access &PFERR_FETCH_MASK ; +/* +* Note! Track the error_code that's common to legacy shadow paging +* and NPT shadow paging as a u16 to guard against unintentionally +* setting any of bits 63:16. Architecturally, the #PF error code is +* 32 bits, and Intel CPUs don't support settings bits 31:16. +*/ u16 errcode =0 ;gpa_t real_gpa;gfn_t gfn;
Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
PATCH v2 2/4 KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
In reply to Kevin ChengWhen KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation), nested_svm_inject_npf_exit() injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always sets PFERR_GUEST_FINAL_MASK even when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.
Set PFERR_GUEST_PAGE_MASK in the error code when a nested page fault occurs during a guest page table walk, and PFERR_GUEST_FINAL_MASK when the fault occurs on the final GPA-to-HPA translation.
Widen error_code in struct x86_exception from u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).
Update nested_svm_inject_npf_exit() to use fault->error_code directly instead of hardcoding the upper bits. Also add a WARN_ON_ONCE if neither PFERR_GUEST_FINAL_MASK nor PFERR_GUEST_PAGE_MASK is set, as this would indicate a bug in the page fault handling code.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 19 +++++++++++++------ 3 files changed, 25 insertions(+), 18 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -280,6 +280,8 @@ enum x86_intercept_stage;#define PFERR_GUEST_RMP_MASK BIT_ULL(31)#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)#define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)#define PFERR_GUEST_ENC_MASK BIT_ULL(34)#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)
arch/x86/kvm/mmu/paging_tmpl.h +10 -12
@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit -* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; +#endif return 0 ; + } slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (real_gpa));if (!kvm_is_visible_memslot (slot))
@@ -452,8 +446,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,#endif real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), access, &walker->fault); -if (real_gpa ==INVALID_GPA ) +if (real_gpa ==INVALID_GPA ) { +#if PTTYPE !=PTTYPE_EPT + walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; +#endif return 0 ; + } walker->gfn = real_gpa >>PAGE_SHIFT ;
arch/x86/kvm/svm/nested.c +13 -6
@@ -40,18 +40,25 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code; +/* +* All nested page faults should be annotated as occurring on the +* final translation *or* the page walk. Arbitrarily choose "final" +* if KVM is buggy and enumerated both or neither. +*/ +if (WARN_ON_ONCE (hweight64 (vmcb->control.exit_info_1 & +PFERR_GUEST_FAULT_STAGE_MASK ) !=1 )) { + vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK ; + vmcb->control.exit_info_1 |=PFERR_GUEST_FINAL_MASK ; + } +nested_svm_vmexit (svm); }
Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
Sean Christopherson
maintainerRe: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 8 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.
Quoted 6 lines
Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).
Stale comment as this was moved to a separate patch.
Quoted 36 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 19 +++++++++++++------ 3 files changed, 25 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -280,6 +280,8 @@ enum x86_intercept_stage;#define PFERR_GUEST_RMP_MASK BIT_ULL(31)#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)#define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)#define PFERR_GUEST_ENC_MASK BIT_ULL(34)#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)arch/x86/kvm/mmu/paging_tmpl.h +0
-3@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit
arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 389
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +2
-8-* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT
I would rather swap the order of patches two and three, so that we end up with a "positive" if-statement. I.e. add EPT first so that we get (spoiler alert):
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;
#endif
Quoted 22 lines
arch/x86/kvm/svm/nested.c +2
-6@@ -40,18 +40,25 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;
arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 60
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+/* +* All nested page faults should be annotated as occurring on the +* final translation *or* the page walk. Arbitrarily choose "final" +* if KVM is buggy and enumerated both or neither. +*/ +if (WARN_ON_ONCE (hweight64 (vmcb->control.exit_info_1 & +PFERR_GUEST_FAULT_STAGE_MASK ) !=1 )) { + vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK ; + vmcb->control.exit_info_1 |=PFERR_GUEST_FINAL_MASK ; + }
This is all kinds of messy. KVM _appears_ to still rely on the hardware-reported address + error_code
if (vmcb->control.exit_code != SVM_EXIT_NPF) {
vmcb->control.exit_info_1 = fault->error_code;
vmcb->control.exit_info_2 = fault->address;
}But then drops bits 31:0 in favor of the fault error code. Then even more bizarrely, bitwise-ORs bits 63:32 and WARNs if multiple bits in PFERR_GUEST_FAULT_STAGE_MASK are set. In practice, the bitwise-OR of 63:32 is _only_ going to affect PFERR_GUEST_FAULT_STAGE_MASK, because the other defined bits are all specific to SNP, and KVM doesn't support nested virtualization for SEV+.
So I don't understand why this isn't simply:
vmcb->control.exit_code = SVM_EXIT_NPF;
vmcb->control.exit_info_1 = fault->error_code;
/*
* All nested page faults should be annotated as occurring on the
* final translation *or* the page walk. Arbitrarily choose "final"
* if KVM is buggy and enumerated both or neither.
*/
if (WARN_ON_ONCE(hweight64(vmcb->control.exit_info_1 &
PFERR_GUEST_FAULT_STAGE_MASK) != 1)) {
vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK;
vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK;
}Which would more or less align with the proposed nEPT handling.
Quoted 7 lines
arch/x86/kvm/svm/nested.c +1
-0+nested_svm_vmexit (svm); }Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
Sean Christopherson
maintainerRe: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 25 lines · Sean Christopherson
Quoted 11 lines
arch/x86/kvm/mmu/paging_tmpl.h +0
-3@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 389
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +2
-8-* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT I would rather swap the order of patches two and three, so that we end up with a "positive" if-statement. I.e. add EPT first so that we get (spoiler alert):
Quoted 5 lines
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;#endif
Scratch this comment, I got my walkers confused.
Re: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
In reply to Sean ChristophersonQuoted 123 lines · Sean Christopherson
Quoted 8 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Quoted 6 lines
Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Stale comment as this was moved to a separate patch.
Quoted 36 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 19 +++++++++++++------ 3 files changed, 25 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -280,6 +280,8 @@ enum x86_intercept_stage;#define PFERR_GUEST_RMP_MASK BIT_ULL(31)#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)#define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)#define PFERR_GUEST_ENC_MASK BIT_ULL(34)#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)arch/x86/kvm/mmu/paging_tmpl.h +0
-3@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 389
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +2
-8-* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT I would rather swap the order of patches two and three, so that we end up with a "positive" if-statement. I.e. add EPT first so that we get (spoiler alert):
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;#endif
Quoted 22 lines
arch/x86/kvm/svm/nested.c +2
-6@@ -40,18 +40,25 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 60
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+/* +* All nested page faults should be annotated as occurring on the +* final translation *or* the page walk. Arbitrarily choose "final" +* if KVM is buggy and enumerated both or neither. +*/ +if (WARN_ON_ONCE (hweight64 (vmcb->control.exit_info_1 & +PFERR_GUEST_FAULT_STAGE_MASK ) !=1 )) { + vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK ; + vmcb->control.exit_info_1 |=PFERR_GUEST_FINAL_MASK ; + }This is all kinds of messy. KVM _appears_ to still rely on the hardware-reported address +
error_codeif (vmcb->control.exit_code != SVM_EXIT_NPF) { vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }But then drops bits 31:0 in favor of the fault error code. Then even more bizarrely, bitwise-ORs bits 63:32 and WARNs if multiple bits in
PFERR_GUEST_FAULT_STAGE_MASKare set. In practice, the bitwise-OR of 63:32 is _only_ going to affectPFERR_GUEST_FAULT_STAGE_MASK, because the other defined bits are all specific to SNP, and KVM doesn't support nested virtualization for SEV+.
Quoted 5 lines
So I don't understand why this isn't simply:
vmcb->control.exit_code = SVM_EXIT_NPF; vmcb->control.exit_info_1 = fault->error_code;
Hmmm yes I do think it can be replaced by this but we would also need to grab the address from the walker. So
vmcb->control.exit_code = SVM_EXIT_NPF;
vmcb->control.exit_info_1 = fault->error_code;
vmcb->control.exit_info_2 = fault->address;For example, in the selftest that I wrote we should be populating the exit_info_2 with the faulting address from the walker, not the original hardware reported address which is related to IO.
Quoted 20 lines
/* * All nested page faults should be annotated as occurring on the * final translation *or* the page walk. Arbitrarily choose "final" * if KVM is buggy and enumerated both or neither. */ if (WARN_ON_ONCE(hweight64(vmcb->control.exit_info_1 & PFERR_GUEST_FAULT_STAGE_MASK) != 1)) { vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK; vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK; }Which would more or less align with the proposed nEPT handling.
Quoted 7 lines
arch/x86/kvm/svm/nested.c +1
-1+nested_svm_vmexit (svm); } --2.53.0.414.gf7e9f6c205-goog
Sean Christopherson
maintainerRe: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 25 lines · Kevin Cheng
Quoted 15 lines · Sean Christopherson
This is all kinds of messy. KVM _appears_ to still rely on the hardware-reported address +
error_codeif (vmcb->control.exit_code != SVM_EXIT_NPF) { vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }But then drops bits 31:0 in favor of the fault error code. Then even more bizarrely, bitwise-ORs bits 63:32 and WARNs if multiple bits in
PFERR_GUEST_FAULT_STAGE_MASKare set. In practice, the bitwise-OR of 63:32 is _only_ going to affectPFERR_GUEST_FAULT_STAGE_MASK, because the other defined bits are all specific to SNP, and KVM doesn't support nested virtualization for SEV+.Quoted 5 lines
So I don't understand why this isn't simply:
vmcb->control.exit_code = SVM_EXIT_NPF; vmcb->control.exit_info_1 = fault->error_code;Hmmm yes I do think it can be replaced by this but we would also need to grab the address from the walker. So
Quoted 7 lines
vmcb->control.exit_code = SVM_EXIT_NPF; vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address;For example, in the selftest that I wrote we should be populating the
exit_info_2with the faulting address from the walker, not the original hardware reported address which is related to IO.
Yeah, sorry for the confusion. I wasn't saying _don't_ include the address, I was just pointing out that the error_code handling can be much simpler.
Re: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
In reply to Sean ChristophersonQuoted 123 lines · Sean Christopherson
Quoted 8 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Quoted 6 lines
Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Stale comment as this was moved to a separate patch.
Quoted 36 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 19 +++++++++++++------ 3 files changed, 25 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -280,6 +280,8 @@ enum x86_intercept_stage;#define PFERR_GUEST_RMP_MASK BIT_ULL(31)#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)#define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)#define PFERR_GUEST_ENC_MASK BIT_ULL(34)#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)arch/x86/kvm/mmu/paging_tmpl.h +0
-3@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 389
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +2
-8-* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT I would rather swap the order of patches two and three, so that we end up with a "positive" if-statement. I.e. add EPT first so that we get (spoiler alert):
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;#endif
Quoted 22 lines
arch/x86/kvm/svm/nested.c +2
-6@@ -40,18 +40,25 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 60
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+/* +* All nested page faults should be annotated as occurring on the +* final translation *or* the page walk. Arbitrarily choose "final" +* if KVM is buggy and enumerated both or neither. +*/ +if (WARN_ON_ONCE (hweight64 (vmcb->control.exit_info_1 & +PFERR_GUEST_FAULT_STAGE_MASK ) !=1 )) { + vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK ; + vmcb->control.exit_info_1 |=PFERR_GUEST_FINAL_MASK ; + }This is all kinds of messy. KVM _appears_ to still rely on the hardware-reported address +
error_codeif (vmcb->control.exit_code != SVM_EXIT_NPF) { vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }But then drops bits 31:0 in favor of the fault error code. Then even more bizarrely, bitwise-ORs bits 63:32 and WARNs if multiple bits in
PFERR_GUEST_FAULT_STAGE_MASKare set. In practice, the bitwise-OR of 63:32 is _only_ going to affectPFERR_GUEST_FAULT_STAGE_MASK, because the other defined bits are all specific to SNP, and KVM doesn't support nested virtualization for SEV+.
Quoted 4 lines
So I don't understand why this isn't simply:
vmcb->control.exit_code = SVM_EXIT_NPF; vmcb->control.exit_info_1 = fault->error_code;
The issue with this is that the PFERR_GUEST_FAULT_STAGE_MASK bits are not set in the walker fault error code for hardware reported NPFs (non-emulator faults).
The active mmu after an NPF exit from L2 is the guest_mmu. In walk_addr_generic, we only set the PFERR_GUEST_FAULT_STAGE_MASK bits in the fault error code if kvm_translate_gpa returns INVALID_GPA (i.e. when mmu == &vcpu->arch.nested_mmu and the translation fails). Otherwise, kvm_translate_gpa just returns the gpa and we don't set the PFERR_GUEST_FAULT_STAGE_MASK bits in the fault error code.
Even when we NPF exit from L2, the hardware-reported exit_info[31:0] isn't accurate which is why we drop 31:0 in favor of the fault error code. I observed the hardware error code vs fault error code while running the KUTs (specifically the tests that expect faults) in
https://lore.kernel.org/all/[email protected]/
and they were not as expected. I believe I saw different error codes because the entry existed in L1's NPT, but not yet in L0's NPT? The hardware-reported error code usually did not reflect the L1 NPT page permission restrictions, only the type of access/op that caused the fault. Additionally, the present bit was not set.
If L2 has never accessed a GPA present in the L1 NPT with restricted permissions, we could see differing error codes between hardware and the fault error codes. I think that is why the code before my change dropped bits 31:0 in favor of the walker fault error code. I'm not entirely sure though so I could be wrong.
Since the guest_mmu walker can't know whether the GPA it's translating is a page table page or a final page, I think we are stuck with combining bits 63:32 from the hardware reported error code and 31:0 of the fault error code for the non-emulator NPF injection case.
Quoted 21 lines
/* * All nested page faults should be annotated as occurring on the * final translation *or* the page walk. Arbitrarily choose "final" * if KVM is buggy and enumerated both or neither. */ if (WARN_ON_ONCE(hweight64(vmcb->control.exit_info_1 & PFERR_GUEST_FAULT_STAGE_MASK) != 1)) { vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK; vmcb->control.exit_info_1 |= PFERR_GUEST_FINAL_MASK; }Which would more or less align with the proposed nEPT handling.
Quoted 7 lines
arch/x86/kvm/svm/nested.c +1
-1+nested_svm_vmexit (svm); } --2.53.0.414.gf7e9f6c205-goog
Re: [PATCH V2 2/4] KVM: SVM: Fix nested NPF injection to set PFERR_GUEST_{PAGE,FINAL}_MASK
Quoted 158 lines · Kevin Cheng
Quoted 123 lines · Sean Christopherson
Quoted 8 lines · Kevin Cheng
When KVM emulates an instruction for L2 and encounters a nested page fault (e.g., during string I/O emulation),
nested_svm_inject_npf_exit()injects an NPF to L1. However, the code incorrectly hardcodes (1ULL << 32) for exit_info_1's upper bits when the original exit was not an NPF. This always setsPFERR_GUEST_FINAL_MASKeven when the fault occurred on a page table page, preventing L1 from correctly identifying the cause of the fault.Quoted 6 lines
Set
PFERR_GUEST_PAGE_MASKin the error code when a nested page fault occurs during a guest page table walk, andPFERR_GUEST_FINAL_MASKwhen the fault occurs on the final GPA-to-HPA translation.Widen
error_codein structx86_exceptionfrom u16 to u64 to accommodate the PFERR_GUEST_* bits (bits 32 and 33).Stale comment as this was moved to a separate patch.
Quoted 36 lines
Update
nested_svm_inject_npf_exit()to usefault->error_codedirectly instead of hardcoding the upper bits. Also add aWARN_ON_ONCEif neitherPFERR_GUEST_FINAL_MASKnorPFERR_GUEST_PAGE_MASKis set, as this would indicate a bug in the page fault handling code.Signed-off-by Kevin Cheng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 22 ++++++++++------------ arch/x86/kvm/svm/nested.c | 19 +++++++++++++------ 3 files changed, 25 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -280,6 +280,8 @@ enum x86_intercept_stage;#define PFERR_GUEST_RMP_MASK BIT_ULL(31)#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)#define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK)#define PFERR_GUEST_ENC_MASK BIT_ULL(34)#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)arch/x86/kvm/mmu/paging_tmpl.h +0
-3@@ -385,18 +385,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), nested_access, &walker->fault); -/* -* FIXME: This can happen if emulation (for of an INS/OUTS -* instruction) triggers a nested page fault. The exit arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 389
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +2
-8-* qualification / exit info field will incorrectly have -* "guest page access" as the nested page fault's cause, -* instead of "guest page structure access". To fix this, -* the x86_exception struct should be augmented with enough -* information to fix the exit_qualification or exit_info_1 -* fields. -*/ -if (unlikely (real_gpa ==INVALID_GPA )) +if (unlikely (real_gpa ==INVALID_GPA )) { +#if PTTYPE !=PTTYPE_EPT I would rather swap the order of patches two and three, so that we end up with a "positive" if-statement. I.e. add EPT first so that we get (spoiler alert):
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;#endif
Quoted 22 lines
arch/x86/kvm/svm/nested.c +2
-6@@ -40,18 +40,25 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,struct vmcb *vmcb = svm->vmcb;if (vmcb->control.exit_code !=SVM_EXIT_NPF ) { -/* -* TODO: track the cause of the nested page fault, and -* correctly fill in the high bits of exit_info_1. -*/ - vmcb->control.exit_code =SVM_EXIT_NPF ; - vmcb->control.exit_info_1 = (1ULL <<32 ); + vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; } + vmcb->control.exit_code =SVM_EXIT_NPF ; vmcb->control.exit_info_1 &= ~0xffffffffULL ; vmcb->control.exit_info_1 |= fault->error_code;arch/x86/kvm/svm/nested.c static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, · line 60
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+/* +* All nested page faults should be annotated as occurring on the +* final translation *or* the page walk. Arbitrarily choose "final" +* if KVM is buggy and enumerated both or neither. +*/ +if (WARN_ON_ONCE (hweight64 (vmcb->control.exit_info_1 & +PFERR_GUEST_FAULT_STAGE_MASK ) !=1 )) { + vmcb->control.exit_info_1 &= ~PFERR_GUEST_FAULT_STAGE_MASK ; + vmcb->control.exit_info_1 |=PFERR_GUEST_FINAL_MASK ; + }This is all kinds of messy. KVM _appears_ to still rely on the hardware-reported address +
error_codeif (vmcb->control.exit_code != SVM_EXIT_NPF) { vmcb->control.exit_info_1 = fault->error_code; vmcb->control.exit_info_2 = fault->address; }But then drops bits 31:0 in favor of the fault error code. Then even more bizarrely, bitwise-ORs bits 63:32 and WARNs if multiple bits in
PFERR_GUEST_FAULT_STAGE_MASKare set. In practice, the bitwise-OR of 63:32 is _only_ going to affectPFERR_GUEST_FAULT_STAGE_MASK, because the other defined bits are all specific to SNP, and KVM doesn't support nested virtualization for SEV+.Quoted 4 lines
So I don't understand why this isn't simply:
vmcb->control.exit_code = SVM_EXIT_NPF; vmcb->control.exit_info_1 = fault->error_code;The issue with this is that the
PFERR_GUEST_FAULT_STAGE_MASKbits are not set in the walker fault error code for hardware reported NPFs (non-emulator faults).The active mmu after an NPF exit from L2 is the
guest_mmu. Inwalk_addr_generic, we only set thePFERR_GUEST_FAULT_STAGE_MASKbits in the fault error code ifkvm_translate_gpareturnsINVALID_GPA(i.e. when mmu ==&vcpu->arch.nested_mmuand the translation fails). Otherwise,kvm_translate_gpajust returns the gpa and we don't set thePFERR_GUEST_FAULT_STAGE_MASKbits in the fault error code.Even when we NPF exit from L2, the hardware-reported exit_info[31:0] isn't accurate which is why we drop 31:0 in favor of the fault error code. I observed the hardware error code vs fault error code while running the KUTs (specifically the tests that expect faults) in
https://lore.kernel.org/all/[email protected]/
and they were not as expected. I believe I saw different error codes because the entry existed in L1's NPT, but not yet in L0's NPT? The hardware-reported error code usually did not reflect the L1 NPT page permission restrictions, only the type of access/op that caused the fault. Additionally, the present bit was not set.If L2 has never accessed a GPA present in the L1 NPT with restricted permissions, we could see differing error codes between hardware and the fault error codes. I think that is why the code before my change dropped bits 31:0 in favor of the walker fault error code. I'm not entirely sure though so I could be wrong.
Quoted 5 lines
Since the
guest_mmuwalker can't know whether the GPA it's translating is a page table page or a final page, I think we are stuck with combining bits 63:32 from the hardware reported error code and 31:0 of the fault error code for the non-emulator NPF injection case.
Oops this was already pointed out in
https://lore.kernel.org/all/[email protected]/ lol
PATCH v2 3/4 KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
In reply to Kevin ChengRemove the OR of EPT_VIOLATION_GVA_IS_VALID and EPT_VIOLATION_GVA_TRANSLATED from the hardware exit qualification when injecting a synthesized EPT violation to L1. The hardware exit qualification reflects the original VM exit, which may not be an EPT violation at all, e.g. if KVM is emulating an I/O instruction and the memory operand's translation through L1's EPT fails. In that case, bits 7-8 of the exit qualification have completely different semantics (or are simply zero), and OR'ing them into the injected EPT violation corrupts the GVA_IS_VALID/GVA_TRANSLATED information.
Even when the original exit is an EPT violation, the hardware bits may not match the current fault. For example, if an EPT violation happened while walking L2's page tables, it's possible that the EPT violation injected by KVM into L1 is for the final address translation, if L1 already had the mappings for L2's page tables in its EPTs but KVM did not have shadow EPTs for them.
Populate EPT_VIOLATION_GVA_IS_VALID and EPT_VIOLATION_GVA_TRANSLATED directly in the page table walker at the kvm_translate_gpa() failure sites, mirroring the existing PFERR_GUEST_PAGE_MASK and PFERR_GUEST_FINAL_MASK population for NPT.
--- arch/x86/kvm/mmu/paging_tmpl.h | 16 +++++++++++++++- arch/x86/kvm/vmx/nested.c | 3 --- 2 files changed, 15 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/paging_tmpl.h +15 -1
@@ -386,8 +386,19 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, nested_access, &walker->fault);if (unlikely (real_gpa ==INVALID_GPA )) { +/* +* Unconditionally set the NPF error_code bits and +* EPT exit_qualification bits for nested page +* faults. The walker doesn't know whether L1 uses +* NPT or EPT, and each injection handler consumes +* only the field it cares about (error_code for +* NPF, exit_qualification for EPT violations), so +* setting both is harmless. +*/ #if PTTYPE !=PTTYPE_EPT walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; + walker->fault.exit_qualification |= +EPT_VIOLATION_GVA_IS_VALID ;#endif return 0 ; }
@@ -449,6 +460,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,if (real_gpa ==INVALID_GPA ) {#if PTTYPE !=PTTYPE_EPT walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; + walker->fault.exit_qualification |= +EPT_VIOLATION_GVA_IS_VALID | +EPT_VIOLATION_GVA_TRANSLATED ;#endif return 0 ; }
@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,* [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables -* [7:8] - Derived from [7:8] of real exit_qualification +* [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */
arch/x86/kvm/vmx/nested.c +0 -3
@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED ); vm_exit_reason =EXIT_REASON_EPT_VIOLATION ; }
Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
Sean Christopherson
maintainerRe: [PATCH V2 3/4] KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
Quoted 16 lines · Kevin Cheng
--- arch/x86/kvm/mmu/paging_tmpl.h | 16 +++++++++++++++- arch/x86/kvm/vmx/nested.c | 3 --- 2 files changed, 15 insertions(+), 4 deletions(-)arch/x86/kvm/mmu/paging_tmpl.h +3
-0@@ -386,8 +386,19 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, nested_access, &walker->fault);if (unlikely (real_gpa ==INVALID_GPA )) { +/* +* Unconditionally set the NPF error_code bits and +* EPT exit_qualification bits for nested page
arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(walk_addr_generic)(struct guest_walker *walker, · line 401
Quoted 10 lines
arch/x86/kvm/mmu/paging_tmpl.h +8
-0+* faults. The walker doesn't know whether L1 uses +* NPT or EPT, and each injection handler consumes +* only the field it cares about (error_code for +* NPF, exit_qualification for EPT violations), so +* setting both is harmless. +*/ #if PTTYPE !=PTTYPE_EPT walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; + walker->fault.exit_qualification |= +EPT_VIOLATION_GVA_IS_VALID ;
This looks all kinds of wrong. Shouldn't it be?
#if PTTYPE == PTTYPE_EPT walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; #else walker->fault.error_code |= PFERR_GUEST_PAGE_MASK;
#endif
Ooooh, right, never mind, this is the case where KVM detects a fault on the L2 GPA => L1 GPA translation when walking L2 GVA=>GPA.
This is very counter-intuitive, and unconditionally setting both is rather ugly. To address both, what if we peek at the guest_mmu to determine whether the fault is EPT vs. NPT? E.g.
arch/x86/kvm/mmu/mmu.c +10 -0
@@ -5263,6 +5263,9 @@ static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,return false; } +static bool kvm_nested_fault_is_ept (struct kvm_vcpu *vcpu, +struct x86_exception *exception); +#define PTTYPE_EPT 18/* arbitrary */ #define PTTYPE PTTYPE_EPT#include "paging_tmpl.h"
@@ -5276,6 +5279,13 @@ static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,#include "paging_tmpl.h" #undef PTTYPE +static bool kvm_nested_fault_is_ept (struct kvm_vcpu *vcpu, +struct x86_exception *exception) +{ +WARN_ON_ONCE (!exception->nested_page_fault); +return vcpu->arch.guest_mmu.page_fault == ept_page_fault; +} +static void __reset_rsvds_bits_mask (struct rsvd_bits_validate *rsvd_check,u64 pa_bits_rsvd,int level,bool nx,bool gbpages,bool pse,bool amd)
arch/x86/kvm/mmu/paging_tmpl.h +12 -18
@@ -387,19 +387,14 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,if (unlikely (real_gpa ==INVALID_GPA )) {/* -* Unconditionally set the NPF error_code bits and -* EPT exit_qualification bits for nested page -* faults. The walker doesn't know whether L1 uses -* NPT or EPT, and each injection handler consumes -* only the field it cares about (error_code for -* NPF, exit_qualification for EPT violations), so -* setting both is harmless. +* Set EPT Violation flags even if the fault is an +* EPT Misconfig, fault.exit_qualification is ignored +* for EPT Misconfigs. */ -#if PTTYPE !=PTTYPE_EPT - walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ; - walker->fault.exit_qualification |= -EPT_VIOLATION_GVA_IS_VALID ; -#endif +if (kvm_nested_fault_is_ept (vcpu, &walker->fault)) + walker->fault.exit_qualification |=EPT_VIOLATION_GVA_IS_VALID ; +else + walker->fault.error_code |=PFERR_GUEST_PAGE_MASK ;return 0 ; }
@@ -458,12 +453,11 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), access, &walker->fault);if (real_gpa ==INVALID_GPA ) { -#if PTTYPE !=PTTYPE_EPT - walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; - walker->fault.exit_qualification |= -EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED ; -#endif +if (kvm_nested_fault_is_ept (vcpu, &walker->fault)) + walker->fault.exit_qualification |=EPT_VIOLATION_GVA_IS_VALID | +EPT_VIOLATION_GVA_TRANSLATED ; +else + walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ;return 0 ; }
Quoted 27 lines
arch/x86/kvm/mmu/paging_tmpl.h +4
-1#endif return 0 ; }@@ -449,6 +460,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,if (real_gpa ==INVALID_GPA ) {#if PTTYPE !=PTTYPE_EPT walker->fault.error_code |=PFERR_GUEST_FINAL_MASK ; + walker->fault.exit_qualification |= +EPT_VIOLATION_GVA_IS_VALID | +EPT_VIOLATION_GVA_TRANSLATED ;#endif return 0 ; }@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,* [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables -* [7:8] - Derived from [7:8] of real exit_qualification +* [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */ arch/x86/kvm/vmx/nested.c +0
-0
arch/x86/kvm/vmx/nested.c static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-3@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED );
Hmm, this isn't quite correct. If KVM injects an EPT Violation (or a #NPF) when handling an EPT Violation (or #NPF) from L2, then KVM _should_ follow hardware.
Aha! I think the easiest way to deal with that is to flag nested page faults that were the result of walking L1's TDP when handling an L2 TDP page fault, and then let vendor code extract the fault information out of hardaware.
Alternatively, we could plumb in VMX's EPT_VIOLATION_GVA_IS_VALID as a synthetic error code, but I think that be harder to follow overall, especially for VMX.
@@ -799,8 +793,10 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
* The page is not mapped by the guest. Let the guest handle it.
*/
if (!r) {
- if (!fault->prefetch)
+ if (!fault->prefetch) {
+ walker.fault.hardware_nested_page_fault = walker.fault.nested_page_fault;
kvm_inject_emulated_page_fault(vcpu, &walker.fault);+ }
return RET_PF_RETRY;
}arch/x86/kvm/vmx/nested.c +6 -1
@@ -443,8 +443,13 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, vm_exit_reason =EXIT_REASON_EPT_MISCONFIG ; exit_qualification =0 ; }else { - exit_qualification = fault->exit_qualification; vm_exit_reason =EXIT_REASON_EPT_VIOLATION ; + + exit_qualification = fault->exit_qualification; +if (fault->hardware_nested_page_fault) + exit_qualification |=vmx_get_exit_qual (vcpu) & + (EPT_VIOLATION_GVA_IS_VALID | +EPT_VIOLATION_GVA_TRANSLATED ); }/*
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-0vm_exit_reason =EXIT_REASON_EPT_VIOLATION ; }Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
Re: [PATCH V2 3/4] KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
Quoted 24 lines
Quoted 14 lines
(quoted diff) +1
-1@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, * [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables - * [7:8] - Derived from [7:8] of real exit_qualification + * [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-3@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED );Hmm, this isn't quite correct. If KVM injects an EPT Violation (or a #NPF) when handling an EPT Violation (or #NPF) from L2, then KVM _should_ follow hardware.
Quoted 3 lines
Aha! I think the easiest way to deal with that is to flag nested page faults that were the result of walking L1's TDP when handling an L2 TDP page fault, and then let vendor code extract the fault information out of hardaware.
Is it not possible that KVM gets an EPT Violation (or a #NPF) on an L2 memory access while the CPU is walking L2's page tables, then KVM walks L1's TDP and finds mappings for the L2 page tables but not the final translation? Or will KVM always just fixup the immediate EPT Violation (or #NPF) by inserting a shadow mapping of L2's page tables and retry the instruction immediately?
Sean Christopherson
maintainerRe: [PATCH V2 3/4] KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
Quoted 28 lines · Yosry Ahmed
Quoted 24 lines
Quoted 14 lines
(quoted diff) +1
-1@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, * [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables - * [7:8] - Derived from [7:8] of real exit_qualification + * [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-3@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED );Hmm, this isn't quite correct. If KVM injects an EPT Violation (or a #NPF) when handling an EPT Violation (or #NPF) from L2, then KVM _should_ follow hardware.
Quoted 3 lines
Aha! I think the easiest way to deal with that is to flag nested page faults that were the result of walking L1's TDP when handling an L2 TDP page fault, and then let vendor code extract the fault information out of hardaware.
Quoted 6 lines
Is it not possible that KVM gets an EPT Violation (or a #NPF) on an L2 memory access while the CPU is walking L2's page tables, then KVM walks L1's TDP and finds mappings for the L2 page tables but not the final translation? Or will KVM always just fixup the immediate EPT Violation (or #NPF) by inserting a shadow mapping of L2's page tables and retry the instruction immediately?
The latter, assuming by "shadow mapping of L2's page tables" out meant "installing a shadow mapping of the faulting L2 GPA according to L1's TDP page tables".
I.e. when servicing an L2 TDP fault, KVM is only resolving the fault for the reported L2 GPA, _not_ the originating L2 GVA (if there is one).
Re: [PATCH V2 3/4] KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
Quoted 37 lines · Sean Christopherson
Quoted 28 lines · Yosry Ahmed
Quoted 24 lines
Quoted 14 lines
(quoted diff) +1
-1@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, * [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables - * [7:8] - Derived from [7:8] of real exit_qualification + * [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-3@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED );Hmm, this isn't quite correct. If KVM injects an EPT Violation (or a #NPF) when handling an EPT Violation (or #NPF) from L2, then KVM _should_ follow hardware.
Quoted 3 lines
Aha! I think the easiest way to deal with that is to flag nested page faults that were the result of walking L1's TDP when handling an L2 TDP page fault, and then let vendor code extract the fault information out of hardaware.
Quoted 6 lines
Is it not possible that KVM gets an EPT Violation (or a #NPF) on an L2 memory access while the CPU is walking L2's page tables, then KVM walks L1's TDP and finds mappings for the L2 page tables but not the final translation? Or will KVM always just fixup the immediate EPT Violation (or #NPF) by inserting a shadow mapping of L2's page tables and retry the instruction immediately?
Quoted 5 lines
The latter, assuming by "shadow mapping of L2's page tables" out meant "installing a shadow mapping of the faulting L2 GPA according to L1's TDP page tables".
I.e. when servicing an L2 TDP fault, KVM is only resolving the fault for the reported L2 GPA, _not_ the originating L2 GVA (if there is one).
I see. Does this need special handling when forwarding #NPFs to L1 as well? Or will fault->error_code have the HW fault bits in nested_svm_inject_npf_exit() in this case?
Sean Christopherson
maintainerRe: [PATCH V2 3/4] KVM: VMX: Don't consult original exit qualification for nested EPT violation injection
Quoted 41 lines · Yosry Ahmed
Quoted 40 lines · Sean Christopherson
Quoted 28 lines · Yosry Ahmed
Quoted 24 lines
Quoted 14 lines
(quoted diff) +1
-1@@ -496,7 +510,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, * [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables - * [7:8] - Derived from [7:8] of real exit_qualification + * [7:8] - Set at the kvm_translate_gpa() call sites above * * The other bits are set to 0. */arch/x86/kvm/vmx/nested.c +0
-0arch/x86/kvm/vmx/nested.c static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +0
-3@@ -444,9 +444,6 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, exit_qualification =0 ; }else { exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED );Hmm, this isn't quite correct. If KVM injects an EPT Violation (or a #NPF) when handling an EPT Violation (or #NPF) from L2, then KVM _should_ follow hardware.
Quoted 3 lines
Aha! I think the easiest way to deal with that is to flag nested page faults that were the result of walking L1's TDP when handling an L2 TDP page fault, and then let vendor code extract the fault information out of hardaware.
Quoted 6 lines
Is it not possible that KVM gets an EPT Violation (or a #NPF) on an L2 memory access while the CPU is walking L2's page tables, then KVM walks L1's TDP and finds mappings for the L2 page tables but not the final translation? Or will KVM always just fixup the immediate EPT Violation (or #NPF) by inserting a shadow mapping of L2's page tables and retry the instruction immediately?
The latter, assuming by "shadow mapping of L2's page tables" out meant "installing a shadow mapping of the faulting L2 GPA according to L1's TDP page tables".
Quoted 6 lines
Quoted 2 lines
I.e. when servicing an L2 TDP fault, KVM is only resolving the fault for the reported L2 GPA, _not_ the originating L2 GVA (if there is one).
I see. Does this need special handling when forwarding #NPFs to L1 as well? Or will
fault->error_codehave the HW fault bits innested_svm_inject_npf_exit()in this case?
Yes, sorry for not making that it explicit. nested_svm_inject_npf_exit() would also need to consult hardware_nested_page_fault.
PATCH v2 4/4 KVM: selftests: Add nested page fault injection test
In reply to Kevin ChengAdd a test that exercises nested page fault injection during L2 execution. L2 executes I/O string instructions (OUTSB/INSB) that access memory restricted in L1's nested page tables (NPT/EPT), triggering a nested page fault that L0 must inject to L1.
The test supports both AMD SVM (NPF) and Intel VMX (EPT violation) and verifies that:
- The exit reason is an NPF/EPT violation
- The access type and permission bits are correct
- The faulting GPA is correct
Three test cases are implemented:
- Unmap the final data page (final translation fault, OUTSB read)
- Unmap a PT page (page walk fault, OUTSB read)
- Write-protect the final data page (protection violation, INSB write)
- Write-protect a PT page (protection violation on A/D update, OUTSB read)
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/nested_npf_test.c | 374 ++++++++++++++++++ 2 files changed, 375 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/nested_npf_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -93,6 +93,7 @@ TEST_GEN_PROGS_x86 += x86/nested_dirty_log_test TEST_GEN_PROGS_x86 += x86/nested_emulation_test TEST_GEN_PROGS_x86 += x86/nested_exceptions_test TEST_GEN_PROGS_x86 += x86/nested_invalid_cr3_test +TEST_GEN_PROGS_x86 += x86/nested_npf_test TEST_GEN_PROGS_x86 += x86/nested_set_state_test TEST_GEN_PROGS_x86 += x86/nested_tsc_adjust_test TEST_GEN_PROGS_x86 += x86/nested_tsc_scaling_test
tools/testing/selftests/kvm/x86/nested_npf_test.c new file +374 -0
374 lines not on this page · raw message
Signature · 1 line
2.53.0.414.gf7e9f6c205-goog
Sean Christopherson
maintainerRe: [PATCH V2 4/4] KVM: selftests: Add nested page fault injection test
Quoted 11 lines · Kevin Cheng
Add a test that exercises nested page fault injection during L2 execution. L2 executes I/O string instructions (OUTSB/INSB) that access memory restricted in L1's nested page tables (NPT/EPT), triggering a nested page fault that L0 must inject to L1.
The test supports both AMD SVM (NPF) and Intel VMX (EPT violation) and verifies that:
- The exit reason is an NPF/EPT violation
- The access type and permission bits are correct
- The faulting GPA is correct
Quoted 6 lines
Three test cases are implemented:
- Unmap the final data page (final translation fault, OUTSB read)
- Unmap a PT page (page walk fault, OUTSB read)
- Write-protect the final data page (protection violation, INSB write)
- Write-protect a PT page (protection violation on A/D update, OUTSB read)
Either in this test or in KUT, we need coverage for validating faults that are reported by hardware, i.e. for faults that _don't_ go through the emulator.
E.g. there's this "todo" of sorts in KUT:
case VMX_EPT_VIOLATION: /* * Exit-qualifications are masked not to account for advanced * VM-exit information. Once KVM supports this feature, this * masking should be removed. */ exit_qual &= ~EPT_VLT_GUEST_MASK;
Or maybe both? I generally prefer selftests for maintenance purposes, and you've already written this test...
Re: [PATCH V2 4/4] KVM: selftests: Add nested page fault injection test
Quoted 34 lines · Sean Christopherson
Quoted 11 lines · Kevin Cheng
Add a test that exercises nested page fault injection during L2 execution. L2 executes I/O string instructions (OUTSB/INSB) that access memory restricted in L1's nested page tables (NPT/EPT), triggering a nested page fault that L0 must inject to L1.
The test supports both AMD SVM (NPF) and Intel VMX (EPT violation) and verifies that:
- The exit reason is an NPF/EPT violation
- The access type and permission bits are correct
- The faulting GPA is correctQuoted 6 lines
Three test cases are implemented:
- Unmap the final data page (final translation fault, OUTSB read)
- Unmap a PT page (page walk fault, OUTSB read)
- Write-protect the final data page (protection violation, INSB write)
- Write-protect a PT page (protection violation on A/D update, OUTSB read)Either in this test or in KUT, we need coverage for validating faults that are reported by hardware, i.e. for faults that _don't_ go through the emulator.
E.g. there's this "todo" of sorts in KUT:
case
VMX_EPT_VIOLATION:
/*
* Exit-qualifications are masked not to account for advanced
* VM-exit information. Once KVM supports this feature, this
* masking should be removed.
*/
exit_qual&= ~EPT_VLT_GUEST_MASK;
Quoted 2 lines
Or maybe both? I generally prefer selftests for maintenance purposes, and you've already written this test...
Does https://lore.kernel.org/all/[email protected]/
count :) These exercise nested_svm_inject_npf_exit()
Actually running these tests, I realized that the expected_fault is incorrect in those KVM unit tests. Because PFERR_GUEST_PAGE_MASK was previously hardcorded, the KUTs had that set in the expected_fault. However, with the NPF injection fixes, I will need to change the KUTs to expect PFERR_GUEST_FINAL_MASK to be set for the different test cases which is actually the correct expectation.