Backport nested EPT violation GVA bit fix
Subsystems (2)
cover Backport nested EPT violation GVA bit fix
Backport of commit 96b067b59ad9 ("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits") to 6.12.y, with its two dependencies.
Patch 3 is the fix: KVM pulls the GVA_IS_VALID and GVA_TRANSLATED bits from vmcs02.EXIT_QUALIFICATION even when the fault injected into L1 was synthesized while emulating an L2 GVA access, i.e. when that field describes an unrelated VM-Exit.
Patches 1 and 2 are prerequisites, tagged Stable-dep-of; neither changes behavior on its own. Patch 1 lets the EPT walker tell a paging-structure access from a final access, patch 2 plumbs in whether a fault came from hardware and renames kvm_inject_emulated_page_fault()'s exported symbol to __kvm_inject_emulated_page_fault().
Context adjustments are noted per patch. Based on v6.12.113, build tested on x86_64 with CONFIG_KVM_INTEL=m and CONFIG_KVM_AMD=m. Not runtime tested under nested virtualization.
Kevin Cheng (1):
KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED
bits
Paolo Bonzini (1):
KVM: x86/mmu: pass PFERR_GUEST_PAGE/FINAL_MASK to kvm_translate_gpa
Sean Christopherson (1):
KVM: x86: Tell ->inject_page_fault() whether or a fault came from
hardware
arch/x86/include/asm/kvm_host.h | 18 ++++++++++++++---- arch/x86/kvm/hyperv.c | 3 ++- arch/x86/kvm/mmu/mmu.c | 3 ++- arch/x86/kvm/mmu/paging_tmpl.h | 26 ++++++++++++++++++++++---- arch/x86/kvm/regs.c | 3 ++- arch/x86/kvm/svm/nested.c | 3 ++- arch/x86/kvm/vmx/nested.c | 26 +++++++++++++++++++++----- arch/x86/kvm/x86.c | 16 +++++++++------- 8 files changed, 74 insertions(+), 24 deletions(-)
Signature · 1 line
2.50.1
PATCH 1/3 KVM: x86/mmu: pass PFERR_GUEST_PAGE/FINAL_MASK to kvm_translate_gpa
From: Paolo Bonzini <[email protected]>
[ Upstream commit bd1c40114b02cf63ba33da8e7df71ed0fc2a3019 ]
The XS/XU bit for EPT are only applied to final accesses, and use the U bit from the page walk itself. While strictly speaking not necessary (any value of PFERR_USER_MASK would be the same for page table accesses, because they're reads and writes only), it is clearer and less hackish to only apply MBEC to PFERR_GUEST_FINAL_MASK. Allow kvm-intel.ko to distinguish the two cases.
[doebel: load_pdptrs() lives in arch/x86/kvm/regs.c in this tree; keep
this tree's mmu_is_nested() in the Hyper-V hunk]
Stable-dep-of: 96b067b59ad9 ("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits")
--- arch/x86/kvm/hyperv.c | 3 ++- arch/x86/kvm/mmu/mmu.c | 3 ++- arch/x86/kvm/mmu/paging_tmpl.h | 7 +++++-- arch/x86/kvm/regs.c | 3 ++- 4 files changed, 11 insertions(+), 5 deletions(-)
arch/x86/kvm/hyperv.c +2 -1
@@ -2055,7 +2055,8 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)* read with kvm_read_guest(). */ if (!hc->fast &&mmu_is_nested (vcpu)) { - hc->ingpa =translate_nested_gpa (vcpu, hc->ingpa,0 ,NULL ); + hc->ingpa =translate_nested_gpa (vcpu, hc->ingpa, +PFERR_GUEST_FINAL_MASK ,NULL );if (unlikely (hc->ingpa ==INVALID_GPA ))return HV_STATUS_INVALID_HYPERCALL_INPUT ; }
arch/x86/kvm/mmu/mmu.c +2 -1
@@ -4136,7 +4136,8 @@ static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, {if (exception) exception->error_code =0 ; -return kvm_translate_gpa (vcpu, mmu, vaddr, access, exception); +return kvm_translate_gpa (vcpu, mmu, vaddr, access |PFERR_GUEST_FINAL_MASK , + exception); }static bool mmio_info_in_cache (struct kvm_vcpu *vcpu,u64 addr,bool direct)
arch/x86/kvm/mmu/paging_tmpl.h +5 -2
@@ -377,7 +377,8 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, walker->pte_gpa[walker->level -1 ] = pte_gpa; real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (table_gfn), - nested_access, &walker->fault); + nested_access |PFERR_GUEST_PAGE_MASK , + &walker->fault);/* * FIXME: This can happen if emulation (for of an INS/OUTS
@@ -445,7 +446,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, gfn +=pse36_gfn_delta (pte);#endif - real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), access, &walker->fault); + real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (gfn), + access |PFERR_GUEST_FINAL_MASK , + &walker->fault);if (real_gpa ==INVALID_GPA )return 0 ;
arch/x86/kvm/regs.c +2 -1
@@ -166,7 +166,8 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)* to an L1 GPA. */ real_gpa =kvm_translate_gpa (vcpu, mmu,gfn_to_gpa (pdpt_gfn), -PFERR_USER_MASK |PFERR_WRITE_MASK ,NULL ); +PFERR_USER_MASK |PFERR_WRITE_MASK | +PFERR_GUEST_PAGE_MASK ,NULL );if (real_gpa ==INVALID_GPA )return 0 ;
Signature · 1 line
2.50.1
PATCH 2/3 KVM: x86: Tell ->inject_page_fault() whether or a fault came from hardware
In reply to Bjoern DoebelFrom: Sean Christopherson <[email protected]>
[ Upstream commit fe0b872d750023eda270cbc01c53ead52b1049d8 ]
When injecting a page fault (including nested TDP faults into L1), tell the injection routine whether or not the fault originated in hardware, i.e. if KVM is effectively forwarding a fault it intercept. For nested TDP fault injection, KVM needs to grab PAGE_WALK vs. GUEST_FINAL information from the VMCB/VMCS, _if_ the fault originated in hardware.
Note, simply checking whether or not the original exit was due a #NPF or EPT Violation isn't sufficient/correct, as the fault being synthesized for L1 may or may not be the "same" fault that triggered a VM-Exit from L2. E.g. if access to emulated MMIO in L2 hits a !PRESENT fault (EPT Violation or #NPF), e.g. because MMIO caching is disabled or it's the first time the GPA has been accessed by L2, then KVM will enter the emulator. If emulating the MMIO instruction then hits a nested TDP fault, e.g. because L2 was accessing MMIO with a MOVSQ (memory-to-memory move), or because L1 has since unmapped the code stream, then the TDP fault synthesized to L1 will not be the same emulated fault the triggered the VM-Exit.
No functional change intended (nothing uses the new param, yet...).
[doebel: keep this tree's split kvm_requeue_exception()/_e()
declarations and EXPORT_SYMBOL_GPL()]
Stable-dep-of: 96b067b59ad9 ("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits")
--- arch/x86/include/asm/kvm_host.h | 18 ++++++++++++++---- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/svm/nested.c | 3 ++- arch/x86/kvm/vmx/nested.c | 3 ++- arch/x86/kvm/x86.c | 16 +++++++++------- 5 files changed, 28 insertions(+), 14 deletions(-)
arch/x86/include/asm/kvm_host.h +14 -4
@@ -451,7 +451,8 @@ struct kvm_mmu {u64 (*get_pdptr)(struct kvm_vcpu *vcpu,int index);int (*page_fault)(struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void (*inject_page_fault)(struct kvm_vcpu *vcpu, -struct x86_exception *fault); +struct x86_exception *fault, +bool from_hardware);gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,struct kvm_mmu *mmu,gpa_t gva_or_gpa,u64 access,struct x86_exception *exception);
@@ -2140,9 +2141,18 @@ void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code);void kvm_queue_exception_p (struct kvm_vcpu *vcpu,unsigned nr,unsigned long payload);void kvm_requeue_exception (struct kvm_vcpu *vcpu,unsigned nr);void kvm_requeue_exception_e (struct kvm_vcpu *vcpu,unsigned nr,u32 error_code); -void kvm_inject_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault); -void kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, -struct x86_exception *fault); +void kvm_inject_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault, +bool from_hardware); +void __kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, +struct x86_exception *fault, +bool from_hardware); + +static inline void kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, +struct x86_exception *fault) +{ +__kvm_inject_emulated_page_fault (vcpu, fault, false); +} +bool kvm_require_cpl (struct kvm_vcpu *vcpu,int required_cpl);bool kvm_require_dr (struct kvm_vcpu *vcpu,int dr);
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -799,7 +799,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault*/ if (!r) {if (!fault->prefetch) -kvm_inject_emulated_page_fault (vcpu, &walker.fault); +__kvm_inject_emulated_page_fault (vcpu, &walker.fault, true);return RET_PF_RETRY ; }
arch/x86/kvm/svm/nested.c +2 -1
@@ -34,7 +34,8 @@#define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECKstatic void nested_svm_inject_npf_exit (struct kvm_vcpu *vcpu, -struct x86_exception *fault) +struct x86_exception *fault, +bool from_hardware) {struct vcpu_svm *svm =to_svm (vcpu);struct vmcb *vmcb = svm->vmcb;
arch/x86/kvm/vmx/nested.c +2 -1
@@ -414,7 +414,8 @@ static void nested_ept_invalidate_addr(struct kvm_vcpu *vcpu, gpa_t eptp, }static void nested_ept_inject_page_fault (struct kvm_vcpu *vcpu, -struct x86_exception *fault) +struct x86_exception *fault, +bool from_hardware) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu);
arch/x86/kvm/x86.c +9 -7
@@ -940,7 +940,8 @@ static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) EMULTYPE_COMPLETE_USER_EXIT); } -void kvm_inject_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault) +void kvm_inject_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault, +bool from_hardware) { ++vcpu->stat.pf_guest;
@@ -957,8 +958,9 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault) fault->address); } -void kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, -struct x86_exception *fault) +void __kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, +struct x86_exception *fault, +bool from_hardware) {struct kvm_mmu *fault_mmu;WARN_ON_ONCE (fault->vector !=PF_VECTOR );
@@ -975,9 +977,9 @@ void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu,kvm_mmu_invalidate_addr (vcpu, fault_mmu, fault->address,KVM_MMU_ROOT_CURRENT ); - fault_mmu->inject_page_fault (vcpu, fault); + fault_mmu->inject_page_fault (vcpu, fault, from_hardware); } -EXPORT_SYMBOL_GPL (kvm_inject_emulated_page_fault); +EXPORT_SYMBOL_GPL (__kvm_inject_emulated_page_fault);void kvm_inject_nmi (struct kvm_vcpu *vcpu) {
@@ -12701,7 +12703,7 @@ bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, fault.nested_page_fault = false; fault.address = work->arch.token; fault.async_page_fault = true; -kvm_inject_page_fault (vcpu, &fault); +kvm_inject_page_fault (vcpu, &fault, false);return true; }else {/*
@@ -12957,7 +12959,7 @@ void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_c fault.address = gva; fault.async_page_fault = false; } - vcpu->arch.walk_mmu->inject_page_fault (vcpu, &fault); + vcpu->arch.walk_mmu->inject_page_fault (vcpu, &fault, true); }EXPORT_SYMBOL_GPL (kvm_fixup_and_inject_pf_error);
Signature · 1 line
2.50.1
PATCH 3/3 KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits
In reply to Bjoern DoebelFrom: Kevin Cheng <[email protected]>
[ Upstream commit 96b067b59ad9ccede585fcadab27543188353bff ]
When injecting an EPT Violation into L2 in response to a fault detected while emulating an L2 GVA access, synthesize the GVA_IS_VALID and GVA_TRANSLATED bits using information provided by the walker, instead of pulling the bits from vmcs02.EXIT_QUALIFICATION. The information in vmcs02.EXIT_QUALIFICATION is valid/correct if and only if the fault being injected into L1 is the direct result of an EPT Violation VM-Exit from L2.
E.g. if KVM is emulating an I/O instruction and the memory operand's translation through L1's EPT fails, using vmcs02.EXIT_QUALIFICATION is wrong as the semantics for EXIT_QUALIFICATION would be for an I/O exit, not an EPT Violation exit.
Opportunistically clean up the formatting for creating the mask of bits to pull from vmcs02.EXIT_QUALIFICATION.
[doebel: no advanced EPT vmexit info here, so the mask keeps only the two GVA bits; set GVA_TRANSLATED via !(access & PFERR_GUEST_PAGE_MASK)]
--- arch/x86/kvm/mmu/paging_tmpl.h | 17 ++++++++++++++++- arch/x86/kvm/vmx/nested.c | 23 +++++++++++++++++++---- 2 files changed, 35 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/paging_tmpl.h +16 -1
@@ -495,7 +495,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,* [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables -* [7:8] - Derived from [7:8] of real exit_qualification +* [7:8] - Derived from "fault stage" access bits * * The other bits are set to 0. */
@@ -509,6 +509,21 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,if (fetch_fault) walker->fault.exit_qualification |=EPT_VIOLATION_ACC_INSTR ; +/* +* KVM doesn't emulate features that access GPAs directly, e.g. +* Intel Processor Trace. Assume the GVA is always valid; when +* propagating faults from hardware, KVM will discard this info +* and use the EXIT_QUALIFICATION bits from the VMCS. +*/ + walker->fault.exit_qualification |=EPT_VIOLATION_GVA_IS_VALID ; + +/* +* Accesses to guest paging structures are not the translation of +* a guest linear address, i.e. GVA_TRANSLATED must be left clear. +*/ +if (!(access &PFERR_GUEST_PAGE_MASK )) + walker->fault.exit_qualification |=EPT_VIOLATION_GVA_TRANSLATED ; +/* * Note, pte_access holds the raw RWX bits from the EPTE, not * ACC_*_MASK flags!
arch/x86/kvm/vmx/nested.c +19 -4
@@ -447,10 +447,25 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, vm_exit_reason =EXIT_REASON_EPT_MISCONFIG ; exit_qualification =0 ; }else { - exit_qualification = fault->exit_qualification; - exit_qualification |=vmx_get_exit_qual (vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | -EPT_VIOLATION_GVA_TRANSLATED ); +const u64 mask =EPT_VIOLATION_GVA_IS_VALID | +EPT_VIOLATION_GVA_TRANSLATED ; + + exit_qualification = fault->exit_qualification & ~mask; + +/* +* Use the EXIT_QUALIFICATION from the VMCS if and only +* if the hardware VM-Exit from L2 was an EPT Violation. +* If the fault is synthesized, then EXIT_QUALIFICATION +* is stale and/or holds entirely different data. And +* conversely, KVM _must_ rely on EXIT_QUALIFICATION if +* the fault came from hardware, because KVM only sees +* and walks the faulting GPA. +*/ +if (from_hardware) + exit_qualification |=vmx_get_exit_qual (vcpu) & mask; +else + exit_qualification |= fault->exit_qualification & mask; + vm_exit_reason =EXIT_REASON_EPT_VIOLATION ; }
Signature · 1 line
2.50.1
Sasha Levin
maintainerQuoted 3 lines
Backport of commit
96b067b59ad9("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits") to 6.12.y, with its two dependencies.
Same as for the 6.18.y series: KVM x86 opts out of stable by default (Documentation/process/maintainer-kvm-x86.rst), and none of the three commits carries Cc: stable or a Fixes: tag.
Bjoern, could you get Sean or Paolo to ack this series (and the matching 6.18.y one) on the thread? I will pick both up once they have.
Signature · 2 lines
Thanks, Sasha
Quoted 10 lines · Sasha Levin
Quoted 3 lines
Backport of commit
96b067b59ad9("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits") to 6.12.y, with its two dependencies.Same as for the 6.18.y series: KVM x86 opts out of stable by default (
Documentation/process/maintainer-kvm-x86.rst), and none of the three commits carries Cc: stable or a Fixes: tag.Bjoern, could you get Sean or Paolo to ack this series (and the matching 6.18.y one) on the thread? I will pick both up once they have.
Quoted 3 lines
Backport of commit
96b067b59ad9("KVM: VMX: Synthesize nested EPT violation GVA_IS_VALID/GVA_TRANSLATED bits") to 6.12.y, with its two dependencies.
Queued the series for 6.12, thanks.
Signature · 2 lines
Thanks, Sasha