KVM: x86: Legal GPA fixes and cleanups
Sean Christopherson
maintainercover KVM: x86: Legal GPA fixes and cleanups
Add helpers to consolidate the GPA reserved bits checks that are scattered all over KVM, and fix a few bugs in the process.
The original motivation was simply to get rid of all the different open coded variations of the checks (there were a lot), but this snowballed into a more ambitious cleanup when I realized common helpers are more or less required to correctly handle repurposed GPA bits, e.g. SEV's C-bit.
The last two patches (use nested VM-Enter failure tracepoints in SVM) aren't directly related to the GPA checks, but the conflicts would be rather messy, so I included them here.
Note, the SEV C-bit changes are technically bug fixes, but getting them in stable kernels would require backporting this entire pile. IMO, it's not worth the effort given that it's extremely unlikely anyone will encounter the bugs in anything but synthetic negative tests.
Based on kvm/queue, commit 3f87cb8253c3 ("KVM: X86: Expose bus lock debug exception to guest").
Sean Christopherson (12):
KVM: x86: Set so called 'reserved CR3 bits in LM mask' at vCPU reset
KVM: nSVM: Don't strip host's C-bit from guest's CR3 when reading
PDPTRs
KVM: x86: Add a helper to check for a legal GPA
KVM: x86: Add a helper to handle legal GPA with an alignment
requirement
KVM: VMX: Use GPA legality helpers to replace open coded equivalents
KVM: nSVM: Use common GPA helper to check for illegal CR3
KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
KVM: x86: Use reserved_gpa_bits to calculate reserved PxE bits
KVM: x86/mmu: Add helper to generate mask of reserved HPA bits
KVM: x86: Add helper to consolidate "raw" reserved GPA mask
calculations
KVM: x86: Move nVMX's consistency check macro to common code
KVM: nSVM: Trace VM-Enter consistency check failures
arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/cpuid.c | 20 +++++- arch/x86/kvm/cpuid.h | 24 +++++-- arch/x86/kvm/mmu/mmu.c | 110 ++++++++++++++++---------------- arch/x86/kvm/mtrr.c | 12 ++-- arch/x86/kvm/svm/nested.c | 35 +++++----- arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/nested.c | 34 +++------- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/x86.c | 11 ++-- arch/x86/kvm/x86.h | 8 +++ 11 files changed, 140 insertions(+), 120 deletions(-)
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 1/12 KVM: x86: Set so called 'reserved CR3 bits in LM mask' at vCPU reset
Set cr3_lm_rsvd_bits, which is effectively an invalid GPA mask, at vCPU reset. The reserved bits check needs to be done even if userspace never configures the guest's CPUID model.
--- arch/x86/kvm/x86.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kvm/x86.c +1 -0
@@ -10091,6 +10091,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)fx_init (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu); + vcpu->arch.cr3_lm_rsvd_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); vcpu->arch.pat =MSR_IA32_CR_PAT_DEFAULT ;
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 2/12 KVM: nSVM: Don't strip host's C-bit from guest's CR3 when reading PDPTRs
In reply to Sean ChristophersonDon't clear the SME C-bit when reading a guest PDPTR, as the GPA (CR3) is in the guest domain.
Barring a bizarre paravirtual use case, this is likely a benign bug. SME is not emulated by KVM, loading SEV guest PDPTRs is doomed as KVM can't use the correct key to read guest memory, and setting guest MAXPHYADDR higher than the host, i.e. overlapping the C-bit, would cause faults in the guest.
Note, for SEV guests, stripping the C-bit is technically aligned with CPU behavior, but for KVM it's the greater of two evils. Because KVM doesn't have access to the guest's encryption key, ignoring the C-bit would at best result in KVM reading garbage. By keeping the C-bit, KVM will fail its read (unless userspace creates a memslot with the C-bit set). The guest will still undoubtedly die, as KVM will use '0' for the PDPTR value, but that's preferable to interpreting encrypted data as a PDPTR.
--- arch/x86/kvm/svm/nested.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm/nested.c +1 -1
@@ -58,7 +58,7 @@ static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index)u64 pdpte;int ret; - ret =kvm_vcpu_read_guest_page (vcpu,gpa_to_gfn (__sme_clr (cr3)), &pdpte, + ret =kvm_vcpu_read_guest_page (vcpu,gpa_to_gfn (cr3), &pdpte,offset_in_page (cr3) + index *8 ,8 );if (ret)return 0 ;
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 3/12 KVM: x86: Add a helper to check for a legal GPA
In reply to Sean ChristophersonAdd a helper to check for a legal GPA, and use it to consolidate code in existing, related helpers. Future patches will extend usage to VMX and SVM code, properly handle exceptions to the maxphyaddr rule, and add more helpers.
No functional change intended.
--- arch/x86/kvm/cpuid.h | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
arch/x86/kvm/cpuid.h +11 -6
@@ -36,9 +36,19 @@ static inline int cpuid_maxphyaddr(struct kvm_vcpu *vcpu)return vcpu->arch.maxphyaddr; } +static inline bool kvm_vcpu_is_legal_gpa (struct kvm_vcpu *vcpu,gpa_t gpa) +{ +return !(gpa >>cpuid_maxphyaddr (vcpu)); +} +static inline bool kvm_vcpu_is_illegal_gpa (struct kvm_vcpu *vcpu,gpa_t gpa) { -return (gpa >=BIT_ULL (cpuid_maxphyaddr (vcpu))); +return !kvm_vcpu_is_legal_gpa (vcpu, gpa); +} + +static inline bool page_address_valid (struct kvm_vcpu *vcpu,gpa_t gpa) +{ +return PAGE_ALIGNED (gpa) &&kvm_vcpu_is_legal_gpa (vcpu, gpa); }struct cpuid_reg {
@@ -324,11 +334,6 @@ static __always_inline void kvm_cpu_cap_check_and_set(unsigned int x86_feature)kvm_cpu_cap_set (x86_feature); } -static inline bool page_address_valid (struct kvm_vcpu *vcpu,gpa_t gpa) -{ -return PAGE_ALIGNED (gpa) && !(gpa >>cpuid_maxphyaddr (vcpu)); -} - static __always_inline bool guest_pv_has(struct kvm_vcpu *vcpu,unsigned int kvm_feature) {
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 4/12 KVM: x86: Add a helper to handle legal GPA with an alignment requirement
In reply to Sean ChristophersonAdd a helper to genericize checking for a legal GPA that also must conform to an arbitrary alignment, and use it in the existing page_address_valid(). Future patches will replace open coded variants in VMX and SVM.
--- arch/x86/kvm/cpuid.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +7 -1
@@ -46,9 +46,15 @@ static inline bool kvm_vcpu_is_illegal_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)return !kvm_vcpu_is_legal_gpa (vcpu, gpa); } +static inline bool kvm_vcpu_is_legal_aligned_gpa (struct kvm_vcpu *vcpu, +gpa_t gpa,gpa_t alignment) +{ +return IS_ALIGNED (gpa, alignment) &&kvm_vcpu_is_legal_gpa (vcpu, gpa); +} +static inline bool page_address_valid (struct kvm_vcpu *vcpu,gpa_t gpa) { -return PAGE_ALIGNED (gpa) &&kvm_vcpu_is_legal_gpa (vcpu, gpa); +return kvm_vcpu_is_legal_aligned_gpa (vcpu, gpa,PAGE_SIZE ); }struct cpuid_reg {
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 5/12 KVM: VMX: Use GPA legality helpers to replace open coded equivalents
In reply to Sean ChristophersonReplace a variety of open coded GPA checks with the recently introduced common helpers.
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 26 +++++++------------------- arch/x86/kvm/vmx/vmx.c | 2 +- 2 files changed, 8 insertions(+), 20 deletions(-)
arch/x86/kvm/vmx/nested.c +7 -19
@@ -775,8 +775,7 @@ static int nested_vmx_check_apicv_controls(struct kvm_vcpu *vcpu, (CC (!nested_cpu_has_vid (vmcs12)) ||CC (!nested_exit_intr_ack_set (vcpu)) ||CC ((vmcs12->posted_intr_nv &0xff00 )) || -CC ((vmcs12->posted_intr_desc_addr &0x3f )) || - CC((vmcs12->posted_intr_desc_addr >> cpuid_maxphyaddr(vcpu))))) + CC(!kvm_vcpu_is_legal_aligned_gpa(vcpu, vmcs12->posted_intr_desc_addr, 64)))) return -EINVAL;/* tpr shadow is needed by all apicv features. */
@@ -789,13 +788,11 @@ static int nested_vmx_check_apicv_controls(struct kvm_vcpu *vcpu,static int nested_vmx_check_msr_switch (struct kvm_vcpu *vcpu,u32 count,u64 addr) { -int maxphyaddr; -if (count ==0 )return 0 ; - maxphyaddr =cpuid_maxphyaddr (vcpu); -if (!IS_ALIGNED (addr,16 ) || addr >> maxphyaddr || - (addr + count *sizeof (struct vmx_msr_entry ) -1 ) >> maxphyaddr) + +if (!kvm_vcpu_is_legal_aligned_gpa (vcpu, addr,16 ) || + !kvm_vcpu_is_legal_gpa (vcpu, (addr + count *sizeof (struct vmx_msr_entry ) -1 )))return -EINVAL ;return 0 ;
@@ -1093,14 +1090,6 @@ static void prepare_vmx_msr_autostore_list(struct kvm_vcpu *vcpu, } } -static bool nested_cr3_valid (struct kvm_vcpu *vcpu,unsigned long val) -{ -unsigned long invalid_mask; - - invalid_mask = (~0ULL ) <<cpuid_maxphyaddr (vcpu); -return (val & invalid_mask) ==0 ; -} -/* * Returns true if the MMU needs to be sync'd on nested VM-Enter/VM-Exit. * tl;dr: the MMU needs a sync if L0 is using shadow paging and L1 didn't
@@ -1152,7 +1141,7 @@ static bool nested_vmx_transition_mmu_sync(struct kvm_vcpu *vcpu)static int nested_vmx_load_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3,bool nested_ept,enum vm_entry_failure_code *entry_failure_code) { -if (CC (!nested_cr3_valid (vcpu, cr3))) { +if (CC (kvm_vcpu_is_illegal_gpa (vcpu, cr3))) { *entry_failure_code =ENTRY_FAIL_DEFAULT ;return -EINVAL ; }
@@ -2666,7 +2655,6 @@ static int nested_vmx_check_nmi_controls(struct vmcs12 *vmcs12)static bool nested_vmx_check_eptp (struct kvm_vcpu *vcpu,u64 new_eptp) {struct vcpu_vmx *vmx =to_vmx (vcpu); -int maxphyaddr =cpuid_maxphyaddr (vcpu);/* Check for memory type validity */ switch (new_eptp &VMX_EPTP_MT_MASK ) {
@@ -2697,7 +2685,7 @@ static bool nested_vmx_check_eptp(struct kvm_vcpu *vcpu, u64 new_eptp) }/* Reserved bits should not be set */ -if (CC (new_eptp >> maxphyaddr || ((new_eptp >>7 ) &0x1f ))) +if (CC (kvm_vcpu_is_illegal_gpa (vcpu, new_eptp) || ((new_eptp >>7 ) &0x1f )))return false;/* AD, if set, should be supported */
@@ -2881,7 +2869,7 @@ static int nested_vmx_check_host_state(struct kvm_vcpu *vcpu,if (CC (!nested_host_cr0_valid (vcpu, vmcs12->host_cr0)) ||CC (!nested_host_cr4_valid (vcpu, vmcs12->host_cr4)) || -CC (!nested_cr3_valid (vcpu, vmcs12->host_cr3))) +CC (kvm_vcpu_is_illegal_gpa (vcpu, vmcs12->host_cr3)))return -EINVAL ; if (CC(is_noncanonical_address(vmcs12->host_ia32_sysenter_esp, vcpu)) ||
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -1114,7 +1114,7 @@ static inline bool pt_can_write_msr(struct vcpu_vmx *vmx)static inline bool pt_output_base_valid (struct kvm_vcpu *vcpu,u64 base) {/* The base must be 128-byte aligned and a legal physical address. */ -return !kvm_vcpu_is_illegal_gpa (vcpu, base) && !(base &0x7f ); +return kvm_vcpu_is_legal_aligned_gpa (vcpu, base,128 ); }static inline void pt_load_msr (struct pt_ctx *ctx,u32 addr_range)
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 6/12 KVM: nSVM: Use common GPA helper to check for illegal CR3
In reply to Sean ChristophersonReplace an open coded check for an invalid CR3 with its equivalent helper.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm/nested.c +1 -1
@@ -367,7 +367,7 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)static int nested_svm_load_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3,bool nested_npt) { -if (cr3 &rsvd_bits (cpuid_maxphyaddr (vcpu),63 )) +if (kvm_vcpu_is_illegal_gpa (vcpu, cr3))return -EINVAL ; if (!nested_npt && is_pae_paging(vcpu) &&
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 7/12 KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
In reply to Sean ChristophersonRename cr3_lm_rsvd_bits to reserved_gpa_bits, and use it for all GPA legality checks. AMD's APM states:
If the C-bit is an address bit, this bit is masked from the guest physical address when it is translated through the nested page tables.
Thus, any access that can conceivably be run through NPT should ignore the C-bit when checking for validity.
For features that KVM emulates in software, e.g. MTRRs, there is no clear direction in the APM for how the C-bit should be handled. For such cases, follow the SME behavior inasmuch as possible, since SEV is is essentially a VM-specific variant of SME. For SME, the APM states:
In this case the upper physical address bits are treated as reserved when the feature is enabled except where otherwise indicated.
Collecting the various relavant SME snippets in the APM and cross- referencing the omissions with Linux kernel code, this leaves MTTRs and APIC_BASE as the only flows that KVM emulates that should _not_ ignore the C-bit.
Note, this means the reserved bit checks in the page tables are technically broken. This will be remedied in a future patch.
Although the page table checks are technically broken, in practice, it's all but guaranteed to be irrelevant. NPT is required for SEV, i.e. shadowing page tables isn't needed in the common case. Theoretically, the checks could be in play for nested NPT, but it's extremely unlikely that anyone is running nested VMs on SEV, as doing so would require L1 to expose sensitive data to L0, e.g. the entire VMCB. And if anyone is running nested VMs, L0 can't read the guest's encrypted memory, i.e. L1 would need to put its NPT in shared memory, in which case the C-bit will never be set. Or, L1 could use shadow paging, but again, if L0 needs to read page tables, e.g. to load PDPTRs, the memory can't be encrypted if L1 has any expectation of L0 doing the right thing.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/cpuid.h | 2 +- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/x86.c | 7 +++---- 6 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -654,7 +654,7 @@ struct kvm_vcpu_arch {int cpuid_nent;struct kvm_cpuid_entry2 *cpuid_entries; -unsigned long cr3_lm_rsvd_bits; +u64 reserved_gpa_bits;int maxphyaddr;int max_tdp_level;
arch/x86/kvm/cpuid.c +1 -1
@@ -194,7 +194,7 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vcpu->arch.cr4_guest_rsvd_bits =__cr4_reserved_bits (guest_cpuid_has, vcpu); - vcpu->arch.cr3_lm_rsvd_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); + vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 );/* Invoke the vendor callback only after the above state is updated. */ static_call (kvm_x86_vcpu_after_set_cpuid)(vcpu);
arch/x86/kvm/cpuid.h +1 -1
@@ -38,7 +38,7 @@ static inline int cpuid_maxphyaddr(struct kvm_vcpu *vcpu)static inline bool kvm_vcpu_is_legal_gpa (struct kvm_vcpu *vcpu,gpa_t gpa) { -return !(gpa >>cpuid_maxphyaddr (vcpu)); +return !(gpa & vcpu->arch.reserved_gpa_bits); }static inline bool kvm_vcpu_is_illegal_gpa (struct kvm_vcpu *vcpu,gpa_t gpa)
arch/x86/kvm/svm/nested.c +1 -1
@@ -241,7 +241,7 @@ static bool nested_vmcb_check_cr3_cr4(struct vcpu_svm *svm,*/ if ((save->efer &EFER_LME ) && (save->cr0 &X86_CR0_PG )) {if (!(save->cr4 &X86_CR4_PAE ) || !(save->cr0 &X86_CR0_PE ) || - (save->cr3 & vcpu->arch.cr3_lm_rsvd_bits)) +kvm_vcpu_is_illegal_gpa (vcpu, save->cr3))return false; }
arch/x86/kvm/svm/svm.c +1 -1
@@ -4079,7 +4079,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (sev_guest (vcpu->kvm)) { best =kvm_find_cpuid_entry (vcpu,0x8000001F ,0 );if (best) - vcpu->arch.cr3_lm_rsvd_bits &= ~(1UL << (best->ebx &0x3f )); + vcpu->arch.reserved_gpa_bits &= ~(1UL << (best->ebx &0x3f )); }if (!kvm_vcpu_apicv_active (vcpu))
arch/x86/kvm/x86.c +3 -4
@@ -1082,8 +1082,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)return 0 ; } -if (is_long_mode (vcpu) && - (cr3 & vcpu->arch.cr3_lm_rsvd_bits)) +if (is_long_mode (vcpu) &&kvm_vcpu_is_illegal_gpa (vcpu, cr3))return 1 ;else if (is_pae_paging (vcpu) && !load_pdptrs (vcpu, vcpu->arch.walk_mmu, cr3))
@@ -9712,7 +9711,7 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)*/ if (!(sregs->cr4 &X86_CR4_PAE ) || !(sregs->efer &EFER_LMA ))return false; -if (sregs->cr3 & vcpu->arch.cr3_lm_rsvd_bits) +if (kvm_vcpu_is_illegal_gpa (vcpu, sregs->cr3))return false; }else {/*
@@ -10091,7 +10090,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)fx_init (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu); - vcpu->arch.cr3_lm_rsvd_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); + vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); vcpu->arch.pat =MSR_IA32_CR_PAT_DEFAULT ;
Signature · 1 line
2.30.0.365.g02bc693789-goog
Re: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 3 lines · Sean Christopherson
- unsigned longcr3_lm_rsvd_bits;
+ u64reserved_gpa_bits;
LAM defines bits above the GFN in CR3:
https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html
KVM doesn't support this today of course, but it might be confusing to try to combine the two concepts.
Sean Christopherson
maintainerRe: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 10 lines · Rick P
Quoted 3 lines · Sean Christopherson
- unsigned longcr3_lm_rsvd_bits;
+ u64reserved_gpa_bits;LAM defines bits above the GFN in CR3:
https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.htmlKVM doesn't support this today of course, but it might be confusing to try to combine the two concepts.
Ah, took me a few minutes, but I see what you're saying. LAM will introduce bits that are repurposed for CR3, but not generic GPAs. And, the behavior is based on CPU support, so it'd make sense to have a mask cached in vcpu->arch as opposed to constantly generating it on the fly.
Definitely agree that having a separate cr3_lm_rsvd_bits or whatever is the right way to go when LAM comes along. Not sure it's worth keeping a duplicate field in the meantime, though it would avoid a small amount of thrash.
Paolo, any thoughts?
Paolo Bonzini
maintainerRe: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 8 lines · Sean Christopherson
Ah, took me a few minutes, but I see what you're saying. LAM will introduce bits that are repurposed for CR3, but not generic GPAs. And, the behavior is based on CPU support, so it'd make sense to have a mask cached in
vcpu->archas opposed to constantly generating it on the fly.Definitely agree that having a separate
cr3_lm_rsvd_bitsor whatever is the right way to go when LAM comes along. Not sure it's worth keeping a duplicate field in the meantime, though it would avoid a small amount of thrash.
We don't even know if the cr3_lm_rsvd_bits would be a field in vcpu->arch, or rather computed on the fly. So renaming the field in vcpu->arch seems like the simplest thing to do now.
Paolo
Re: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
Ah, took me a few minutes, but I see what you're saying. LAM will introduce
bits that are repurposed for CR3, but not generic GPAs. And, the behavior is
based on CPU support, so it'd make sense to have a mask cached invcpu->arch
as opposed to constantly generating it on the fly.Quoted 6 lines
Definitely agree that having a separate
cr3_lm_rsvd_bitsor whatever is the
right way to go when LAM comes along. Not sure it's worth keeping a duplicate
field in the meantime, though it would avoid a small amount of thrash.
Quoted 3 lines
We don't even know if the
cr3_lm_rsvd_bitswould be a field invcpu->arch, or rather computed on the fly. So renaming the field invcpu->archseems like the simplest thing to do now.
Fair enough. But just to clarify, I meant that I thought the code would be more confusing to use illegal gpa bit checks for checking cr3. It seems they are only incidentally the same value. Alternatively there could be something like a is_rsvd_cr3_bits() helper that just uses reserved_gpa_bits for now. Probably put the comment in the wrong place. It's a minor point in any case.
Sean Christopherson
maintainerRe: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 17 lines · Rick P
Quoted 16 lines · Paolo Bonzini
Quoted 14 lines · Sean Christopherson
Ah, took me a few minutes, but I see what you're saying. LAM will introduce
bits that are repurposed for CR3, but not generic GPAs. And, the behavior is
based on CPU support, so it'd make sense to have a mask cached invcpu->arch
as opposed to constantly generating it on the fly.Definitely agree that having a separate
cr3_lm_rsvd_bitsor whatever is the
right way to go when LAM comes along. Not sure it's worth keeping a duplicate
field in the meantime, though it would avoid a small amount of thrash.
Quoted 7 lines
Quoted 3 lines
We don't even know if the
cr3_lm_rsvd_bitswould be a field invcpu->arch, or rather computed on the fly. So renaming the field invcpu->archseems like the simplest thing to do now.Fair enough. But just to clarify, I meant that I thought the code would be more confusing to use illegal gpa bit checks for checking cr3. It seems they are only incidentally the same value.
Hmm, yeah, bits 63:52 are incidental. Bits 52:M are not, though. If/when we need to special case CR3, I would like to take a similar approach to __reset_rsvds_bits_mask(), where the high reserved bits start from reserved_gpa_bits and mask off the bits that can't be encoded into a PxE.
Quoted 3 lines
Alternatively there could be something like a
is_rsvd_cr3_bits()helper that just usesreserved_gpa_bitsfor now. Probably put the comment in the wrong place. It's a minor point in any case.
That thought crossed my mind, too. Maybe kvm_vcpu_is_illegal_cr3() to match the gpa helpers?
Paolo Bonzini
maintainerRe: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 5 lines · Sean Christopherson
Quoted 3 lines
Alternatively there could be something like a
is_rsvd_cr3_bits()helper that just usesreserved_gpa_bitsfor now. Probably put the comment in the wrong place. It's a minor point in any case.That thought crossed my mind, too. Maybe
kvm_vcpu_is_illegal_cr3()to match the gpa helpers?
Yes, that's certainly a good name but it doesn't have to be done now. Or at least, if you do it, somebody is guaranteed to send a patch after one month because the wrapper is useless. :)
Paolo
Sean Christopherson
maintainerRe: [PATCH 07/12] KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
Quoted 10 lines · Paolo Bonzini
Quoted 5 lines · Sean Christopherson
Quoted 3 lines
Alternatively there could be something like a
is_rsvd_cr3_bits()helper that just usesreserved_gpa_bitsfor now. Probably put the comment in the wrong place. It's a minor point in any case.That thought crossed my mind, too. Maybe
kvm_vcpu_is_illegal_cr3()to match the gpa helpers?Yes, that's certainly a good name but it doesn't have to be done now. Or at least, if you do it, somebody is guaranteed to send a patch after one month because the wrapper is useless. :)
LOL, I can see myself doing that...
Sean Christopherson
maintainerPATCH 8/12 KVM: x86: Use reserved_gpa_bits to calculate reserved PxE bits
In reply to Sean ChristophersonUse reserved_gpa_bits, which accounts for exceptions to the maxphyaddr rule, e.g. SEV's C-bit, for the page {table,directory,etc...} entry (PxE) reserved bits checks. For SEV, the C-bit is ignored by hardware when walking pages tables, e.g. the APM states:
Note that while the guest may choose to set the C-bit explicitly on instruction pages and page table addresses, the value of this bit is a don't-care in such situations as hardware always performs these as private accesses.
Such behavior is expected to hold true for other features that repurpose GPA bits, e.g. KVM could theoretically emulate SME or MKTME, which both allow non-zero repurposed bits in the page tables. Conceptually, KVM should apply reserved GPA checks universally, and any features that do not adhere to the basic rule should be explicitly handled, i.e. if a GPA bit is repurposed but not allowed in page tables for whatever reason.
Refactor __reset_rsvds_bits_mask() to take the pre-generated reserved bits mask, and opportunistically clean up its code, e.g. to align lines and comments.
Practically speaking, this is change is a likely a glorified nop given the current KVM code base. SEV's C-bit is the only repurposed GPA bit, and KVM doesn't support shadowing encrypted page tables (which is theoretically possible via SEV debug APIs).
--- arch/x86/kvm/cpuid.c | 10 ++-- arch/x86/kvm/mmu/mmu.c | 104 ++++++++++++++++++++--------------------- arch/x86/kvm/x86.c | 3 +- 3 files changed, 58 insertions(+), 59 deletions(-)
arch/x86/kvm/cpuid.c +7 -3
@@ -188,16 +188,20 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_update_pv_runtime (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu); -kvm_mmu_reset_context (vcpu); + vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 );kvm_pmu_refresh (vcpu); vcpu->arch.cr4_guest_rsvd_bits =__cr4_reserved_bits (guest_cpuid_has, vcpu); - vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); -/* Invoke the vendor callback only after the above state is updated. */ static_call (kvm_x86_vcpu_after_set_cpuid)(vcpu); + +/* +* Except for the MMU, which needs to be reset after any vendor +* specific adjustments to the reserved GPA bits. +*/ +kvm_mmu_reset_context (vcpu); }static int is_efer_nx (void )
arch/x86/kvm/mmu/mmu.c +50 -54
184 lines not on this page · raw message
arch/x86/kvm/x86.c +1 -2
@@ -761,8 +761,7 @@ static int kvm_read_nested_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn,static inline u64 pdptr_rsvd_bits (struct kvm_vcpu *vcpu) { -return rsvd_bits (cpuid_maxphyaddr (vcpu),63 ) |rsvd_bits (5 ,8 ) | -rsvd_bits (1 ,2 ); +return vcpu->arch.reserved_gpa_bits |rsvd_bits (5 ,8 ) |rsvd_bits (1 ,2 ); }/*
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 9/12 KVM: x86/mmu: Add helper to generate mask of reserved HPA bits
In reply to Sean ChristophersonAdd a helper to generate the mask of reserved PA bits in the host.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -5
@@ -4123,6 +4123,11 @@ static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu, vcpu->arch.reserved_gpa_bits, execonly); } +static inline u64 reserved_hpa_bits (void ) +{ +return rsvd_bits (shadow_phys_bits,63 ); +} +/* * the page table on host is the shadow page table for the page * table in guest or amd nested guest, its mmu features completely
@@ -4142,7 +4147,7 @@ reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context)*/ shadow_zero_check = &context->shadow_zero_check;__reset_rsvds_bits_mask (vcpu, shadow_zero_check, -rsvd_bits (shadow_phys_bits,63 ), +reserved_hpa_bits (), context->shadow_root_level, uses_nx,guest_cpuid_has (vcpu,X86_FEATURE_GBPAGES ),is_pse (vcpu), true);
@@ -4179,14 +4184,13 @@ reset_tdp_shadow_zero_bits_mask(struct kvm_vcpu *vcpu,if (boot_cpu_is_amd ())__reset_rsvds_bits_mask (vcpu, shadow_zero_check, -rsvd_bits (shadow_phys_bits,63 ), +reserved_hpa_bits (), context->shadow_root_level, false,boot_cpu_has (X86_FEATURE_GBPAGES ), true, true);else __reset_rsvds_bits_mask_ept (shadow_zero_check, -rsvd_bits (shadow_phys_bits,63 ), - false); +reserved_hpa_bits (), false);if (!shadow_me_mask)return ;
@@ -4206,7 +4210,7 @@ reset_ept_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context, bool execonly) {__reset_rsvds_bits_mask_ept (&context->shadow_zero_check, -rsvd_bits (shadow_phys_bits,63 ), execonly); +reserved_hpa_bits (), execonly); }#define BYTE_MASK (access) \
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 10/12 KVM: x86: Add helper to consolidate "raw" reserved GPA mask calculations
In reply to Sean ChristophersonAdd a helper to generate the mask of reserved GPA bits _without_ any adjustments for repurposed bits, and use it to replace a variety of open coded variants in the MTRR and APIC_BASE flows.
No functional change intended.
--- arch/x86/kvm/cpuid.c | 12 +++++++++++- arch/x86/kvm/cpuid.h | 1 + arch/x86/kvm/mtrr.c | 12 ++++++------ arch/x86/kvm/x86.c | 4 ++-- 4 files changed, 20 insertions(+), 9 deletions(-)
arch/x86/kvm/cpuid.c +11 -1
@@ -188,7 +188,7 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_update_pv_runtime (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu); - vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); + vcpu->arch.reserved_gpa_bits =kvm_vcpu_reserved_gpa_bits_raw (vcpu);kvm_pmu_refresh (vcpu); vcpu->arch.cr4_guest_rsvd_bits =
@@ -242,6 +242,16 @@ int cpuid_query_maxphyaddr(struct kvm_vcpu *vcpu)return 36 ; } +/* +* This "raw" version returns the reserved GPA bits without any adjustments for +* encryption technologies that usurp bits. The raw mask should be used if and +* only if hardware does _not_ strip the usurped bits, e.g. in virtual MTRRs. +*/ +u64 kvm_vcpu_reserved_gpa_bits_raw (struct kvm_vcpu *vcpu) +{ +return rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); +} +/* when an old userspace process fills a new kernel module */ int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid *cpuid,
arch/x86/kvm/cpuid.h +1 -0
@@ -30,6 +30,7 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx, bool exact_only);int cpuid_query_maxphyaddr (struct kvm_vcpu *vcpu); +u64 kvm_vcpu_reserved_gpa_bits_raw (struct kvm_vcpu *vcpu);static inline int cpuid_maxphyaddr (struct kvm_vcpu *vcpu) {
arch/x86/kvm/mtrr.c +6 -6
@@ -75,7 +75,7 @@ bool kvm_mtrr_valid(struct kvm_vcpu *vcpu, u32 msr, u64 data)/* variable MTRRs */ WARN_ON (!(msr >=0x200 && msr <0x200 +2 *KVM_NR_VAR_MTRR )); - mask = (~0ULL ) <<cpuid_maxphyaddr (vcpu); + mask =kvm_vcpu_reserved_gpa_bits_raw (vcpu);if ((msr &1 ) ==0 ) {/* MTRR base */ if (!valid_mtrr_type (data &0xff ))
@@ -351,14 +351,14 @@ static void set_var_mtrr_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data)if (var_mtrr_range_is_valid (cur))list_del (&mtrr_state->var_ranges[index].node); -/* Extend the mask with all 1 bits to the left, since those -* bits must implicitly be 0. The bits are then cleared -* when reading them. +/* +* Set all illegal GPA bits in the mask, since those bits must +* implicitly be 0. The bits are then cleared when reading them. */ if (!is_mtrr_mask) cur->base = data;else - cur->mask = data | (-1LL <<cpuid_maxphyaddr (vcpu)); + cur->mask = data |kvm_vcpu_reserved_gpa_bits_raw (vcpu);/* add it to the list if it's enabled. */ if (var_mtrr_range_is_valid (cur)) {
@@ -426,7 +426,7 @@ int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)else *pdata = vcpu->arch.mtrr_state.var_ranges[index].mask; - *pdata &= (1ULL <<cpuid_maxphyaddr (vcpu)) -1 ; + *pdata &= ~kvm_vcpu_reserved_gpa_bits_raw (vcpu); }return 0 ;
arch/x86/kvm/x86.c +2 -2
@@ -408,7 +408,7 @@ int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info) {enum lapic_mode old_mode =kvm_get_apic_mode (vcpu);enum lapic_mode new_mode =kvm_apic_mode (msr_info->data); -u64 reserved_bits = ((~0ULL ) <<cpuid_maxphyaddr (vcpu)) |0x2ff | +u64 reserved_bits =kvm_vcpu_reserved_gpa_bits_raw (vcpu) |0x2ff | (guest_cpuid_has (vcpu,X86_FEATURE_X2APIC ) ?0 :X2APIC_ENABLE );if ((msr_info->data & reserved_bits) !=0 || new_mode ==LAPIC_MODE_INVALID )
@@ -10089,7 +10089,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)fx_init (vcpu); vcpu->arch.maxphyaddr =cpuid_query_maxphyaddr (vcpu); - vcpu->arch.reserved_gpa_bits =rsvd_bits (cpuid_maxphyaddr (vcpu),63 ); + vcpu->arch.reserved_gpa_bits =kvm_vcpu_reserved_gpa_bits_raw (vcpu); vcpu->arch.pat =MSR_IA32_CR_PAT_DEFAULT ;
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 11/12 KVM: x86: Move nVMX's consistency check macro to common code
In reply to Sean ChristophersonMove KVM's CC() macro to x86.h so that it can be reused by nSVM. Debugging VM-Enter is as painful on SVM as it is on VMX.
Rename the more visible macro to KVM_NESTED_VMENTER_CONSISTENCY_CHECK to avoid any collisions with the uber-concise "CC".
--- arch/x86/kvm/vmx/nested.c | 8 +------- arch/x86/kvm/x86.h | 8 ++++++++ 2 files changed, 9 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -7
@@ -21,13 +21,7 @@ module_param_named(enable_shadow_vmcs, enable_shadow_vmcs, bool, S_IRUGO);static bool __read_mostly nested_early_check =0 ;module_param (nested_early_check, bool,S_IRUGO ); -#define CC (consistency_check) \ -({ \ - bool failed = (consistency_check); \ - if (failed) \ - trace_kvm_nested_vmenter_failed(#consistency_check, 0); \ - failed; \ -}) +#define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK/* * Hyper-V requires all of these, so mark them as supported even though
arch/x86/kvm/x86.h +8 -0
@@ -8,6 +8,14 @@#include "kvm_cache_regs.h" #include "kvm_emulate.h" +#define KVM_NESTED_VMENTER_CONSISTENCY_CHECK (consistency_check) \ +({ \ + bool failed = (consistency_check); \ + if (failed) \ + trace_kvm_nested_vmenter_failed(#consistency_check, 0); \ + failed; \ +}) +#define KVM_DEFAULT_PLE_GAP 128#define KVM_VMX_DEFAULT_PLE_WINDOW 4096#define KVM_DEFAULT_PLE_WINDOW_GROW 2
Signature · 1 line
2.30.0.365.g02bc693789-goog
Sean Christopherson
maintainerPATCH 12/12 KVM: nSVM: Trace VM-Enter consistency check failures
In reply to Sean ChristophersonUse trace_kvm_nested_vmenter_failed() and its macro magic to trace consistency check failures on nested VMRUN. Tracing such failures by running the buggy VMM as a KVM guest is often the only way to get a precise explanation of why VMRUN failed.
--- arch/x86/kvm/svm/nested.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-)
arch/x86/kvm/svm/nested.c +19 -14
@@ -29,6 +29,8 @@#include "lapic.h" #include "svm.h" +#define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK +static void nested_svm_inject_npf_exit (struct kvm_vcpu *vcpu,struct x86_exception *fault) {
@@ -216,14 +218,13 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)static bool nested_vmcb_check_controls (struct vmcb_control_area *control) { -if ((vmcb_is_intercept (control,INTERCEPT_VMRUN )) ==0 ) +if (CC (!vmcb_is_intercept (control,INTERCEPT_VMRUN )))return false; -if (control->asid ==0 ) +if (CC (control->asid ==0 ))return false; -if ((control->nested_ctl &SVM_NESTED_CTL_NP_ENABLE ) && - !npt_enabled) +if (CC ((control->nested_ctl &SVM_NESTED_CTL_NP_ENABLE ) && !npt_enabled))return false;return true;
@@ -240,32 +241,36 @@ static bool nested_vmcb_check_cr3_cr4(struct vcpu_svm *svm,* CR0.PG && EFER.LME. */ if ((save->efer &EFER_LME ) && (save->cr0 &X86_CR0_PG )) { -if (!(save->cr4 &X86_CR4_PAE ) || !(save->cr0 &X86_CR0_PE ) || -kvm_vcpu_is_illegal_gpa (vcpu, save->cr3)) +if (CC (!(save->cr4 &X86_CR4_PAE )) || +CC (!(save->cr0 &X86_CR0_PE )) || +CC (kvm_vcpu_is_illegal_gpa (vcpu, save->cr3)))return false; } -return kvm_is_valid_cr4 (&svm->vcpu, save->cr4); +if (CC (!kvm_is_valid_cr4 (vcpu, save->cr4))) +return false; + +return true; }/* Common checks that apply to both L1 and L2 state. */ static bool nested_vmcb_valid_sregs (struct vcpu_svm *svm,struct vmcb_save_area *save) { -if (!(save->efer &EFER_SVME )) +if (CC (!(save->efer &EFER_SVME )))return false; -if (((save->cr0 &X86_CR0_CD ) ==0 && (save->cr0 &X86_CR0_NW )) || - (save->cr0 & ~0xffffffffULL )) +if (CC ((save->cr0 &X86_CR0_CD ) ==0 && (save->cr0 &X86_CR0_NW )) || +CC (save->cr0 & ~0xffffffffULL ))return false; -if (!kvm_dr6_valid (save->dr6) || !kvm_dr7_valid (save->dr7)) +if (CC (!kvm_dr6_valid (save->dr6)) ||CC (!kvm_dr7_valid (save->dr7)))return false;if (!nested_vmcb_check_cr3_cr4 (svm, save))return false; -if (!kvm_valid_efer (&svm->vcpu, save->efer)) +if (CC (!kvm_valid_efer (&svm->vcpu, save->efer)))return false;return true;
@@ -367,12 +372,12 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)static int nested_svm_load_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3,bool nested_npt) { -if (kvm_vcpu_is_illegal_gpa (vcpu, cr3)) +if (CC (kvm_vcpu_is_illegal_gpa (vcpu, cr3)))return -EINVAL ;if (!nested_npt &&is_pae_paging (vcpu) && (cr3 !=kvm_read_cr3 (vcpu) ||pdptrs_changed (vcpu))) { -if (!load_pdptrs (vcpu, vcpu->arch.walk_mmu, cr3)) +if (CC (!load_pdptrs (vcpu, vcpu->arch.walk_mmu, cr3)))return -EINVAL ; }
Signature · 1 line
2.30.0.365.g02bc693789-goog
Paolo Bonzini
maintainerQuoted 12 lines · Sean Christopherson
Add helpers to consolidate the GPA reserved bits checks that are scattered all over KVM, and fix a few bugs in the process.
The original motivation was simply to get rid of all the different open coded variations of the checks (there were a lot), but this snowballed into a more ambitious cleanup when I realized common helpers are more or less required to correctly handle repurposed GPA bits, e.g. SEV's C-bit.
The last two patches (use nested VM-Enter failure tracepoints in SVM) aren't directly related to the GPA checks, but the conflicts would be rather messy, so I included them here.
Quoted 7 lines
Note, the SEV C-bit changes are technically bug fixes, but getting them in stable kernels would require backporting this entire pile. IMO, it's not worth the effort given that it's extremely unlikely anyone will encounter the bugs in anything but synthetic negative tests.
Based on kvm/queue, commit
3f87cb8253c3("KVM: X86: Expose bus lock debug exception to guest").
Queued 1 for 5.11 and 2-10 for 5.12; the VMCB01/VMCB02 patches are unlikely to make it in 5.12 so 11-12 won't be in kvm/next anytime soon---but you don't have to care about them anyway.
Paolo
Quoted 30 lines
Sean Christopherson (12):
KVM: x86: Set so called 'reserved CR3 bits in LM mask' at vCPU reset
KVM: nSVM: Don't strip host's C-bit from guest's CR3 when reading
PDPTRs
KVM: x86: Add a helper to check for a legal GPA
KVM: x86: Add a helper to handle legal GPA with an alignment
requirement
KVM: VMX: Use GPA legality helpers to replace open coded equivalents
KVM: nSVM: Use common GPA helper to check for illegal CR3
KVM: x86: SEV: Treat C-bit as legal GPA bit regardless of vCPU mode
KVM: x86: Usereserved_gpa_bitsto calculate reserved PxE bits
KVM: x86/mmu: Add helper to generate mask of reserved HPA bits
KVM: x86: Add helper to consolidate "raw" reserved GPA mask
calculations
KVM: x86: Move nVMX's consistency check macro to common code
KVM: nSVM: Trace VM-Enter consistency check failuresarch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/cpuid.c | 20 +++++- arch/x86/kvm/cpuid.h | 24 +++++-- arch/x86/kvm/mmu/mmu.c | 110 ++++++++++++++++---------------- arch/x86/kvm/mtrr.c | 12 ++-- arch/x86/kvm/svm/nested.c | 35 +++++----- arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/nested.c | 34 +++------- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/x86.c | 11 ++-- arch/x86/kvm/x86.h | 8 +++ 11 files changed, 140 insertions(+), 120 deletions(-)