KVM: nVMX: Honor L1-owned CR0 bits for LMSW
Subsystems (5)
Sean Christopherson
maintainerv3 cover KVM: nVMX: Honor L1-owned CR0 bits for LMSW
Fix a bug where KVM ignores vmcs12's CR0 guest/host mask when emulating an LMSW from L2, i.e. an LMSW tha exits to L0 but not to L1.
v3:
- Handle the read_shadow in handle_set_cr0() (and for CR4, because why not).
- Kill off kvm_lmsw() entirely.
- Add a bunch of comments to the selftest to explain the dangers of running with CR0.TS=1.
- Test MOV CR0 as well as LMSW.
- Add a patch to take full ownership of the must-be-1 CR0 and CR4 bits.
v2:
- https://lore.kernel.org/all/[email protected]
- Patch 2: order the CR0 store before FNOP and check L2's whole CR0.
v1: https://lore.kernel.org/all/[email protected]
Mahmoud Abumandour (3):
KVM: nVMX: Honor L1-owned CR0 bits when emulating intercepted LMSW for
L2
KVM: nVMX: Synthesize #GP if L2 writes a "bad" CR0 value via LMSW
KVM: selftests: Verify KVM handling of L1-owned CR0 bits on writes
from L2
Sean Christopherson (2):
KVM: nVMX: Preserve L1-owned bits in CR{0,4} read shadows
KVM: nVMX: Take full ownership of the post-VMXON CR{0,4} must-be-1
bits
Documentation/virt/kvm/api.rst | 9 ++- arch/x86/kvm/regs.c | 6 -- arch/x86/kvm/regs.h | 1 - arch/x86/kvm/vmx/nested.c | 30 +------- arch/x86/kvm/vmx/vmx.c | 40 +++++----- .../selftests/kvm/include/x86/processor.h | 5 ++ .../selftests/kvm/x86/feature_msrs_test.c | 2 +- .../kvm/x86/nested_exceptions_test.c | 73 +++++++++++++++++++ 8 files changed, 106 insertions(+), 60 deletions(-)
base-commit: 7e60162b16f9da0d2fa83da5c1723b6853a0ac1d
Signature · 1 line
2.56.0.385.gac671c819a-goog
Sean Christopherson
maintainerPATCH v3 1/5 KVM: nVMX: Preserve L1-owned bits in CR{0,4} read shadows
When emulating an intercepted MOV CR{0,4} for L2, i.e. when L2 is toggling CR{0,4} bits that L0 intercepts but L1 does not, explicitly preserve L1-owned bits in the vmcs02 read shadow instead of subtly relying on the interception logic to preserve the bits. For straight MOV instructions, relying on the interception logic works, because KVM would have synthesized a nested VM-Exit to L1 if L2 were toggling L1-owned bits, i.e. if L2 were explicitly changing any of the shadowed bits.
However, for LMSW, which preserves bits 31:4, and conditionally preserves CR0.PE (bit 0), the effective value being set by L2 can diverge from the read shadow *without* triggering a nested VM-Exit. E.g. if CR0.PE=1 in "hardware", CR0_READ_SHADOW.PE=0, and CR0_GUEST_HOST_MASK.PE=1, then a LMSW with CR0.PE=0 won't VM-Exit, but will preserve/set CR0.PE=1 in "hardware".
Explicitly preserving the shadowed bits will allow using handle_set_cr0() to honor L1's guest/host mask when emulating LMSW.
--- arch/x86/kvm/vmx/vmx.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
arch/x86/kvm/vmx/vmx.c +13 -17
@@ -5629,22 +5629,19 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val) {if (is_guest_mode (vcpu)) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); -unsigned long orig_val = val; +u64 mask = vmcs12->cr0_guest_host_mask;/* -* We get here when L2 changed cr0 in a way that did not change -* any of L1's shadowed bits (see nested_vmx_exit_handled_cr), -* but did change L0 shadowed bits. So we first calculate the -* effective cr0 value that L1 would like to write into the -* hardware. It consists of the L2-owned bits from the new -* value combined with the L1-owned bits from L1's guest_cr0. +* L2 is writing a CR0 value that isn't modifying any of L1's +* shadowed bits (see nested_vmx_exit_handled_cr()), but *is* +* modifying L0 shadowed bits. Set the "hardware" CR0 and the +* read shadow using the L2-owned bits from the new value, +* combined with the L1-owned bits. */ - val = (val & ~vmcs12->cr0_guest_host_mask) | - (vmcs12->guest_cr0 & vmcs12->cr0_guest_host_mask); - -if (kvm_set_cr0 (vcpu, val)) + val &= ~mask; +if (kvm_set_cr0 (vcpu, val | (vmcs12->guest_cr0 & mask)))return 1 ; -vmcs_writel (CR0_READ_SHADOW , orig_val); +vmcs_writel (CR0_READ_SHADOW , val | (vmcs12->cr0_read_shadow & mask));return 0 ; }else {return kvm_set_cr0 (vcpu, val);
@@ -5655,14 +5652,13 @@ static int handle_set_cr4(struct kvm_vcpu *vcpu, unsigned long val) {if (is_guest_mode (vcpu)) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); -unsigned long orig_val = val; +u64 mask = vmcs12->cr4_guest_host_mask;/* analogously to handle_set_cr0 */ - val = (val & ~vmcs12->cr4_guest_host_mask) | - (vmcs12->guest_cr4 & vmcs12->cr4_guest_host_mask); -if (kvm_set_cr4 (vcpu, val)) + val &= ~mask; +if (kvm_set_cr4 (vcpu, val | (vmcs12->guest_cr4 & mask)))return 1 ; -vmcs_writel (CR4_READ_SHADOW , orig_val); +vmcs_writel (CR4_READ_SHADOW , val | (vmcs12->cr4_read_shadow & mask));return 0 ; }else return kvm_set_cr4 (vcpu, val);
Signature · 1 line
2.56.0.385.gac671c819a-goog
Sean Christopherson
maintainerPATCH v3 2/5 KVM: nVMX: Honor L1-owned CR0 bits when emulating intercepted LMSW for L2
In reply to Sean ChristophersonFrom: Mahmoud Abumandour <[email protected]>
When emulating an intercepted LMSW on behalf L2, i.e. when LMSW exits to L0 but not to L1, emulate it via handle_set_cr0(), as is done for MOV to CR0, so that KVM honor's the guest/host mask and read shadow from vmcs12. This fixes a bug where so L2 could change L1-owned CR0 bits, e.g. clear TS, and read their effective values instead of the read shadow.
Drop kvm_lmsw() as it is no longer used, and opportunistically trace the effectively CR0 value (previously, KVM traced the raw CR0.PE value), e.g. so that users don't have to piece together that CR0.PE will be preserved if it was already '1'.
--- arch/x86/kvm/regs.c | 6 ------ arch/x86/kvm/regs.h | 1 - arch/x86/kvm/vmx/vmx.c | 6 ++++-- 3 files changed, 4 insertions(+), 9 deletions(-)
arch/x86/kvm/regs.c +0 -6
@@ -303,12 +303,6 @@ int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_set_cr0); -void kvm_lmsw (struct kvm_vcpu *vcpu,unsigned long msw) -{ - (void )kvm_set_cr0 (vcpu,kvm_read_cr0_bits (vcpu, ~0x0eul ) | (msw &0x0f )); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_lmsw); -int kvm_set_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3) {bool skip_tlb_flush = false;
arch/x86/kvm/regs.h +0 -1
@@ -62,7 +62,6 @@ int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val);unsigned long kvm_get_dr (struct kvm_vcpu *vcpu,int dr);unsigned long kvm_get_dr6_fixed_1 (struct kvm_vcpu *vcpu);unsigned long kvm_get_cr8 (struct kvm_vcpu *vcpu); -void kvm_lmsw (struct kvm_vcpu *vcpu,unsigned long msw);int load_pdptrs (struct kvm_vcpu *vcpu,unsigned long cr3);static inline bool is_long_mode (struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/vmx.c +4 -2
@@ -5743,8 +5743,10 @@ static int handle_cr(struct kvm_vcpu *vcpu)break ;case 3 :/* lmsw */ val = (exit_qualification >>LMSW_SOURCE_DATA_SHIFT ) &0x0f ; -trace_kvm_cr_write (0 , (kvm_read_cr0_bits (vcpu, ~0xful ) | val)); -kvm_lmsw (vcpu, val); + val |=kvm_read_cr0_bits (vcpu, ~0x0eul ); + +trace_kvm_cr_write (cr, val); + (void )handle_set_cr0 (vcpu, val);return kvm_skip_emulated_instruction (vcpu);default :
Signature · 1 line
2.56.0.385.gac671c819a-goog
Sean Christopherson
maintainerPATCH v3 3/5 KVM: nVMX: Synthesize #GP if L2 writes a "bad" CR0 value via LMSW
In reply to Sean ChristophersonFrom: Mahmoud Abumandour <[email protected]>
Synthesize a #GP if writing CR0 when emulating LMSW fails, as it's possible for LMSW to #GP on a bad value when executed post-VMXON, i.e. if LMSW attempts to set/clear bits in violation of MSR_IA32_VMX_CR0_FIXED{0,1}.
Outside of VMX operation, LMSW can only arbitrarily modify bits 3:1 and can only set bit 0 (PE). I.e. LMSW can't modify any CR0 bits that are unsupported or are subject to additional checks and thus should never #GP due to a bad value.
Note, KVM has a virtualization hole where it doesn't enforce must-be-1 CR0 or CR4 bits that KVM doesn't intercepted. E.g. if userspace marked CR0.TS as a must-be-1 bit (while post-VMXON), then the guest could clear CR0.TS. That hole will be plugged shortly.
Note #2, with the virtualization hole plugged, it *should* be impossible for LMSW to #GP on a bad value. Unfortunately, KVM's save/restore ioctls allow userspace to load a CR0 value that violates the must-be-1 masks, i.e. userspace can make the impossible happen (and plugging *those* holes is all but guaranteed to be a waste of time). Ideally, KVM would WARN, but since userspace could trip the WARN, take the easy route and handle failure appropriately.
--- arch/x86/kvm/vmx/vmx.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/vmx.c +2 -3
@@ -5746,9 +5746,8 @@ static int handle_cr(struct kvm_vcpu *vcpu) val |=kvm_read_cr0_bits (vcpu, ~0x0eul );trace_kvm_cr_write (cr, val); - (void )handle_set_cr0 (vcpu, val); - -return kvm_skip_emulated_instruction (vcpu); + err =handle_set_cr0 (vcpu, val); +return kvm_complete_insn_gp (vcpu, err);default :break ; }
Signature · 1 line
2.56.0.385.gac671c819a-goog
Sean Christopherson
maintainerPATCH v3 4/5 KVM: nVMX: Take full ownership of the post-VMXON CR{0,4} must-be-1 bits
In reply to Sean ChristophersonEffectively drop support for restoring/stuffing fixed-0 CR{0,4} MSRs, as restoring to anything other than KVM's hardcoded value is broken, and is absolutely not worth fixing. KVM doesn't adjust the "guest owned" masks based on the MSR values, and so if userspace were to require any of the passed-through bits to be '1', KVM would fail to prevent the guest from clearing a must-be-1 bit.
Even worse, if userspace disables KVM_X86_QUIRK_STUFF_FEATURE_MSRS without ever setting the CR{0,4} FIXED0 MSRs, then KVM would allow running with an impossible configuration, as KVM would advertise to L1 that there are *no* must-be-1 bits, but couldn't honor that because VM-Entry would fail due to the bits still being must-be-1 in real hardware. E.g. KVM would tell the guest that CR4.VMXE can be toggled at will, in complete violation of VMXON.
Don't try to fix KVM, as no known VMM sets values other than KVM's own hardcoded value. KVM already requires the incoming value be a superset of KVM's value, i.e. KVM doesn't allowed reducing the set of must-be-1 bits, and so it's only possible for userspace to require *extra* bits to always be 1. Drop KVM's "support" before anyone tries to utilize it, e.g. to harden a guest or something, as honoring things like CR0.TS in particular would be painful, e.g. KVM would need to handle CLTS interception.
--- Documentation/virt/kvm/api.rst | 9 +++--- arch/x86/kvm/vmx/nested.c | 30 ++----------------- arch/x86/kvm/vmx/vmx.c | 3 ++ .../selftests/kvm/x86/feature_msrs_test.c | 2 +- 4 files changed, 11 insertions(+), 33 deletions(-)
Documentation/virt/kvm/api.rst +5 -4
@@ -8764,10 +8764,11 @@ KVM_X86_QUIRK_STUFF_FEATURE_MSRS By default, at vCPU creation, KVM set i.e. treats the feature MSRs like CPUID leaves and gives userspace full control of the vCPU model definition. This quirk does - not affect VMX MSRs CR0/CR4_FIXED1 (0x487 - and 0x489), as KVM does now allow them to - be set by userspace (KVM sets them based on - guest CPUID, for safety purposes). + not affect VMX MSRs CR0/CR4_FIXED{0,1} + (0x486 - 0x489), as KVM does now allow them + to be set by userspace (KVM sets them based + on the virtual CPU model, for simplicity and + safety purposes). KVM_X86_QUIRK_IGNORE_GUEST_PAT By default, on Intel platforms, KVM ignores guest PAT and forces the effective memory
arch/x86/kvm/vmx/nested.c +2 -28
@@ -1508,33 +1508,6 @@ static int vmx_restore_vmx_ept_vpid_cap(struct vcpu_vmx *vmx, u64 data)return 0 ; } -static u64 *vmx_get_fixed0_msr (struct nested_vmx_msrs *msrs,u32 msr_index) -{ -switch (msr_index) { -case MSR_IA32_VMX_CR0_FIXED0 : -return &msrs->cr0_fixed0; -case MSR_IA32_VMX_CR4_FIXED0 : -return &msrs->cr4_fixed0; -default : -BUG (); - } -} - -static int vmx_restore_fixed0_msr (struct vcpu_vmx *vmx,u32 msr_index,u64 data) -{ -const u64 *msr =vmx_get_fixed0_msr (&vmcs_config.nested, msr_index); - -/* -* 1 bits (which indicates bits which "must-be-1" during VMX operation) -* must be 1 in the restored value. -*/ -if (!is_bitwise_subset (data, *msr,-1ULL )) -return -EINVAL ; - - *vmx_get_fixed0_msr (&vmx->nested.msrs, msr_index) = data; -return 0 ; -} -/* * Called when userspace is restoring VMX MSRs. *
@@ -1577,8 +1550,9 @@ int vmx_set_vmx_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data)case MSR_IA32_VMX_MISC :return vmx_restore_vmx_misc (vmx, data);case MSR_IA32_VMX_CR0_FIXED0 : +return data && data != vmcs_config.nested.cr0_fixed0;case MSR_IA32_VMX_CR4_FIXED0 : -return vmx_restore_fixed0_msr (vmx, msr_index, data); +return data && data != vmcs_config.nested.cr4_fixed0;case MSR_IA32_VMX_CR0_FIXED1 :case MSR_IA32_VMX_CR4_FIXED1 :/*
arch/x86/kvm/vmx/vmx.c +3 -0
@@ -7712,7 +7712,10 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)struct vcpu_vmx *vmx =to_vmx (vcpu);struct kvm_cpuid_entry2 *entry; + vmx->nested.msrs.cr0_fixed0 = vmcs_config.nested.cr0_fixed0; vmx->nested.msrs.cr0_fixed1 =0xffffffff ; + + vmx->nested.msrs.cr4_fixed0 = vmcs_config.nested.cr4_fixed0; vmx->nested.msrs.cr4_fixed1 =X86_CR4_PCE ;#define cr4_fixed1_update (_cr4_mask, _reg, _cpuid_mask) do { \
tools/testing/selftests/kvm/x86/feature_msrs_test.c +1 -1
@@ -14,7 +14,7 @@static bool is_kvm_controlled_msr (u32 msr) { -return msr ==MSR_IA32_VMX_CR0_FIXED1 || msr ==MSR_IA32_VMX_CR4_FIXED1 ; +return msr >=MSR_IA32_VMX_CR0_FIXED0 && msr <=MSR_IA32_VMX_CR4_FIXED1 ; }/*
Signature · 1 line
2.56.0.385.gac671c819a-goog
Sean Christopherson
maintainerPATCH v3 5/5 KVM: selftests: Verify KVM handling of L1-owned CR0 bits on writes from L2
In reply to Sean ChristophersonFrom: Mahmoud Abumandour <[email protected]>
Expand the nested exceptions test to verify KVM's handling of VMX's CR0 guest/host masks, i.e. of VMX's conditional interception of CR0 writes. LMSW in particular is nasty, as it preserves bits 31:4 as well as CR0.PE if it was already set.
Verify that MOV CR0 and LMSW in L2 that exits to L0, i.e. isn't intercepted by L1, doesn't modify an L1-owned bits and reads the correct value from the read shadow. Use CR0.TS to do the bulk of validation, as its value in hardware can be easily inferred by triggering an #NM in L2. Specifically, run with CR0.TS=1 in hardware but CR0.TS=0 in the read shadow, and verify that non-intercepted writes to CR0 from L2 preserve the CR0.TS values in hardware *and* the shadow.
Note, the downside of using CR0.TS is that any SIMD instructions in L2 will hit a #NM, which is especially problematic since ucall(), e.g. for guest prints and asserts, includes a memset(), for which the compiler likes to generate SSE instructions. To avoid false passes and false failures, have L2 snapshot the CR0 it sees, but do the actual validation in L1. E.g. if L2 observed the wrong value, then doing GUEST_ASSERT_EQ() could get a false pass as memset() would trigger the #NM expected by L1 (when L2 is trying to report the failure).
Include CR0.NE and CR0.PE in the divergent shadow/hardware values as well for extra coverage, e.g. CR0.NE is a must-be-1 bit, and CR0.PE is preserved by LMSW, but if and only if it's already '1'.
--- .../selftests/kvm/include/x86/processor.h | 5 ++ .../kvm/x86/nested_exceptions_test.c | 73 +++++++++++++++++++ 2 files changed, 78 insertions(+)
tools/testing/selftests/kvm/include/x86/processor.h +5 -0
@@ -617,6 +617,11 @@ static inline void set_cr0(u64 val) __asm__ __volatile__("mov %0, %%cr0" : :"r" (val) :"memory" ); } +static inline void lmsw (u8 val) +{ + __asm__ __volatile__("lmsw %w0" : :"r" (val) :"memory" ); +} +static inline u64 get_cr2 (void ) {u64 cr2;
tools/testing/selftests/kvm/x86/nested_exceptions_test.c +73 -0
91 lines not on this page · raw message
Signature · 1 line
2.56.0.385.gac671c819a-goog
Quoted 11 lines · Sean Christopherson
Fix a bug where KVM ignores vmcs12's CR0 guest/host mask when emulating an LMSW from L2, i.e. an LMSW tha exits to L0 but not to L1.
v3:
- Handle theread_shadowinhandle_set_cr0()(and for CR4, because why not).
- Kill offkvm_lmsw()entirely.
- Add a bunch of comments to the selftest to explain the dangers of running with CR0.TS=1.
- Test MOV CR0 as well as LMSW.
- Add a patch to take full ownership of the must-be-1 CR0 and CR4 bits.
Quoted 1 line
[...]
Applied to kvm-x86 vmx, thanks!
[1/5] KVM: nVMX: Preserve L1-owned bits in CR{0,4} read shadows
https://github.com/kvm-x86/linux/commit/3193e9153882 [2/5] KVM: nVMX: Honor L1-owned CR0 bits when emulating intercepted LMSW for L2 https://github.com/kvm-x86/linux/commit/7e538a7bb16b [3/5] KVM: nVMX: Synthesize #GP if L2 writes a "bad" CR0 value via LMSW https://github.com/kvm-x86/linux/commit/7672fa397f00 [4/5] KVM: nVMX: Take full ownership of the post-VMXON CR{0,4} must-be-1 bits https://github.com/kvm-x86/linux/commit/565c14684d6f [5/5] KVM: selftests: Verify KVM handling of L1-owned CR0 bits on writes from L2 https://github.com/kvm-x86/linux/commit/798e54a8c0bc