KVM: x86: MSR filtering and related fixes
Subsystems (4)
Sean Christopherson
maintainerv2 cover KVM: x86: MSR filtering and related fixes
Rework the MSR filtering implementation to treat a given filter instance as an atomic unit, and to properly protect it with SRCU.
Fix two nVMX bugs related to MSR filtering (one directly, one indirectly), and additional cleanup on top.
Regarding the macro insanity in patch 03, I verified the before and after binary output for vmx_set_intercept_for_msr() was identical, again. This required wrapping "if (msr <= 0x1fff)" with (un)likely in both the before and after versions; gcc made seemingly random choices without forcing it to favor a specific branch.
v2:
- Collect R-b. [Paolo, Alex].
- Make the macro insanity slightly less insane. [Paolo]
v1: https://lkml.kernel.org/r/[email protected]
Sean Christopherson (4):
KVM: x86: Protect userspace MSR filter with SRCU, and set
atomically-ish
KVM: nVMX: Handle dynamic MSR intercept toggling
KVM: VMX: Macrofy the MSR bitmap getters and setters
KVM: nVMX: Clean up x2APIC MSR handling for L2
Documentation/virt/kvm/api.rst | 6 +- arch/x86/include/asm/kvm_host.h | 17 ++-- arch/x86/kvm/vmx/nested.c | 161 +++++++++++++------------------- arch/x86/kvm/vmx/vmx.c | 67 +------------ arch/x86/kvm/vmx/vmx.h | 28 ++++++ arch/x86/kvm/x86.c | 109 ++++++++++++--------- 6 files changed, 172 insertions(+), 216 deletions(-)
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Sean Christopherson
maintainerPATCH v2 1/4 KVM: x86: Protect userspace MSR filter with SRCU, and set atomically-ish
Fix a plethora of issues with MSR filtering by installing the resulting filter as an atomic bundle instead of updating the live filter one range at a time. The KVM_X86_SET_MSR_FILTER ioctl() isn't truly atomic, as the hardware MSR bitmaps won't be updated until the next VM-Enter, but the relevant software struct is atomically updated, which is what KVM really needs.
Similar to the approach used for modifying memslots, make arch.msr_filter a SRCU-protected pointer, do all the work configuring the new filter outside of kvm->lock, and then acquire kvm->lock only when the new filter has been vetted and created. That way vCPU readers either see the old filter or the new filter in their entirety, not some half-baked state.
Yuan Yao pointed out a use-after-free in ksm_msr_allowed() due to a TOCTOU bug, but that's just the tip of the iceberg...
- Nothing is __rcu annotated, making it nigh impossible to audit the code for correctness.
- kvm_add_msr_filter() has an unpaired smp_wmb(). Violation of kernel coding style aside, the lack of a smb_rmb() anywhere casts all code into doubt.
- kvm_clear_msr_filter() has a double free TOCTOU bug, as it grabs count before taking the lock.
- kvm_clear_msr_filter() also has memory leak due to the same TOCTOU bug.
The entire approach of updating the live filter is also flawed. While installing a new filter is inherently racy if vCPUs are running, fixing the above issues also makes it trivial to ensure certain behavior is deterministic, e.g. KVM can provide deterministic behavior for MSRs with identical settings in the old and new filters. An atomic update of the filter also prevents KVM from getting into a half-baked state, e.g. if installing a filter fails, the existing approach would leave the filter in a half-baked state, having already committed whatever bits of the filter were already processed.
[*] https://lkml.kernel.org/r/[email protected]
--- Documentation/virt/kvm/api.rst | 6 +- arch/x86/include/asm/kvm_host.h | 17 ++--- arch/x86/kvm/x86.c | 109 +++++++++++++++++++------------- 3 files changed, 78 insertions(+), 54 deletions(-)
Documentation/virt/kvm/api.rst +4 -2
@@ -4806,8 +4806,10 @@ If an MSR access is not permitted through the filtering, it generates a allows user space to deflect and potentially handle various MSR accesses into user space. -If a vCPU is in running state while this ioctl is invoked, the vCPU may -experience inconsistent filtering behavior on MSR accesses. +Note, invoking this ioctl with a vCPU is running is inherently racy. However, +KVM does guarantee that vCPUs will see either the previous filter or the new +filter, e.g. MSRs with identical settings in both the old and new filter will +have deterministic behavior. 4.127 KVM_XEN_HVM_SET_ATTR --------------------------
arch/x86/include/asm/kvm_host.h +9 -8
@@ -931,6 +931,12 @@ enum kvm_irqchip_mode { KVM_IRQCHIP_SPLIT, /* created with KVM_CAP_SPLIT_IRQCHIP */ }; +struct kvm_x86_msr_filter { +u8 count; +bool default_allow:1 ; +struct msr_bitmap_range ranges[16 ]; +}; +#define APICV_INHIBIT_REASON_DISABLE 0#define APICV_INHIBIT_REASON_HYPERV 1#define APICV_INHIBIT_REASON_NESTED 2
@@ -1025,16 +1031,11 @@ struct kvm_arch {bool guest_can_read_msr_platform_info;bool exception_payload_enabled; +bool bus_lock_detection_enabled; +/* Deflect RDMSR and WRMSR to user space when they trigger a #GP */ u32 user_space_msr_mask; - -struct { -u8 count; -bool default_allow:1 ; -struct msr_bitmap_range ranges[16 ]; - } msr_filter; - -bool bus_lock_detection_enabled; +struct kvm_x86_msr_filter __rcu *msr_filter;struct kvm_pmu_event_filter __rcu *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread;
arch/x86/kvm/x86.c +65 -44
187 lines not on this page · raw message
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Sean Christopherson
maintainerPATCH v2 2/4 KVM: nVMX: Handle dynamic MSR intercept toggling
In reply to Sean ChristophersonAlways check vmcs01's MSR bitmap when merging L0 and L1 bitmaps for L2, and always update the relevant bits in vmcs02. This fixes two distinct, but intertwined bugs related to dynamic MSR bitmap modifications.
The first issue is that KVM fails to enable MSR interception in vmcs02 for the FS/GS base MSRs if L1 first runs L2 with interception disabled, and later enables interception.
The second issue is that KVM fails to honor userspace MSR filtering when preparing vmcs02.
Fix both issues simultaneous as fixing only one of the issues (doesn't matter which) would create a mess that no one should have to bisect. Fixing only the first bug would exacerbate the MSR filtering issue as userspace would see inconsistent behavior depending on the whims of L1. Fixing only the second bug (MSR filtering) effectively requires fixing the first, as the nVMX code only knows how to transition vmcs02's bitmap from 1->0.
Move the various accessor/mutators buried in vmx.c into vmx.h so that they can be shared by the nested code.
--- arch/x86/kvm/vmx/nested.c | 108 +++++++++++++++++--------------------- arch/x86/kvm/vmx/vmx.c | 67 ++--------------------- arch/x86/kvm/vmx/vmx.h | 63 ++++++++++++++++++++++ 3 files changed, 115 insertions(+), 123 deletions(-)
arch/x86/kvm/vmx/nested.c +49 -59
@@ -475,29 +475,6 @@ static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,return 0 ; } -/* -* Check if MSR is intercepted for L01 MSR bitmap. -*/ -static bool msr_write_intercepted_l01 (struct kvm_vcpu *vcpu,u32 msr) -{ -unsigned long *msr_bitmap; -int f =sizeof (unsigned long ); - -if (!cpu_has_vmx_msr_bitmap ()) -return true; - - msr_bitmap =to_vmx (vcpu)->vmcs01.msr_bitmap; - -if (msr <=0x1fff ) { -return !!test_bit (msr, msr_bitmap +0x800 / f); - }else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) { - msr &=0x1fff ; -return !!test_bit (msr, msr_bitmap +0xc00 / f); - } - -return true; -} -/* * If a msr is allowed by L0, we should check whether it is allowed by L1. * The corresponding bit will be cleared unless both of L0 and L1 allow it.
@@ -551,6 +528,34 @@ static inline void enable_x2apic_msr_intercepts(unsigned long *msr_bitmap) } } +#define BUILD_NVMX_MSR_INTERCEPT_HELPER (rw) \ +static inline \ +void nested_vmx_set_msr_##rw##_intercept(struct vcpu_vmx *vmx, \ + unsigned long *msr_bitmap_l1, \ + unsigned long *msr_bitmap_l0, u32 msr) \ +{ \ + if (vmx_test_msr_bitmap_##rw(vmx->vmcs01.msr_bitmap, msr) || \ + vmx_test_msr_bitmap_##rw(msr_bitmap_l1, msr)) \ + vmx_set_msr_bitmap_##rw(msr_bitmap_l0, msr); \ + else \ + vmx_clear_msr_bitmap_##rw(msr_bitmap_l0, msr); \ +} +BUILD_NVMX_MSR_INTERCEPT_HELPER(read) +BUILD_NVMX_MSR_INTERCEPT_HELPER(write) + +static inline void nested_vmx_set_intercept_for_msr(struct vcpu_vmx *vmx, + unsigned long *msr_bitmap_l1, + unsigned long *msr_bitmap_l0, + u32 msr, int types) +{ + if (types & MSR_TYPE_R) + nested_vmx_set_msr_read_intercept(vmx, msr_bitmap_l1, + msr_bitmap_l0, msr); + if (types & MSR_TYPE_W) + nested_vmx_set_msr_write_intercept(vmx, msr_bitmap_l1, + msr_bitmap_l0, msr); +} +/* * Merge L0's and L1's MSR bitmap, return false to indicate that * we do not use the hardware.
@@ -558,10 +563,11 @@ static inline void enable_x2apic_msr_intercepts(unsigned long *msr_bitmap)static inline bool nested_vmx_prepare_msr_bitmap (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12) { +struct vcpu_vmx *vmx =to_vmx (vcpu);int msr;unsigned long *msr_bitmap_l1; -unsigned long *msr_bitmap_l0 =to_vmx (vcpu)->nested.vmcs02.msr_bitmap; -struct kvm_host_map *map = &to_vmx (vcpu)->nested.msr_bitmap_map; +unsigned long *msr_bitmap_l0 = vmx->nested.vmcs02.msr_bitmap; +struct kvm_host_map *map = &vmx->nested.msr_bitmap_map;/* Nothing to do if the MSR bitmap is not in use. */ if (!cpu_has_vmx_msr_bitmap() ||
@@ -612,42 +618,26 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, } } -/* KVM unconditionally exposes the FS/GS base MSRs to L1. */ -nested_vmx_disable_intercept_for_msr (msr_bitmap_l1, msr_bitmap_l0, -MSR_FS_BASE ,MSR_TYPE_RW ); - -nested_vmx_disable_intercept_for_msr (msr_bitmap_l1, msr_bitmap_l0, -MSR_GS_BASE ,MSR_TYPE_RW ); - -nested_vmx_disable_intercept_for_msr (msr_bitmap_l1, msr_bitmap_l0, -MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); -/* -* Checking the L0->L1 bitmap is trying to verify two things: -* -* 1. L0 gave a permission to L1 to actually passthrough the MSR. This -* ensures that we do not accidentally generate an L02 MSR bitmap -* from the L12 MSR bitmap that is too permissive. -* 2. That L1 or L2s have actually used the MSR. This avoids -* unnecessarily merging of the bitmap if the MSR is unused. This -* works properly because we only update the L01 MSR bitmap lazily. -* So even if L0 should pass L1 these MSRs, the L01 bitmap is only -* updated to reflect this when L1 (or its L2s) actually write to -* the MSR. +* Always check vmcs01's bitmap to honor userspace MSR filters and any +* other runtime changes to vmcs01's bitmap, e.g. dynamic pass-through. */ -if (!msr_write_intercepted_l01 (vcpu,MSR_IA32_SPEC_CTRL )) -nested_vmx_disable_intercept_for_msr ( - msr_bitmap_l1, msr_bitmap_l0, -MSR_IA32_SPEC_CTRL , -MSR_TYPE_R |MSR_TYPE_W ); - -if (!msr_write_intercepted_l01 (vcpu,MSR_IA32_PRED_CMD )) -nested_vmx_disable_intercept_for_msr ( - msr_bitmap_l1, msr_bitmap_l0, -MSR_IA32_PRED_CMD , -MSR_TYPE_W ); - -kvm_vcpu_unmap (vcpu, &to_vmx (vcpu)->nested.msr_bitmap_map, false); +nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0, +MSR_FS_BASE ,MSR_TYPE_RW ); + +nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0, +MSR_GS_BASE ,MSR_TYPE_RW ); + +nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0, +MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); + +nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0, +MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW ); + +nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0, +MSR_IA32_PRED_CMD ,MSR_TYPE_W ); + +kvm_vcpu_unmap (vcpu, &vmx->nested.msr_bitmap_map, false);return true; }
arch/x86/kvm/vmx/vmx.c +3 -64
85 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +63 -0
69 lines not on this page · raw message
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Sean Christopherson
maintainerPATCH v2 3/4 KVM: VMX: Macrofy the MSR bitmap getters and setters
In reply to Sean ChristophersonAdd builder macros to generate the MSR bitmap helpers to reduce the amount of copy-paste code, especially with respect to all the magic numbers needed to calc the correct bit location.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.h | 77 ++++++++++-------------------------------- 1 file changed, 17 insertions(+), 60 deletions(-)
arch/x86/kvm/vmx/vmx.h +17 -60
@@ -393,68 +393,25 @@ void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, bool value);void vmx_update_cpu_dirty_logging (struct kvm_vcpu *vcpu); -static inline bool vmx_test_msr_bitmap_read (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -return test_bit (msr, msr_bitmap +0x000 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -return test_bit (msr &0x1fff , msr_bitmap +0x400 / f); -return true; -} - -static inline bool vmx_test_msr_bitmap_write (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -return test_bit (msr, msr_bitmap +0x800 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -return test_bit (msr &0x1fff , msr_bitmap +0xc00 / f); -return true; -} - -static inline void vmx_clear_msr_bitmap_read (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -__clear_bit (msr, msr_bitmap +0x000 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -__clear_bit (msr &0x1fff , msr_bitmap +0x400 / f); -} - -static inline void vmx_clear_msr_bitmap_write (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -__clear_bit (msr, msr_bitmap +0x800 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -__clear_bit (msr &0x1fff , msr_bitmap +0xc00 / f); -} - -static inline void vmx_set_msr_bitmap_read (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -__set_bit (msr, msr_bitmap +0x000 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -__set_bit (msr &0x1fff , msr_bitmap +0x400 / f); -} - -static inline void vmx_set_msr_bitmap_write (ulong *msr_bitmap,u32 msr) -{ -int f =sizeof (unsigned long ); - -if (msr <=0x1fff ) -__set_bit (msr, msr_bitmap +0x800 / f); -else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) -__set_bit (msr &0x1fff , msr_bitmap +0xc00 / f); +#define __BUILD_VMX_MSR_BITMAP_HELPER (rtype, action, bitop, access, base) \ +static inline rtype vmx_##action##_msr_bitmap_##access(unsigned long *bitmap, \ + u32 msr) \ +{ \ + int f = sizeof(unsigned long); \ + \ + if (msr <= 0x1fff) \ + return bitop##_bit(msr, bitmap + base / f); \ + else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff)) \ + return bitop##_bit(msr & 0x1fff, bitmap + (base + 0x400) / f); \ + return (rtype)true; \ } +#define BUILD_VMX_MSR_BITMAP_HELPERS (ret_type, action, bitop) \ + __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0x0) \ + __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 0x800) +BUILD_VMX_MSR_BITMAP_HELPERS(bool, test, test) +BUILD_VMX_MSR_BITMAP_HELPERS(void, clear, __clear) +BUILD_VMX_MSR_BITMAP_HELPERS(void, set, __set)static inline u8 vmx_get_rvi (void ) {
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Sean Christopherson
maintainerPATCH v2 4/4 KVM: nVMX: Clean up x2APIC MSR handling for L2
In reply to Sean ChristophersonClean up the x2APIC MSR bitmap intereption code for L2, which is the last holdout of open coded bitmap manipulations. Freshen up the SDM/PRM comment, rename the function to make it abundantly clear the funky behavior is x2APIC specific, and explain _why_ vmcs01's bitmap is ignored (the previous comment was flat out wrong for x2APIC behavior).
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 53 +++++++++++---------------------------- arch/x86/kvm/vmx/vmx.h | 8 ++++++ 2 files changed, 22 insertions(+), 39 deletions(-)
arch/x86/kvm/vmx/nested.c +14 -39
@@ -476,44 +476,19 @@ static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu, }/* -* If a msr is allowed by L0, we should check whether it is allowed by L1. -* The corresponding bit will be cleared unless both of L0 and L1 allow it. +* For x2APIC MSRs, ignore the vmcs01 bitmap. L1 can enable x2APIC without L1 +* itself utilizing x2APIC. All MSRs were previously set to be intercepted, +* only the disable intercept case needs to be handled. */ -static void nested_vmx_disable_intercept_for_msr (unsigned long *msr_bitmap_l1, -unsigned long *msr_bitmap_nested, -u32 msr,int type) +static void nested_vmx_disable_intercept_for_x2apic_msr (unsigned long *msr_bitmap_l1, +unsigned long *msr_bitmap_l0, +u32 msr,int type) { -int f =sizeof (unsigned long ); +if (type &MSR_TYPE_R && !vmx_test_msr_bitmap_read (msr_bitmap_l1, msr)) +vmx_clear_msr_bitmap_read (msr_bitmap_l0, msr); -/* -* See Intel PRM Vol. 3, 20.6.9 (MSR-Bitmap Address). Early manuals -* have the write-low and read-high bitmap offsets the wrong way round. -* We can control MSRs 0x00000000-0x00001fff and 0xc0000000-0xc0001fff. -*/ -if (msr <=0x1fff ) { -if (type &MSR_TYPE_R && - !test_bit (msr, msr_bitmap_l1 +0x000 / f)) -/* read-low */ -__clear_bit (msr, msr_bitmap_nested +0x000 / f); - -if (type &MSR_TYPE_W && - !test_bit (msr, msr_bitmap_l1 +0x800 / f)) -/* write-low */ -__clear_bit (msr, msr_bitmap_nested +0x800 / f); - - }else if ((msr >=0xc0000000 ) && (msr <=0xc0001fff )) { - msr &=0x1fff ; -if (type &MSR_TYPE_R && - !test_bit (msr, msr_bitmap_l1 +0x400 / f)) -/* read-high */ -__clear_bit (msr, msr_bitmap_nested +0x400 / f); - -if (type &MSR_TYPE_W && - !test_bit (msr, msr_bitmap_l1 +0xc00 / f)) -/* write-high */ -__clear_bit (msr, msr_bitmap_nested +0xc00 / f); - - } +if (type &MSR_TYPE_W && !vmx_test_msr_bitmap_write (msr_bitmap_l1, msr)) +vmx_clear_msr_bitmap_write (msr_bitmap_l0, msr); }static inline void enable_x2apic_msr_intercepts (unsigned long *msr_bitmap)
@@ -582,7 +557,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,/* * To keep the control flow simple, pay eight 8-byte writes (sixteen * 4-byte writes on 32-bit systems) up front to enable intercepts for -* the x2APIC MSR range and selectively disable them below. +* the x2APIC MSR range and selectively toggle those relevant to L2. */ enable_x2apic_msr_intercepts (msr_bitmap_l0);
@@ -601,17 +576,17 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, } } -nested_vmx_disable_intercept_for_msr ( +nested_vmx_disable_intercept_for_x2apic_msr ( msr_bitmap_l1, msr_bitmap_l0,X2APIC_MSR (APIC_TASKPRI ),MSR_TYPE_R |MSR_TYPE_W );if (nested_cpu_has_vid (vmcs12)) { -nested_vmx_disable_intercept_for_msr ( +nested_vmx_disable_intercept_for_x2apic_msr ( msr_bitmap_l1, msr_bitmap_l0,X2APIC_MSR (APIC_EOI ),MSR_TYPE_W ); -nested_vmx_disable_intercept_for_msr ( +nested_vmx_disable_intercept_for_x2apic_msr ( msr_bitmap_l1, msr_bitmap_l0,X2APIC_MSR (APIC_SELF_IPI ),MSR_TYPE_W );
arch/x86/kvm/vmx/vmx.h +8 -0
@@ -393,6 +393,14 @@ void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, bool value);void vmx_update_cpu_dirty_logging (struct kvm_vcpu *vcpu); +/* +* Note, early Intel manuals have the write-low and read-high bitmap offsets +* the wrong way round. The bitmaps control MSRs 0x00000000-0x00001fff and +* 0xc0000000-0xc0001fff. The former (low) uses bytes 0-0x3ff for reads and +* 0x800-0xbff for writes. The latter (high) uses 0x400-0x7ff for reads and +* 0xc00-0xfff for writes. MSRs not covered by either of the ranges always +* VM-Exit. +*/ #define __BUILD_VMX_MSR_BITMAP_HELPER (rtype, action, bitop, access, base) \ static inline rtype vmx_##action##_msr_bitmap_##access(unsigned long *bitmap, \ u32 msr) \
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog